PagerDuty Blog

This looks like a great tool for corporate IT departments too. From their website (http://www.pagerduty.com), I like how it works with any monitoring system as long as the tool can send email. That would make it easy to integrate.

Disclaimer: I work at http://www.pagerduty.com so feel free to tar and feather accordingly. Carrying the duty pager is a painful experience for some fraction of companies, BUT the long term trends are promising. Here's what I'd keep an eye out for (I've been on call for ~5 years): * Does being on call affect your other commitments? At PD we scale back the number of predicted story points by ~50 for the devs that are on call. * Are you empowered to permanently fix the root cause of whatever woke you up? (that's where that 50% of time goes) If you aren't, that's a big red flag. Not all developers take advantage of it, but the ones that do are much happier once they kill the root cause with fire. * Are you compensated for on call? Among our customers, we have a few that pay $500/week for on call duty, that seems to be the rate at which you can easily find people to swap shifts with. * Make sure you are off call sometimes. Seriously. * Who owns the pain report? Someone needs to track how often (and when) people are disturbed and make sure that you are making progress as a team (Github's Ops team is amazingly good at this). If the house is always on fire, you're not a firefighter, you're a person who lives in a flaming house. * Is it a NOC model, where you can write down common things to try to solve a type of problem (and then you're only paged as an exception) or are you paged for everything? (That's a severe over simplification) * What is the expected response…

Another favorite bug investigation of you're into this sort of thing: https://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-pois…

Most FAANG-level software engineering jobs will require being on-call with something like PagerDuty [0]. Though I guess you could get by with using just text message notifications and not necessarily the app. [0] https://www.pagerduty.com/

We use Elixir very heavily here at PagerDuty. The vast majority of our services are written in Elixir. There's generally a shared sentiment across engineering that developing in Elixir is indeed rather satisfying. From a technical standpoint, it's highly suited to our business due to how well it does with concurrency along with its fault tolerance. Here's a nice blog post by one of our Principal engineers on Elixir, it's a great read: https://www.pagerduty.com/blog/elixir-at-pagerduty/

https://www.pagerduty.com/ It's an on-call management system that receives events from multiple sources and routes to the appropriate people.

PagerDuty | Senior Software Engineer, Analytics | Toronto | ONSITE PagerDuty is how software services manage their resiliency. From paging the right person when there's an outage, to orchestrating a response, to conducting a postmortem, to obtaining data-driven insights that improve processes and performance. Come work on my team in Toronto. Great culture. Great Benefits. A real company with serious revenues. Disciplined management. PagerDuty has it all, and yes, we're part of the YC family. I'm [email protected] if you'd like to hear more before formally applying. https://www.pagerduty.com/careers/jobs/?jobid=a0K370000073FAHEA2 p.s. We're hiring lots of other engineers and engineering managers in Toronto and San Francisco, not to mention opportunities in product, sales, marketing, support... We're successful and growing.

Paged as in alerted or summoned - this service https://www.pagerduty.com/

PagerDuty | Multiple Roles | San Francisco or Toronto | Full Time, Remote https://www.pagerduty.com/careers PagerDuty is growing like crazy and we need great engineers to help us build what Incident Management should look like. Every company is becoming more dependent on software, and with that, they are becoming more dependent on the people they need to run their software. We aim to help people build great software by being able to rapidly learn and get better. We are hiring across the entire stack: Backend, Frontend, Infrastructure, Mobile, Security, etc. Check out our jobs site, if there is a role that you are interested in, feel free to get in touch with me directly or shoot over any questions.

PagerDuty - San Francisco and Toronto, Full time Thinking of moving to SF/SV? Make the web a more stable place. Our offices are full of a wide range of great people, working on a problem that matters. We need engineers of all flavors and stripes, engineering managers, sales, recruiters and especially more evangelists. http://bit.ly/1xjBVWz In Toronto? Take the TTC to work in Silicon Valley. We were started in Canada, and still do some of our toughest engineering work in a great office a block from St. Andrew Station. We're looking to build out our engineering team as well as hire a lead technical recruiter. http://bit.ly/1K4yt7t For more reasons to work here: http://www.pagerduty.com/company/work-with-us/

PagerDuty - San Francisco and Toronto, Full time In Toronto? Take the TTC to work in Silicon Valley. We were started in Canada, and still do some of our toughest engineering work in a great office a block from St .Andrew Station. It's not just engineers, we're also looking for technical sourcers and a cunning desktop sysadmin. http://pduty.me/hnposttoronto Thinking of moving to SF/SV? Make the web a more stable place. Our offices are full of a wide range of great people, working on a problem that matters. We need engineers of all flavours and stripes, engineering managers, sales, recruiters and especially more evangelists. http://pduty.me/hnpostsf For more reasons to work here: http://www.pagerduty.com/company/work-with-us/

Congratulations! PagerDuty has an amazing technical team and their focus on reliability is an inspiration (just see their post on Failure Fridays[1]). Which is super important for when your systems are at the worst and require immediate manual help. [1] http://blog.pagerduty.com/2013/11/failure-friday-at-pagerduty/

pagerduty [0] does this and more. You also don't need to host your own infra for the alert system. [0]http://www.pagerduty.com/

I'd love to get feedback on pricing etc, to see how far off I am, if you have a second to email me? If not, nbd. http://blog.pagerduty.com/2011/03/on-call-best-practices-part-1/ This is a series of posts that have pretty sane defaults; I personally would not do a daily rotation, but rather rotations of 5 days, and alternating weekends (one guy does M-F, one guy does Sat/Sund) and you switch off.

You should checkout the following blog post as a possible solution to item 1 on your list. You can setup a phone number that will page the on-call person. http://blog.pagerduty.com/2012/08/how-to-provide-24x7-phone-support-us…

PagerDuty (http://www.pagerduty.com) is hiring both interns and full-time software engineers. We're still quite small - 5 people. We're located in the SOMA district of San Francisco. Email: [email protected].

We use PagerDuty (http://pagerduty.com) at HipChat and while I absolutely loathe being woken up by it, it's helped us identify issues during off-peak hours much more quickly. But no matter what systems you have in place or how many hundreds of deploys you've done, there's always a new way for things to break.

Just getting Pager Duty for 6 months for $17 is a good deal, if you need it. It'd be $72 normally (OK, $60 because they have a 30-day free trial thing). Yeah, the absence of linkage is lame. http://pagerduty.com/ http://kb.weebly.com/faq-pro.html http://crowdbooster.com/ http://www.co2stats.com/

Let me be clearer - the link does not load for me. http://blog.pagerduty.com is dead here. http://pagerduty.com loads fine though.

Pager Duty is out and can be used today: http://pagerduty.com

For those of us unfamiliar, this quote is from a TV show, not a real CEO. Don't get me wrong, it's a damned good parody, but let's not mistake fiction for reality. Some real ones: > None of this would be possible without you, our leadership, and our board — thank you for your grit and resilience, your commitment to our customers and your support of our values and people. I am reminded in moments like this, of something Martin Luther King said, that “the ultimate measure of a [leader] is not where [they] stand in the moments of comfort and convenience, but where [they] stand in times of challenge and controversy.” https://www.pagerduty.com/blog/letter-to-employees/ > We had to lay off 90% of our support team because of this AI chatbot. Tough? Yes. Necessary? Absolutely. https://www.boredpanda.com/ceo-fires-support-staff-after-ai-outperform… And because a picture is worth 1000 words, Braden Wallake's cryface selfie https://www.reddit.com/r/technology/comments/wlg92h/ceos_linkedin_cryi…

PagerDuty | San Francisco | Toronto | Atlanta PagerDuty is the leading digital operations management platform for organizations. Over 10,000 enterprises and small to mid-size organizations globally trust PagerDuty to improve digital operations, drive revenue, mitigate threats, protect assets, and delight customers. We were included in the 2017 Deloitte Technology Fast 500 for the second year in a row, Inc. 500 and Forbes Cloud 100 lists as well as the 2018 Best Places to Work in the Bay Area. We're hiring for a number of roles: Senior Software Engineer | San Francisco, Toronto, Atlanta https://jobs.lever.co/pagerduty/6f501ae4-a5ef-46d0-955a-d3417b147b9b Senior Database Automation Engineer San Francisco, Toronto, Atlanta https://www.pagerduty.com/careers/jobs/?jobid=a0K37000008p93CEAQ Senior Data Engineer, Analytics | San Francisco, Toronto, Atlanta https://jobs.lever.co/pagerduty/73c0c2cf-e786-4438-8c94-5a20e3b88585 And many more: https://jobs.lever.co/pagerduty?lever-origin=applied&lever-source%…

PagerDuty | San Francisco | Toronto | Remote PagerDuty is the leading digital operations management platform for organizations. Over 10,000 enterprises and small to mid-size organizations globally trust PagerDuty to improve digital operations, drive revenue, mitigate threats, protect assets, and delight customers. We were included in the 2017 Deloitte Technology Fast 500 for the second year in a row, Inc. 500 and Forbes Cloud 100 lists as well as the 2018 Best Places to Work in the Bay Area. We're hiring for a number of roles: Senior Software Engineer | San Francisco, Toronto, Remote https://jobs.lever.co/pagerduty/6f501ae4-a5ef-46d0-955a-d3417b147b9b Senior Database Automation Engineer | San Francisco, Toronto, Remote https://jobs.lever.co/pagerduty/1c2ac0da-1499-4e17-9de9-08995687ab3c Senior iOS Engineer | Toronto https://jobs.lever.co/pagerduty/e8eb05ff-adf0-4f10-a26d-d5637234ed92 And many more: https://www.pagerduty.com/careers/

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here. Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images. In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side. We also have evidence from access logs provided by Linode…

This might work at low traffic levels, but there are better ways. In our internal error catcher, we've seen deploys that caused 20,000+ errors in under a minute. Even when we used similar techniques to mail ourselves errors on a low traffic site, it was important to introduce dead time, not alerting again for (say) 5 mins after the first alert so as not to flood our inbox. A useful technique btw is to use a ring buffer to collect debug logging, and when an error occurs dump this buffer into the message along with the stacktrace. It gives you more context when you're not logging debug to disk, and is fast. (see eg http://www.exampler.com/writing/ring-buffer.pdf, https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SMTP… Services like https://www.pagerduty.com/ can contact you on more channels when problems happen, and also deal with dead time etc not /re/ alerting you constantly. The Slack mobile client would have got every one of those messages as a push. Pagerduty won't help with aggregating/exploring stacktraces though, for that, there's eg https://airbrake.io or for mobile apps, https://try.crashlytics.com/. Finally there's also http://www.splunk.com/ for aggregating logs; you can build some quite complex queries from it and do alerting on the results (not quite as fancy as pagerduty, but functional). There are many other tools in this space, worth looking around to see if there's any SaaS you can use or crib ideas from.

pagerduty http://www.pagerduty.com/

It's actually pretty straightforward without a lot of get-outs. See Service Credits in the Service Level Agreement [1]. 1: https://www.pagerduty.com/standard-service-level-agreement/

There was the recent use of "Dutonians" by Pager Duty's CEO https://www.pagerduty.com/blog/letter-to-employees/

The Gizmodo article is from today, but the letter itself was posted the 24th. Here's the original source: https://www.pagerduty.com/blog/improving-pagerdutys-operational-resili… Here's a Tweet quoting it at 11am PST: https://twitter.com/noahchestnut/status/1617960469565931523 And the tweet in the parent was ~4 hours later at 2:52pm PST: https://twitter.com/mattstratton/status/1618018859763798016

Sorry... "People First" https://www.pagerduty.com/blog/pagerdutys-people-first-approach-to-com…