Kitchen Soap

Perhaps more accurate is that there is no one root cause, but a bunch of contributing factors. I had a quick search to find my favourite article on this one, it might be https://willgallego.com/2018/04/02/no-seriously-root-cause-is-a-fallac… as it demolishes Five Whys but https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly… is a classic

As always it depends on the organization or team. I have worked on teams where the title was nothing more than a title and didn't relate to their skill level or work ethic. One of my favorite posts on the subject comes from John Allspaw: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/ An important part of the equation is a companies engineering structure. Camille Fournier has a great set of slides about creating structure earlier rather than later and why it is needed: http://www.slideshare.net/CamilleFournier1/how-to-go-from-structureles…

This is by far the best article I've ever read on the subject. http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/

I have bookmarked John Allspaw's thorough writeup about this topic because I figure it's worth re-reading to internalize: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/ He also makes reference to the Dreyfus model of skill acquisition which is worth keeping in mind: http://en.wikipedia.org/wiki/Dreyfus_model_of_skill_acquisition

A great read on this topic: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/

This has been posted and discussed several times on HN and I consider it to be the best approach: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/# I'm not here to "win", I'm here to implement the best technical solution.

The post is quite poor and suffer a lot from hindsight bias. Following article is so much better: http://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/

Here's the best take I've read on the subject, by John Allspaw of Etsy: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/ Choice quote: "I expect a “senior” engineer to be a mature engineer." He elaborates at length about what that means in the post.

Which is unfortunate because tmarthal's link is quite good: > Here is a great article on what it means to be a Senior engineer: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/ > The long answer is maturity. Then the author goes to explain what maturity brings to the discipline of software engineering.

That's a very good point, John Allspaw from Etsy has a great post on that topic: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/

Here is a great article on what it means to be a Senior engineer: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/?utm_… The long answer is maturity. Then the author goes to explain what maturity brings to the discipline of software engineering.

Here's a dump of blogs I follow that have lots of devops-oriented contents. http://www.kartar.net/ http://www.kitchensoap.com/ http://www.masterzen.fr/ http://blog.nistu.de/ http://www.jedi.be/blog/ http://www.krisbuytaert.be/blog/ http://www.unixdaemon.net/ http://www.rottenbytes.info/ http://blog.lusis.org/ http://bitfieldconsulting.com/ http://agilesysadmin.net/

Both of John Allspaw's books (the latter two on your list) look good from their table of contents. And if you're in doubt, John is now VP of Ops at Etsy and came from Flickr before that: http://www.kitchensoap.com/about-me/ His blog is interesting too: http://www.kitchensoap.com/ So without having read the books, I would shoot for the latter 2 if I wanted to have hard copies around to introduce me to this kind of stuff.

Why do software engineers insist on being paid when they never deliver any working software? Imagine a world in which programmers were only paid once their code hit production. My hypothesis is that estimation skills would improve at a geometric rate. Said differently: good programmers are not afraid of making estimates. This classic article says it all better than I could: https://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/ From The Unwritten Laws of Engineering, quoted in the article: Promises, schedules, and estimates are necessary and important instruments in a well-ordered business. Many engineers fail to realize this, or habitually try to dodge the irksome responsibility for making commitments. You must make promises based upon your own estimates for the part of the job for which you are responsible, together with estimates obtained from contributing departments for their parts. No one should be allowed to avoid the issue by the old formula, “I can’t give a promise because it depends upon so many uncertain factors." Also from the article: Avoiding responsibility for estimates is another way of saying, “I’m not ready to be relied upon for building critical pieces of infrastructure.”

There is a really, really interesting breakdown of this document on kitchensoap.com [0] The writer breaks down why this document is not a post-mortem despite superficial similarities. >Again, the purpose of the doc is to point out where Knight violated rules. It is not: 1) a description of the multiple trade-offs that engineering at Knight made or considered when designing fault-tolerance in their systems, or 2) how Knight as an organization evolved over time to focus on evolving some procedures and not others, or 3) how engineers anticipated in preparation for deploying support for the new RLP effort on Aug 1, 2012. >To equate any of those things with violation of a rule is a cognitive leap that we should stay very far away from. >It’s worth mentioning here that the document only focuses on failures, and makes no mention of successes. How Knight succeeded during diagnosis and response is unknown to us, so a rich source of data isn’t available. Because of this, we cannot pretend the document to give explanation. He also makes an interesting point related to what you're saying: the SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it. >What is deemed “appropriate”, it would seem, is dependent on the outcome. Had an accident? It was not appropriate control. Didn’t have an accident? It must be appropriate control. This would mean that Knight Capital did have appropriate controls the day…

This article is older now but is still a great read. It has a different list which I liked a lot (the below is directly from the article): Self-Serving Bias – basically: if something is good, it’s probably because of something I did or thought of. If it’s bad, it’s probably the doing of someone else. Fundamental Attribution Error – basically: the bad results that someone else got from his work must have something to do with how he is, personally (stupid, clumsy, sloppy, etc.) whereas if I get bad results, it’s because of the context that I was in, the pressure I was under, the situation I was in, etc. Hindsight Bias – (it is said that this is the most-studied phenomenon in the history of modern psychology) basically: after an untoward or negative event (a severe bug, an outage, etc.) “I knew it all along!”. It is the very strong tendency to view the past more simply than it was in reality. You can tell there is Hindsight Bias going on when descriptions involve counterfactuals, or “…they should have…”, or “…how did they not see that, it’s so obvious!”. Outcome Bias – like above, this comes up after a surprising or negative event. If the event was very damaging, expensive to clean up, or severe, then the decisions or actions that contributed to that event are judged to be very stupid, reckless, or negligent. The judgement is proportional to how severe the event was. Planning Fallacy – (related to the point about making estimates under uncertainty, above) basically: being…

It's a sign of poor management that someone has to be fired when something goes wrong, outages are learning situations for all involved, and it is widely held that the person who took the action that caused an outage is not responsible, but that all involved are responsible. See John Allspaw's Swiss Cheese Theory : http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-… . [ Edit: I guess it's not Allspaw's model, but he applies it to systems engineering rather well - http://en.wikipedia.org/wiki/Swiss_cheese_model ] "Accidents emerge from a confluence of conditions and occurrences that are usually associated with the pursuit of success, but in this combination—each necessary but only jointly sufficient—able to trigger failure instead." The person who pushed the button is not at fault, the manager is not at fault, the guy who designed the button is not at fault - all are jointly responsible. Blaming the intern does, however, reflect extremely poorly on Itoh and everyone else in the chain of command. A superior who demands retribution for a simple mistake that happened to cause him or her pain is basically worthless. But, I forget, we're talking about Ronald Reagan.

This comment has now been both down-voted and up-voted multiple times. Would one of you down-voters care to explain why you feel this point is unworthy of discussion? A blog post [0] that was up-voted near the top of Hacker News last week made the same point in terms that apparently are more palatable to some readers. If you haven't read it, I encourage you to do so. Two of the academic papers/book chapters [1, 2] that the blogger cites specifically discuss risk assessment of nuclear power stations in the context I described. All of this stuff is well known. It's why web-sites' uptimes are never anywhere near the five nines advertised and why we have "flash crashes" in our financial markets. The focus of my original comment is not only on identifying and assigning probabilities to causes of failures that are inherently difficult to discern, but also on the inadequacy of metrics like expected value in quantifying the effect of rare but spectacular outcomes. Is it really not even worth discussing these problems with our risk-assessment methodologies when the price of failure is as high as it is in case of an uncontrolled meltdown of a nuclear reactor? 0. http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-… 1. http://csel.eng.ohio-state.edu/woods/error/fund_surp.pdf 2. http://crpit.com/confpapers/CRPITV86Qureshi.pdf

First, I highly recommend "Deployment is just a part of dev/ops cooperation" http://www.kitchensoap.com/2009/12/12/devops-cooperation-doesnt-just-h… Second, this isn't a question unique to the dev/ops relationship. What about the dev v. project manager? If you have a good team of developers, why have a PM? Having worked as a dev, a system engineer, and newly in the PM department, everything I've done since being a 'code monkey' is motivated by a desire to make the dev's jobs easier. Maybe that's because of my roots, but I think it makes the most sense. Where I work, we have a useful distinction between Sysadmins and System Engineers. Broadly speaking, admins are the i-dotters and t-crossers and do things with well-defined processes while engineers are responsible for everything else. As an engineer, I wanted to make as much information about the production environment available to the developers as possible. I also worked to make sure the dev environment would mirror production as closely as possible given that we only had one dev sandbox (requests for a staging environment were declined). These two things, combined with being a good teacher, meant I rarely had requests for CLI access to any of our environments.

Before the cloud, people were trying to reduce the mean time between failure (MTBF) essentially trying to prevent a thing from failing. With cloud, people are trying to recover as quickly as possible (mean time to recovery) accepting that things will fail —- it’s about how fast you can react to it. John Allspaw (previously CTO at Etsy) has written about this: https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/

Hindsight is a helluva drug. The SEC report cannot be viewed as a “postmortem.” https://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/

There is no root cause. https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly… https://blog.acolyer.org/2016/02/10/how-complex-systems-fail/ http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fai…

Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly…

That moral seems related to a meme floating around that Mean Time To Recovery is more important than Mean Time Between Failures. I think it came from, or through, John Allspaw: https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/ It's popular with a lot of people i know, because it's one of those ideas that's counterintuitive but believable, and so makes you feel really smart for knowing it.

For a deeper look from the systems engineering side, check out http://mit.edu/psas, specifically the book-length treatment in “Engineering a Safer World”. For applications of similar ideas to cloud software and devops-style environments, https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly… is also helpful!

It's implied by their claim that there is a single responsible individual. If someone else reviewed it and said "yep, looks good, deploy the change" how is there one "individual responsible"? The idea of an isolated root cause or single human error in the failure of complex systems is bogus anyway. I'm a huge fan of the work in this area championed by John Allspaw [0]. [0]: https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly…

Check this out: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/

For me, John Allspaw's article On Being A Senior Engineer[1] lays it all out pretty well. It addresses the issues we all seem to have with this article better than I could. Generation X (and even more so generation Y) are cultures of immediate gratification. I’ve worked with a staggering number of engineers that expect the “career path” to take them to the highest ranks of the engineering group inside 5 years just because they are smart. This is simply impossible in the staggering numbers I’ve witnessed. Not everyone can be senior. If, after five years, you are senior, are you at the peak of your game? After five more years will you not have accrued more invaluable experience? What then? “Super engineer”? Five more years? “Super-duper engineer.” I blame the youth of our discipline for this affliction. http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/

HOWTO: Be more productive http://www.aaronsw.com/weblog/productivity On Being A Senior Engineer http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/