Doug Seven

The risk is context dependent. It could be a great idea or it could be the end of the company. Classic story: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

My favorite disaster of all time: http://fortune.com/2012/08/02/why-knight-lost-440-million-in-45-minute… More details with references: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

> A container is just a fancy executable. Not just that, but containers also have a lot of knowledge and tooling around them for running services more painlessly. I suggest that you familiarize yourself with this excellent site: https://12factor.net/ > No one ever talked about "executable orchestration". Actually thousands of collective developer-years have gone into developing Java EE, OSGi and also figuring out how to run "containers" inside of Tomcat, GlassFish and many other web servers, which basically were executables or modules that contained application logic that had to be run. Both in the Java ecosystem and many others, it was eventually decided that attempts like this (as well as going in the separate direction and shipping VMs or VM images, a la Vagrant) don't really work that nicely and containers were a middle ground that people settled on - unified bundles of almost everything (sans kernel) that any given application might need to run. Of course, other interesting projects like Flatpak, AppImage, snaps and even functions-as-a-service all concern themselves with how any piece of executable code should be run, especially in the case of latter. And then there are systems which attempt to distribute work across any number of worker nodes, actually you can just look at the entire HPC industry. Thus, I believe that one can definitely say that a plethora of options for running things has been explored and will continue to be something that people research and…

> and they all forgot how to use git revert? Implying they use any form of VCS at all... The Knightmare fiasco might serve as a useful datapoint for re-calibrating your expectations: http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ > I hate to sound like a conspiracy wingnut, but it seems unlikely that there were major outages at NYSE, United Airlines, and the WSJ all on the same day and it's merely a coincidence. My expectations are calibrated at roughly "this is noteworthy?" Taking the long view of things, it seems unlikely it's taken this long for there to be three major outages on the same day. Well, rather, there have been - one wee little "yet another OpenSSL security vulnerability" and half the internet catches on fire, it seems. Or Michael Jackson dies and the full internet catches on fire. (EDIT: Or an AWS region goes down and you make a drinking game out of what popular websites are down because of it.) Or, even ignoring mass issues like that - I suspect it's likely there have been "three major outages on the same day" before - and we simply didn't notice. I wasn't aware of the United Airlines outage, for example. I didn't notice this. I had to have it called to my attention. > and they all forgot how to use git revert? This also ignores the fact that git revert is insufficient for something as simple as an SQL schema change if you don't want to lose data. Rolling back your blog's comments is one thing - they were mostly spam…

Interesting. Five years prior, this story was posted on this blog: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

> Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late? Reminiscent of Knight Capital losing $440 million in 45 minutes via feature flags: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

The risk of feature flags is well illustrated by the downfall of Knight Capital Group: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

Absolutely, just like with software engineering missing the edge cases can drive you out of business. https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ I am not sure why people think that cloud is magically will shield them from every bad decision.

There's a large category of complex systems where you have to get it right in advance because of the consequences of problems; anything avionics or real-time, for example. Or you can lose a lot of money without blowing things up but still faster than the on-call humans can respond: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ (High-reliability engineering is very much a different, more expensive, less agile culture from software startups, and I worry that the culture is bleeding across in inappropriate ways. The "self-driving" car with a "safety driver" is an extreme example of this: an on-call human that's supposed to respond to operational problems in an extremely short timeframe, but also provides an opportunity to blame the human rather than the software)

I've worked on high frequency trading software infrastructure, including having been involved in serious bugs which lead to a lot of broken trades and lost money, and would have said with anything like proper architecture and governance you can't lose $400 million in 30 minutes. But Knight Capital did manage to lose $440 million in 45 minutes, due in large part to what I would call improper architecture and governance. So there you go. https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/a…

I am not a visionary like the people you mentioned, but one thing that I can see go wrong with large-scale AI automation is not that the scenario of robot uprising or something, but more of software bugs that we all are familiar with. There have been lots of cases where a bug in an automated trading system causes millions of losses[0][1], if we have a larger system that manages everything from the power grid, the gas lines, self-driving cars, your smart home, to your own personal calendar, it's not hard to imagine the potential damage if there is even a single bug. Can be things like edge cases that nobody thought of, or simply unexpected sensor reading caused by natural disaster, etc. [0](http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/) [1](http://www.bloomberg.com/news/articles/2013-08-20/goldman-says-exchang…

You should be patient and kind with everyone. Although, patient and kind does not included adding untested code to production to "fix" something late on a Sunday night. Insisting on proper deployment almost always makes developers[1] irritable. It also prevents http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ 1) I have more years as a developer than system admin (11 vs 7 and 5 as something I'm still not sure).

Indeed: http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

$400 million down the drain http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

"This couldn’t be further from the truth. The use of HTML5 + JavaScript in Windows 8 is about as proprietary as using XAML + .NET." http://dougseven.com/2011/10/24/language-choice-in-windows-8-is-not-ab… Just because it uses JS doesn't mean its portable! It has proprietary APIS + hooks making it non portable.

> I developed an financial servic application with 1MB+ source of code (not including 3rd party libs) within 3 month - thats more than 1 million keystrokes that created a system that works perfectly fine and produces exactly the results as specified How do you know it "works fine"? How do you know there aren't tons of caveats as soon as some boundary conditions are exceeded? Certainly not because you actually understand it all. Just yesterday we had "Engineers do not get to make startup mistakes when they build ledgers".[1] I sure hope it's just for your personal use because it would be completely irresponsible otherwise, to the point of criminal neglect. This is a Knightmare[2] waiting to happen and I can only hope it will only your own money that's effected and that you won't ruin anyone's lives with this. [1]: https://news.ycombinator.com/item?id=42269227 [2]: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/