SysAdvent

Looks like the Sysadvent has started again today, after taking last year off. Its great for those of us on the more OPS side of DEVOPS: https://sysadvent.blogspot.com/

My favourites are: - https://blog.acolyer.org/ - http://sysadvent.blogspot.com/

Some aspects of zero downtime DB migrations require your changes and deployment protocols to follow certain rules, so I'm not sure a generic automated solution is possible. I was writing and speaking a fair amount about this back in 2012 and the concepts are still basically the same today -- I'll outline the core ideas here and provide a few links with more configuration examples, etc. at the end. The basic idea behind zero downtime DB migrations is to use a Blue-Green deployment model. Blue-Green deployment uses two identical production environments, called "Blue" and "Green", with a router or proxy setup in front of them. Only one of the environments serves traffic, while the other is idle. For our example, let's say Blue is initially live. When you deploy new application code, it would be deployed to Green and tested. Once you're happy with it, you switch the router or proxy to send requests to Green. Blue becomes idle, and then you update Blue. With MySQL as the database, I like to call this setup "Active-Passive", using a Master-Master infrastructure. Normally, Master-Master implies an "Active-Active" setup with each instance being able accept reads and writes. (People that have used Active-Active Master-Master MySQL recognize the big downside: conflicting writes break replication.) Using an Active-Passive approach for supporting zero downtime schema changes lets you avoid the problem of conflicting writes break replication. Basically, think of Active-Passive as…

Config data accessible via HTTP instead of files. Although its not limited to config data necessarily. I suggest reading http://sysadvent.blogspot.com/2013/12/day-20-distributed-configuration…

This is similar to the Sysadvent 2013 post by TR Jordan on using an `X-Trace` header to track users across your stack: http://sysadvent.blogspot.com/2013/12/day-5-gentle-introduction-to-x-t…

"Have you run tcpdump or strace?" If you're curious about where to start debugging, this blog post gives a good rundown: http://sysadvent.blogspot.com/2008/12/sysadmin-advent-day-1.html

A few things come to mind along with what's already been mentioned: Assorted Non-Book Readings: - https://randsinrepose.com/dont-skip-this/ - http://www.stilldrinking.org/programming-sucks - https://sysadvent.blogspot.com/2019/12/day-21-being-kind-to-3am-you.ht… - https://donellameadows.org/archives/leverage-points-places-to-interven… - https://grugbrain.dev/ - https://changelog.com/posts/rich-hickeys-greatest-hits - https://github.com/papers-we-love/papers-we-love/blob/master/design/ou… - http://www.cs.unc.edu/techreports/86-020.pdf Books: - The Phoenix Project: Gene Kim, Kevin Behr, George Spafford - Deep Work: Cal Newport - The Art of Leadership: Michael Lopp (Rands) - Extreme Ownership: Jocko Willink and Leif Babin (once you get through the ho-rah stuff it's a fantastic read) - Time Management for System Administrators: O'Reilly (fantastic for interrupt driven work) - Elements of Clojure: Zachary Tellman (not just for Clojure devs)

With servers of even the low end of the middle, there should always be redundant system disks. The next step up in servers from what you have should use RAID-1 mirrored disks. Availability is a sliding scale in terms of reliability and money. You don't pay for stopping things that will happen, you pay to stop things that /could/ happen. The additional cost of a real RAID card plus an additional disk is insurance against the single drive failure that would make your machine unavailable. Your redundant power supplies do the same thing. As do your redundant NICs. Disaster planning is the art of asking, "What if..." and having answers. Of course, for every answer, a certain amount of money is required. My advice to you is to either go to the cloud now, before you invest more money in a physical infrastructure, or if thats not possible, buy "real" hardware and don't use what are essentially glorified laptops in a server case. Get a server that is rackmountable, has redundant hard drives (and if the server has system disks and data disks, configure it so that the system disks are a mirrored RAID-1 pair, and the data disks are not RAID-5), that has redundant power supplies, and redundant network cards. It really does sound expensive when you list it out like that, but those features are so standard that you can't really avoid them if you buy middle-class or higher servers. Here's a guest blog post I did on redundancy last year for SysAdvent:…