Tom Limoncelli

The path of a lot of system administrators from amateur to professional resembles that of a child who is exceptionally gifted at building erector sets being hired to construct a pedestrian bridge. Then, if the bridge doesn’t fall, the kid gets to build bridges designed to handle interstate traffic. This is the key point. It's SO HARD to grow as a sysadmin, because there are very few people who can pass on information. We have pretty much one textbook (TPoSNA: http://everythingsysadmin.com/ ), and a few organizations like LOPSA and USENIX, but there are almost no Windows or Mac specific sysadmins who join. There are also places where regulation is already happening, albeit in a top down manner, like HIPPA. I doubt top down "you must do this" regulation would help, but accreditation where people have to know and ought to follow best practices would likely do more good.

It would be helpful if you could tell me where you got the impression that I was encouraging use on cloud-storage services. Was it in the README? the code? a post I made on my blog? I ask because I'd like to correct that doc/post/etc. so that others don't get the same impression. Tom Limoncelli http://the-cloud-book.com http://everythingsysadmin.com P.S. Blackbox was written to safely store the occasional secret file (SSL certs, mostly) in a Puppet repo stored on an in-company source code server. Even though I personally knew everyone with access to the repo, root access to the repo server, or access to where the backup tapes are stored... it became essential to encrypt those secrets. Honest, I don't trust the people who handle our backup tapes... one of them is me and I'm the least trustworthy person I know. Even more importantly, now that I could open up the repo to all my coworkers, it enabled me to collaborate across the company. They could read my code, even submit pull requests, and I didn't have to worry that they had access to SSL certs.

I feel like engineering is used in the very broad sense, but books/blogs by Tom Limoncelli (et al) helped me a lot in the past in better planning and structuring the systems I worked with. https://everythingsysadmin.com/

Much better option: puppet, chef, ansible, or any other configuration management / automated systems administration tool. Stop logging in to boxes to change them. Instead, make a change in your version-controlled config repository, get a machine to test it, and if it works, roll it out to all the machines with that role. Or do half of them, and wait for things to settle, and then do the other half. Lots of options. You might want to read The Practice of System Administration -- http://everythingsysadmin.com/

For Python my coworker (and friend) Tom has a great project called timetravelpdb: http://everythingsysadmin.com/2014/04/time-travel-pdb.html explains a bit on it, the repo is https://github.com/TomOnTime/timetravelpdb Basically it's an extension of pdb that allows you to jump back and forth in time.

On the topic of learning ops, I want to second my recommendation from another thread for "The Practice of System and Network Administration" and "The Practice of Cloud System Administration" These books focus on principles, rather than specific technologies, and I love them for it. http://everythingsysadmin.com/books.html

That system is rather similar to “The Cycle System”, as recommended by the book Time Management for System Administrators¹, (also a series of lectures²) by Thomas A. Limoncelli, except the Bullet Journal uses months, where The Cycle System uses days. 1) http://everythingsysadmin.com/books.html 2) http://www.tomontime.com/

That does look interesting but it might be too philosophical for many hackers. I would instead recommend Time Management for Sysadmins, Tim Lemoncelli, O'Reilly and Pragmatic Thinking and Learning, Andy Hunt, PragProg. Both are interesting and helpful but much more practical, especially TMSA, which I should dust off and re-read. http://everythingsysadmin.com/books.html http://pragprog.com/book/ahptl/pragmatic-thinking-and-learning

I think Stack Exchange has a secret weapon which will likely greatly improve their backend systems. Tom Limoncelli, who used to work at Google as a Site reliability engineer (SRE), now works at Stack Exchange [1]. He pretty much wrote the bible for sysadmins entitle "The Practice of System and Network Administration" [2]. I wouldn't be surprised if we start seeing more posts like this! [1] http://everythingsysadmin.com/2013/09/the-team-im-on-at-stack-exchan.h… [2] http://www.amazon.com/dp/0321492668/tomontime-20

The first quote is difficult to describe for me, but let's call it "poetic", as my brain is too fried to think of a better term for it. Basically, my interpretation of what he's saying is: if your bakery isn't tech-savvy, even at a basic level, they could be getting killed by reviewers (or some other bogeyman) online without knowing it. The second quote is pretty easy to support with sources. Google has three _official_ languages that can be deployed into production, and Python is one of them. See [this post from a Googler](http://everythingsysadmin.com/2011/01/python-is-better-than-perl6.html… for instance. I have no idea how hard a rule this is, but many a Googler has mentioned it. I think it's worthy of being called a 'fact', and thus completely valid in the article in question.

One thing that will be very important: XmlHttpRequest. I recommend this post, it helped me to understand the basics: http://everythingsysadmin.com/2010/06/learning-ajax-after-ignoring-i.h…

As a sysadmin who wishes he'd discovered this book earlier, I have no problem with recommending "The Practice of System and Network Administration" as a great resource: http://everythingsysadmin.com/aboutbook.html It's definitely an OS-agnostic book, so if you're looking for "how do we do task/process/design A with OS B?", it's not the right fit. It'd still be worth it if you're expecting to have to do anything except the very shortest-term systems administration.

I agree that JSON isn't a very good syntax for humans to produce. But I think Tom Limoncelli has already written most that needs to be said: http://everythingsysadmin.com/2015/06/toml-vs-json.html I prefer yaml for deep nesting over toml, though. But whatever the config syntax, comment-preserving parsing would be very nice for config files.

Would this list of books be a good starting point? http://everythingsysadmin.com/books.html

I can't improve on this: http://everythingsysadmin.com/2011/06/avoid-using-the-term-cloud-com.h…

I'm a programmer, but believe it's important to be well-rounded. To balance my programming skills with sysadmin skills, I've started reading "The Practice of System and Network Administration": http://everythingsysadmin.com/aboutbook.html http://www.amazon.com/Practice-System-Network-Administration-Second/dp… The book was praised by multiple HN commenters ( http://news.ycombinator.com/item?id=1665915 for example), and I'm glad I listened to their advice. I've only read a few chapters so far (it's a big book!), but I already like it. It contains a lot of good insights, and I like that it's not focusing on a specific OS / platform. Also, you don't have to read it in a linear fashion, you can pick and choose chapters depending on your current needs (moving into a new data center? implementing a security policy?). Thumbs up.

Offtopic question: this article links to http://everythingsysadmin.com/2011/01/python-is-better-than-perl6.html where author writes that google have official UI language. Anyone knows what is the name of that language? Is it something like XUL but for web?

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system. I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced < 3 months before the outage. [1] http://everythingsysadmin.com/2012/09/devops-google-reveals-their-di.h…

Yes, very constructive. A more detailed reply: 3) really? the core of devops is to be data-driven in your decisions. how can you decide if you are maintaining the right uptime if you don't measure it? 8) I agree with you, as does the text. I think you may have reversed what I wrote. 9) The good ones write so that they "think before they do". On a larger team it is important to communicate what you are about to do, or what you have done. I prefer to write mini design docs. The team I'm on does this and I like it so much I want to spread the word. 11: Again, the team I'm on does this and it works so well I want to spread the word. I see I need to expand this out to explain why, not just how. 14: I'll clarify that the point is not to make big changes on your production system. Whether it is qa+live or a zillion steps including dev, qa, UAT, pre-prod, canary and prod. As long as it isn't zero steps. 22: refresh policy: This is for PCs (non-servers). I'll clarify. 23: The last part makes your point. I'll rewrite to make it more evident 28: "Anti-malware? Really? In 2011? I'm sure you have a whole blog post". Yes, I do: http://everythingsysadmin.com/2011/04/apt.html Thanks for the reminder to add a link! (and if you are blown away that I had to list this, you can imagine my surprise about finding sites that violated this one!)

"There is a new TLD called .feedback. It is a scam and ICANN should be ashamed of approving it. The people that run .feedback have pre-registered 'for free' 5,000 major companies. As a result you can go to sites like http://www.google.feedback/ and http://www.gandi.feedback/ and http://www.stackoverflow.feedback/ and more." From: https://everythingsysadmin.com/2017/06/the-feedback-scam.html Previously on HN: https://news.ycombinator.com/item?id=14662107