Charity Majors

Charity Majors (https://charity.wtf) is a great writer and speaker, and her work on observability is directly relevant to infra at scale.

Worth noting: Charity Majors' blog on management and tech: https://charity.wtf/ She says several things I've always thought were taboo to even think, including the fact that management shouldn't be viewed as a promotion, it's a completely separate job and some people are better at it than others. My limited experience: it's a separate job, it's a lot more things to juggle and carry, and it mostly makes sense that they get paid more. I have zero experience on director/c-level jobs, so I'm not going to speculate there.

I'm not sure about SRE specific or CI specific writing but at risk of answering another question, here's what comes to mind for me. In the Operations space my favorite writer is Charity Majors, although a lot of what she talks about isn't OPs specific: https://charity.wtf/ Rachel by the Bay is also great: https://rachelbythebay.com/w/ Totally not related, but Phil Eaton's posts on how databases work are super interesting and lack fluff. https://notes.eatonphil.com/ In terms of CI and builds, I try to write some on this topic for the Earthly Blog. My Bazel article was recently on Hn. Here are my posts: https://earthly.dev/blog/authors/adam/ Warning though, I also write about lots of other stuff. Are there specific CI topics you'd like to see covered?

You should definitely subscribe to Charity Major’s blog. Every post gets a “oh, of course” from me. https://charity.wtf/

If you like Julia Evans style you may also like Charity Majors aka @mipsytipsy on twitter https://charity.wtf/ Also I use the open source Django project Newsblur as my RSS reader, and follow Samuel's blog: http://blog.newsblur.com/ A couple others I like: https://watirmelon.blog https://martinfowler.com

The author: > Charity Majors is cofounder and CTO at Honeycomb.io, a platform that helps engineering teams debug and improve their software applications. > https://charity.wtf/ Blog, latest article an apologia for DEI, "The diversity of your teams over the long run rests on your ability to build an inclusive culture and equitable policies."... "Don’t underestimate what a competitive advantage diversity can be" Hmm. Why does no one argue for equitable balance of political views, or religions, or anything to do with ideas at all? Is anyone even measuring ideational diversity? Power-sharing in most countries is principally concerned with it, for example. How many 'inclusive' teams are extremely politically, and ideologically, homogenous? Clearly there's a apologia for a certain worldview at work here: some sunk-cost investment in a certain notion of inclusion, which is entails (if not aims for) moral and ideological uniformity. Offensive to this notion of 'inclusion' is excellence, since excellence isn't uniform nor even normal -- it's pareto distributed. An excellent runner is exponentially better than a normal one, and sampling uniformly from all groups ("inclusively") produces bad runners. According to experience, and all evidence, the best talent arises out of healthy competition and cooperation with the best talent -- it does not arise out of uniform, nor even, normal distributions. If you sample uniformly, then race all against all eventually a powerlaw arises --…

> I also wanted to cover what it's like coming from engineering management and founding companies to going back to being an individual contributor. (Spoiler: incredibly enjoyable.) I've done the IC to engineering manager back to IC thing and it is indeed a huge relief to learn that it's OK to do that. My favorite piece of writing on that is The Engineer/Manager Pendulum by Charity Majors: https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/ Charity makes a very convincing case that it's OK to swing from manager to IC and back again several times over the course of your career and that doing so will make you more effective at both of those things.

Do you think you might like management if you knew it didn’t have to be permanent? Highly recommend Charity Majors’ blog about this if you haven’t already https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/amp/

Structured/JSON saves a tonne of time building regex parsers. The regex parsing at query-time is also pretty expensive. This is where Splunk excels - dealing with the noise with powerful querying. ClickHouse is also very performant at this, we hear. It's an expensive task though (computationally and cost wise) I thought this was well put together from Better Stack: https://betterstack.com/community/guides/logging/logging-best-practice… Charity, CTO of Honeycomb has strong views (which we enjoy a lot): https://charity.wtf/tag/observability-2-0/ - they come at it from a tracing/OT angle which is Honeycomb's forte, but we agree a lot on the intended outcomes - actionable (not spammy/noisy) + make it easy to gather the variable/state context in the context of a single event.

Becoming a Manager by Linda Hill. Not a how-to, but it gives you a preview of the challenge and the blind spots you might have as you become a manager. Another that I haven't read but has been recommended to me is The Manager's Path. This article is also good: https://charity.wtf/2019/01/04/engineering-management-the-pendulum-or-…

Worth mentioning that management vs. IC is not a decision for life. Charity Majors wrote about the opportunities and challenges when switching throughout one's career between management and individual contributor [1] and [2]. [1] https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/ [2] https://charity.wtf/2022/03/24/twin-anxieties-of-the-engineer-manager-…

https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/ is a great read on this.

Personally, I would recommend listening to Thoughtworks' Techonology Podcast. Their approach in general to running in the cloud is a pretty good one, which I'd characterize as platform engineering-centric. Anything Will Larsen writes is bound to be worth looking at and thinking about in the context of scaling cloud operations and operational complexity. His concept of the 'pierceable abstraction' is _absolutely key_ in my opinion. Additionally, the following blog posts are fairly good introductions: https://charity.wtf/2018/10/24/ten-platform-commandments/ https://martinfowler.com/articles/talk-about-platforms.html https://lethain.com/pierceable-abstractions/ https://srvaroa.github.io/paas/infrastructure/platform/kubernetes/clou… Hope that helps - had to dash this off fairly quickly.

This might be a good read for you: https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/

Seniority is in the eyes of the beholder. But what you say makes little sense. You think you're not senior enough because you can't debug piece of garbage? > I've also thought about transitioning to a PM role Maybe you're just tired of working with code. That's alright after 10 years. And it's ok to want to try management. I can recommend reading this https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/

As the last piece of most architectures to see every write request (caches will see every request, including reads), web servers are the logical place to capture system wide data. (Caches, if configurable, may be a slightly better option.) I haven't done a lot of nginx config, but appreciated the pointers in this post. On the other hand, it was a bit short and no pointer to the next installment. By the way, if you haven't read the blog of Honeycomb founder, it's full of interesting provocative opinions: https://charity.wtf/

Great article on that here: https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/

This is great, and solves two ongoing problems we have with TF: - The new 'State environment' feature should resolve the issues discussed here: https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstat… - The new locking feature means we don't need to use https://github.com/gruntwork-io/terragrunt

Definitely hear you on that, and we've taken steps to mitigate Terraform's "surface area" after we encountered some problematic use cases. This excellent article [1] does a nice job of talking about why it's important to keep your tfstate files small and isolated. Since we started doing that, working with Terraform has been much nicer (and safer!). [1] https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstat…

the second part was more interesting https://charity.wtf/2016/05/31/operational-best-practices-serverless/

I've read a lot of Charity Majors' stuff, and in some ways this opinion is typical of her, and in some ways it is not. Obviously, she's a big proponent of engineers taking full ownership of their work. That's core to observability, which is what her day job is all about: https://charity.wtf/2020/03/03/observability-is-a-many-splendored-thin… Yet one of the pieces she's well known for is "The Engineer/Manager Pendulum," where she argues that some of the best managers and some of the best ICs are people who have gone back and forth between the two roles: https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/ In one sense, that is consistent with her opinion about architects being a BS position, sort of analogous to calling out useless middle managers. But one of my takeaways from her pendulum opinion is that those senior ICs who have pendulumed are better off precisely because they bring a management skillset to IC work and an IC skillset to management work. And a high enough level, that manager's eye might involve recognizing that helping other engineers design their systems is imparting more engineering influence than whatever any one person can do at the code level. And in these cases, I'd argue you're still delivering engineering value rather than just project-management value. Charity knows a lot of these senior ICs, and so it's not as if she's dismissing their potential value out of ignorance. So I'm a little surprised that she's throwing the baby out…

I've experienced this my entire professional life. I think this is the cause: https://charity.wtf/2020/09/06/if-management-isnt-a-promotion-then-eng… tl;dr: Management should be a supporting role, but because it's closer to the business (and usually staffed by "Idealists" as described here: https://leanpub.com/developerhegemony), it's given a more powerful position, and does whatever it can to execute as needed by the business. Of course everyone (managers included) also want to keep the power they have and gain more power/money over time. Developers' time isn't respected because managers have more power and building software is hard, so rather than building partnerships and supporting everyone as needed, then just squeeze harder, treat people lower on the org chart as lower worth as humans, and generally don't shield or abstract things away as much as they should (often not at all). From the article: > The goal of most software projects is to deliver to deadlines and budget of the plan. The real goal of a software project is to create the right software, but instead decisions focus on the meeting the plan not creating software. The plan is focused on instead of the software because the plan is the artifact of the more important people, the managers. They're higher up on the ladder because they're more loyal to the company. Even when this isn't the case they're still higher up because it's safer to have them in power rather than devs who can…

There are companies out there who value the experience of senior software engineers (and I mean real seniority, not the job title that you update to senior after 3 years in the field). Such companies will have a career ladder up to Principal or Distinguished engineer. You will have to share your knowledge, of course, but without having to give up coding and do people management, if you don't want that. I'm 39, happy infrastructure engineer, individual contributor. I tried being team lead for some years, but didn't really enjoy people management. That's okay. Charity Majors wrote some great posts about that: - https://charity.wtf/2019/09/08/reasons-not-to-be-a-manager/ - https://charity.wtf/2019/11/23/questionable-advice-after-being-a-manag… - https://charity.wtf/2020/09/01/the-official-authorized-list-of-legitim… - https://charity.wtf/2020/09/06/if-management-isnt-a-promotion-then-eng…

Great comment! My corollary would be that "Many managers have never had a great role model". In my first role as an engineering manager I thought it was the logical career progression and promotion from lead, but my preparation was more along the lines of "I've had managers, I can do what I think they do". I was very fortunate that my company at the time was supportive with follow-up training and building a peer group of newly minted managers, but it would have been more ideal to have some prior leadership & management training so I was more aware of the role and responsibilities I was signing up for. I think training provides helpful insight into intentional motions and being a competent manager, but it is hard to beat experience and learning from some great role models. One secret I would tell any engineers considering management: "management is a career change, not a promotion". Charity Majors has some excellent blog posts on this, including the Engineer/Manager Pendulum: https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/.

Charity Majors https://charity.wtf/

Dan Luu has a list of programming blogs you might like: https://danluu.com/programming-blogs/. The rest of his blog is good too! a few of my favorite blogs: - http://blog.acolyer.org for fantastic daily summaries of CS papers. - http://stephaniehurlburt.com/blog/ -- she has a business creating a new compression algorithm and I love reading about it - https://rachelbythebay.com/w/ is pure gold for weird debugging stories - https://accidentallyquadratic.tumblr.com/ is always a fun read - http://wingolog.org/ on building compilers - http://composition.al/blog -- Lindsey Kuper on her programming languages research - aphyr's blog on distributed systems, of course - https://charity.wtf/ - http://www.pgbovine.net/writings.htm -- Philip Guo is a CS professor whose blog on his experiences in academia I really like - http://whilefalse.blogspot.com by Camille Fournier, mostly on engineering management - http://larahogan.me/blog/ by Lara Hogan, on engineering management Also I think this comment from Dan's blog (https://danluu.com/about/) is very true and important: > I view that as a sign there’s a desperate shortage of understandable explanation of technical topics. There’s nothing here that most of my co-workers don’t know (with the exception of maybe three or four posts where I propose novel ideas). It’s just that they don’t blog and I do. I’m not going to try to convince you to start writing a blog, since that has to be something you want to do, but I will point out…

This is excellent, I'm going to use this to separate production from test environments using separate accounts. Using VPCs, security groups, IAMs, tags, etc. is great, but those are all constructs you have to take care to implement correctly yourself within your deploy scripts. One slip-up and you could accidentally expose environments to each other, or worse make a mistake that deletes or reconfigures resources in an environment you didn't intend to. Or an errant process in a test environment could unexpectedly gobble up too many resources, resulting in production being throttled. So separate accounts is the safest isolation approach, but historically there have been drawbacks (this blog post discusses some[1]), like billing is more expensive (due how AWS charges for usage and you won't be sharing reserved instances anymore) and I wasn't sure how to handle things that need to be cross-environment, like Route53 (DNS). Looking forward to trying this out! [1]: https://charity.wtf/2016/03/23/aws-networking-environments-and-you/

Also (1 hour ago) https://news.ycombinator.com/item?id=46673948 And https://charity.wtf/2026/01/19/bring-back-ops-pride-xpost/

I know this is a tongue in cheek casual comment, but this article is a really good and important counterpoint: https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-…

Here’s some good food for thought on the engineer/manager decision, [0] and [1]. Good luck! [0] https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/ [1] https://charity.wtf/2019/01/04/engineering-management-the-pendulum-or-…