Marc Brooker

It turns out that best of 2 random draws outperforms best of 1, best of 3, and best of all in many load balancing scenarios: https://brooker.co.za/blog/2012/01/17/two-random.html It's a bit unintuitive, but they key idea is roughly 'If you're working on stale load data (as always), best of 2 strikes the right balance between distributing load evenly and giving more work to less loaded hosts'. If you do 'best of k', you end up with herd behavior, overloading one host. 'best of 1' sends too much traffic to slow hosts.

Try giving Marc Brooker’s blog on this a read: https://brooker.co.za/blog/2012/01/17/two-random.html It is only better than leastconn when you have stale information to base your decision on. If you have perfect, live information, best will always be optimal.

This is great, really worth reading if you're interested in transactions. I liked it so much I wrote up how the model applies to Amazon Aurora DSQL at https://brooker.co.za/blog/2025/04/17/decomposing.html It's interesting because of DSQL's distributed nature, and the decoupling between durability and application to storage in our architecture.

> durability is a spectrum This is the truth. My favourite example of this is MemoryDB's "multi-AZ (multi-datacenter) durability" for Valkey - there's a good write-up here https://brooker.co.za/blog/2024/04/25/memorydb.html

My understanding is that cold starts on containerized Lambdas is actually better than non-containerized for some workloads, because using containers allows Lambda to do better caching of the code, as well as lazy-loading. YMMV of course based on exactly what image you use (eg if you're not using a common base, like Ubuntu or Amazon Linux, you won't get as much benefit from the caching) and how much custom code you have (like hundreds of MBs worth). There's a very interesting blog post about it here, as well an an accompanying whitepaper: https://brooker.co.za/blog/2023/05/23/snapshot-loading.html

Really great research work from Kay Ousterhout. Sparrow extends the "power of two choices" [1] to a more general formulation, and makes it really cheap to find very good choices (i.e., low-load servers) to run a task, without having to check the load on every server. As the number of tasks and number of servers grow, keeping an accurate picture of the load on every server makes the scheduler become a bottleneck. Kay's work showed a very interesting point in this design space for fast and high-quality scheduling. Her PhD work went on to really understand the performance of complex distributed programs, like large Spark jobs, and a lot of at least earlier performance instrumentation in Spark is due to her.[2] [1] https://brooker.co.za/blog/2012/01/17/two-random.html [2] http://kayousterhout.org/

https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf and > Best of 2 is good because it combines the best of both worlds: it uses real information about load to pick a host (unlike random), but rejects herd behavior much more strongly than the other two approaches. https://brooker.co.za/blog/2012/01/17/two-random.html

Awesome. Looking forward to additional content. If possible, would be great to get Mark Brooker (Principal at AWS) to provide some notes on bridging the gap between CAP theorem and how AWS relaxes constraints for building out Elastic Block Storage (EBS). https://brooker.co.za/blog/2014/07/16/pacelc.html

If you're interested in building distributed systems, then I thoroughly recommend Marc Brooker's blog: https://brooker.co.za/blog/

Mostly a reminder/clarification of things I knew, but a good and welcome one well-stated, because I probably sometimes forget. (I don't do performance work a lot). But this: > If you must use latency to measure efficiency, use mean (avg) latency. Yes, average latency Not sure if I ever thought about it before, but after following the link[1] where OP talks more about it, they've convinced me. Definitely want mean latency at least in addition to median, not median alone. [1]: https://brooker.co.za/blog/2017/12/28/mean.html

Here [1] is a good article on the "random pick 2" idea in case anyone is interested. The paper is definitely worth reading too. It's definitely one of my favorite techniques that I keep in my "toolbox". Really simple, but not (at least to me) intuitive. [1] https://brooker.co.za/blog/2012/01/17/two-random.html

Readers would also be interested in this excellent Raft consensus algorithm visualization : http://thesecretlivesofdata.com/raft/ As well as : "Ice cream & distributed systems" : https://brooker.co.za/blog/2014/10/25/ice-cream.html

Another great read on this subject "Exactly-Once Delivery May Not Be What You Want" https://brooker.co.za/blog/2014/11/15/exactly-once.html

I think framing it as a site for people that are positive about AI is probably the wrong way to go, but around it kf you don't want just a bunch of AI psychosis posts and nothing else. I think it would be better if, instead of being defined in terms of being "positive" about AI (in the sense of saying that everything to do with AI is all great and rosy and awesome, and thinking every new AI thing is the next big thing, and praising every vibe-coded piece of slop), the focus was explicitly on having mature, software engineering-focused discussions of how to actually understand, make, and use AI and AI tools/harnesses, where it's taken as common ground that these are tools, not magical genies that will usher in the rapture of the nerds or make software development obsolete, nor uniquely evil technologies that only fascists could ever use or like or that are completely useless, or tools that only talentless hacks who don't care about the craft, good use. So basically to avoid shallow dismissals and endless culture of war arguments. I would love to have a place that discusses the pros and cons of different ways of approaching specification, natural language programming, DSLs, property-based testing, behavior-driven development, lightweight formal methods like model checking, or even more heavyweight formal methods like doing proofs, etc, as means of using the extra time and lowered cognitive load allowed by AI coding agents to actually increase the reliability and improve the…

Exactly this. DynamoDB is a pain in the ass if you want to do too many relational or arbitrary queries. It's not for data exploration. It is my favourite database though (next to S3)! For cases where my queries are pretty much known upfront, and I want predictable great performance. As Marc Brooker wrote in [1], "DynamoDB’s Best Feature: Predictability". I consistently get single digit millisecond GETs, 10-15ms PUTs, and a few more milliseconds for TransactWriteItems. Are you able to complex joins? No. Are you able to do queries based on different hash/sort keys easily? Not without adding GSIs or a new table. The issue in the past few years was the whole craze around "single-table design". Folks took it literally as having to shove all their data in a single table, instead of understanding the reason and the cases that worked well. And with ongoing improvements of DynamoDB those cases were getting fewer and fewer over time. But, that's what tradeoffs are about. With on-demand tables, one-shot transactions, actually serverless storage/scaling, and predictable performance you get very, very far. 1. https://brooker.co.za/blog/2022/01/19/predictability.html

Its exactly why PC/console gamers (generally, not talking about GP) make terrible customers and Apple is right to not play with fire by courting them too closely - they're loud, cheap, immature, mercurial, and demanding, and being associated with them is probably a net negative for brand. Let them stew in forums arguing over red vs green, tinkering with and breaking the warranty of their PC parts, being disloyal to the brand they loved 5 minutes ago, etc. Better strategy is to make sure the door isn't closed on gaming for those that want to use their expensive Macs to occasionally play (protect the downside), rather than swing the door open enthusiastically for gamers to rush in. Re: worrying about fun vs tinkering, I'm reminded of the 4 quadrants of hobbies. What we see on forums are generally gamers interested in gear & discussing, not 'doing the hobby'. https://brooker.co.za/blog/2023/04/20/hobbies.html

> Also, I see you amended your original comment with a jab against me for the type keys post. I didn't realize you were the same person who wrote that blog post, and I edited it to include it when I realized. I linked your amended version, not your initial version if that's of any worth. And you're correct it's not closed. I still stand by that particular proposal is going nowhere. You can make any proposal you want. It hasn't even moved to the design stage yet in 3 years, so I'm not sure why it's something you hang on to as a signal error handling is changing. Take generics for example, and the years (decade) of work that it took. This is unsolicited blog feedback: Your blog and writing style doesn't read like it comes from a place of humility and learning, but from a place of authority. Sometimes that's OK, but in your case, feels unwarranted. And the constant push to get it on to HN or Reddit or N other platforms so it can spread... Thought leadership as an aspirational goal has never been something I look for in blogs I read (I recoil and go elsewhere when I sense this is the goal). I understand it's hard work, and you want others to see it, but just someone else's perspective. Some blogs that I really enjoy (and I hope others emulate so I'm sharing): - http://neugierig.org/software/blog/ - https://brooker.co.za/blog/

I've had that experience on multiple occasions. Initially, when younger, I did attribute it to myopism. Today, I recognize that credibility also needs to be earned. All architects are unhappy except those who manage to find the right client. The client-architect relationship is fundamental. A successful architectural undertaking is minimally a two variable equation. Client (aka stakeholders) are hugely important. (this is related imo to your post: https://brooker.co.za/blog/2020/10/19/big-changes.html - ) On a related tangent, I've been re-reading "The Open Hand - Essays on Le Corbusier". Specifically, the "Le Corbusier at Pessac - Professional and Client Responsibilities" essay by Brian Brace Taylor. https://www.worldcat.org/title/open-hand-essays-on-le-corbusier/oclc/3… Pessac: https://www.archdaily.com/940878/le-corbusiers-cite-fruges-lessons-fro… That project basically failed its intended purpose. The client, M. Frugès, was apparently an enlightened capitalist, willing to invest in the vision of the young architect, even when incurring financial losses, because of Pessac. Corbu also learned, on the job, how to, and how not to, build his modernist vision. The whole thing was a fiasco on many levels. But it was a necessary project for progress in the field. Which is the point: architectural innovation is costly and even great architects leave a trail of failed early projects. Yes, Corbu went on to master his field, and he did…

Thanks Marc, I'm glad, you too! Ever since I came across https://brooker.co.za/blog/2018/01/01/balls-into-bins.html, I've been really excited to see and follow all the fantastic design work you've been doing. >> VR's view change protocol for leader election is also entirely in-memory so it's far more fault tolerant compared to Raft if you have a storage fault model and not only a network fault model. > Only in the fail-stop model, right? Or does this property extend to other models (like omissions)? You're far more knowledgeable about the domain... but I was thinking that with VRR's completely in-memory view change and replication protocol, this would then extend past the fail-stop model to include byzantine disk storage faults (misdirected reads/writes, corruption, latent sector errors) since VRR requires no guarantees from the disk in order for the consensus protocol to be correct, whereas Raft does. To me this is just another reason that makes VRR such a fantastic protocol. To be fair, I guess we could say that Raft assumes a fail-stop disk model, but then again Raft is supposed to be a practical implementation and disks are not fail-stop in reality. I'm sure you're also well familiar with WISC's Protocol-Aware Recovery for Consensus-Based Storage: https://www.usenix.org/system/files/conference/fast18/fast18-alagappan… Beyond that, I've also been wondering about this from the perspective of taking VRR's in-memory view change for the leader election phase, but…

> this does not mean that a general algorithm has to be slow when run on typical instances True, but we have a few decades of experience with model checkers running on real programs. Yes, they work very well and are quite affordable, but their domain is rather specific (there is amazing work on model checkers designed to expand their domain and work on ever-larger programs, but the results aren't in yet). > Unless you're generating programs randomly, programming generally entails carrying at least an informal proof of correctness in mind. There are various results showing that the difference in size between an informal proof (even when correct, which, as experience shows often isn't[1]) and a formal one may be exponential. > We need better tooling, more clever proof-finding algorithm to bridge the gaps, clearer interfaces. Absolutely, but we must understand there are fundamental problems in play. > Importantly, it is not a computational barrier. There is nothing about the formal verification of typical software that intrinsically requires expensive computation. This is unknown, but not very promising; certainly not promising enough to assume this will definitely be possible some day (let alone soon). See the result about symbolic model checking (basically the cost of proof that is allowed to exploit linguistic constructs) that I mention in my post. It turns out that in the general case (this time, we're not talking about completely "random" programs, but "random"…

https://brooker.co.za/blog/2022/02/28/retries.html Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful

part of it https://muratbuffalo.blogspot.com https://www.allthingsdistributed.com/ https://brooker.co.za/blog/ https://surfingcomplexity.blog/

Thanks for the article. I have two questions/comments: 1. The N-ring fade-in is quite neat. I guess without the constraint of hash parity rendezvous hashing [0] could have been an elegant approach since it has support for weights (and generally better statistical properties than consistent hashing based on rings). 2. You mention still having the fallback of your existing load balancer. Is this a temporary thing during rollout or do you intend to keep it long-term. Asking because I generally tend to stay clear of fallbacks in distributed systems, as they introduce bimodality and metastable failures [1] [2]. [0] https://en.wikipedia.org/wiki/Rendezvous_hashing [1] https://builder.aws.com/content/3EuS9Sakq7L3VLQIF3qzfMfke1Y/avoiding-f… [2] https://brooker.co.za/blog/2021/05/24/metastable.html

There was a post on here the other day explaining this exact phenomenon: https://brooker.co.za/blog/2026/06/19/waiting.html

Was reminded of this article when I read what you said: https://brooker.co.za/blog/2026/06/19/waiting.html I have an ancient Windows 98 computer which I used to use at my grandparents' home until a few years ago. All the problems you mentioned did exist. However, the machine still felt more responsive than today's laptops. Something being objectively slow is very different from it feeling slow when you use it. As an example, I never had Word autosave turned on. It was always explicit (even if this meant I'd usually lose what I was working on when the computer lost power)

Completely agree that it's off-putting. The author indeed has only ever worked in academia per his LinkedIn. But disagree that this is a path to unemployment. At work we go very fast and yet I think fast is compatible with each of those points, just not in all situations. Marc Brooker, distinguished eng at AWS, gives much more useful advice for industry, as you'd expect given his almost 30 years in industry. https://brooker.co.za/blog/2026/03/25/ic-junior.html

It's the four hobbies all over again: https://brooker.co.za/blog/2023/04/20/hobbies.html

Very cool work! Did you investigate using the Power of two random choices method for your load balancing algorithm ? https://brooker.co.za/blog/2012/01/17/two-random.html https://medium.com/the-intuition-project/load-balancing-the-intuition-…

DSQL only uses Postgres for the query processor layer, so it doesn't require a replication library within postgres itself. Definitely NOT from DSQL. > We’re not using any of the storage or transaction processing parts of PostgreSQL, but are using the SQL engine, an adapted version of the planner and optimizer, and the client protocol implementation. [1] Rather, DSQL seems to do its region replication using the distributed journal abstraction [2]. [1] https://brooker.co.za/blog/2024/12/04/inside-dsql.html [2] https://brooker.co.za/blog/2024/12/06/inside-dsql-cap.html

There is backlog, and then there is engaging in the four quadrants of hobby interaction[1]. [1] https://brooker.co.za/blog/2023/04/20/hobbies.html