Murat Demirbas

While reading Aleksey’s blog[0] linked from yours it struck me that this sounds a great deal like (real world) traffic engineering. A backup occurs, the traffic grinds to a halt, and after the blockage is removed it takes a very long time for the system to recover. If traffic is heavy enough, even just a small slowdown is enough to cause prolonged pain. [0]: http://charap.co/metastable-failures-in-distributed-systems/ (Having said that, my traffic engineering cluefulness is roughly at the “casual reader glancing at headlines” level, if that, but I found the comparison interesting.) I’m unfamiliar with HotOS, so I’m curious about the context behind this sentence: > But it's a HotOS paper, and this sure is a hot topic, so I won't hold the lack of a rigorous investigation of the background against the authors.

There’s also the part where the disaster recovery site apparently fell over under the load (which, OK, is a thing that might happen) and they needed to code up limits on the fly (and that is not OK; I don’t have the slightest idea how one might test this, but if you’re building a “disaster” site it seems like you’d need to figure it out): > When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. We implemented rate limits to get the request volume under control. This seems not to be mentioned in the bullet points at the end of the text (which are otherwise reasonable). And now I’m curious—how do you design cold failover when the system is complex enough to be metastable[1] and you can’t afford to test it on live traffic? I can guess which techniques you could use to build it, it’s the design and testing part (knowing the techniques actually work in your situation) that’s the problem. One other thing that seems to have gone completely unmentioned: > Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. [... W]e made the call at 13:40 UTC to fail over to Cloudflare's disaster recovery sites located in Europe. Why did the decision take so long? I can imagine it can’t be made lightly, but two hours seems like too much hesitation, even if there was an expectation that power would be restored imminently for most of…

> This isn’t a showstopper event, it’s a process stopper event. Again, the PAR paper describes the specific examples where a single disk sector failure on a single replica can be a showstopper event — for the whole cluster, even precluding reconfiguration since the log might no longer be functional. I know I've been hammering this point (single local disk sector failure = global cluster data loss or else global cluster unavailability), but it's really why PAR is essential for a distributed database, and why it won best paper at FAST '18. > Could you point me to the place in the paper? Sure, just grep the 2012 VSR Revisited paper throughout for "recovery protocol" to follow the discussion thread through the paper. It's one of the four main sub-protocols in the VSR paper. Here's the link to the paper itself: http://pmg.csail.mit.edu/papers/vr-revisited.pdf The protocol is section 4.3 but it's good to see the surrounding discussion in the paper where it's referenced. We also discussed this aspect of VSR in some extra detail at Aleksey Charapko's DistSys Reading Group recently: http://charap.co/reading-group-viewstamped-replication-revisited/

I think this paper is super important, and anybody who designs or runs big systems should read it and take the core point to heart. As system designers, we're very used to thinking about systems as 'stable' and 'unstable', where stability is good, and instability is bad. What this paper points out is that many kinds of distributed systems have multiple 'stable' modes, some of which are modes where the system is stable (in a control theory sense), but not doing any useful work from the client's perspective. This is dangerous, because the system won't kick itself out of this "stable but down" mode without something changing: human input, a control plane taking action, etc. I don't think this paper covers anything particularly new, but writing it down in this form, with the evidence they present, is very valuable. Hopefully this paper will deepen the conversation about applying control theory to distributed systems design and control problems, and allow a more theoretical approach to be taken to the design of these systems to avoid common causes of instability and bistability. One of the authors has a great summary of the paper on his blog: http://charap.co/metastable-failures-in-distributed-systems/ I wrote a summary and discussion too: https://brooker.co.za/blog/2021/05/24/metastable.html

> I don't care about consistency. Fair nuff, I asked because I was curious if you were assuming that eventual consistency was necessary for high availability. > Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second). Yup, looks like my concern above is the case. A single consensus replica set will do that no problem. I'd use raft if you're in a single geo location or epaxos otherwise (the epaxos source is a little early and needs eyeballs/production diversity testing, but the algorithm is absolutely solid). There's no reason to borrow all the troubles of CRDT's when it's not needed. Edit: Example from the epaxos paper, this is latency vs throughput for 3 replicas in one geo location: http://charap.co/wp-content/uploads/2015/10/latency_over_throughput.pn…