Jepsen

If you enjoy this essay, you might also enjoy another classic, "Typing the technical interview"[0] by Aphyr (a.k.a. Kyle Kingsbury, the person behind Jepsen[1] distributed systems tests.) [0] https://aphyr.com/posts/342-typing-the-technical-interview [1] https://jepsen.io/

It isn't easy, but all of this can (and should) be tested. It will produce far better and more reliable code. You can certainly run tests 151 times. In fact, if you think you're going to find a bug after that many runs, a test suite is perfect for automating the execution. Don't believe me? Jespen is a great example of difficult distributed testing done well: https://jepsen.io/

EDIT: Title has been slightly changed, now. It was originally "Jepsen testing of rqlite, the distributed DB built on Raft and SQLite" Hmm, perhaps a bit of confusion from the title. It sounds like they ran the Jepsen suite of tests against rqlite, which is great, but not done _by_ Jepsen / Kyle (https://jepsen.io). Others have done this themselves too and that's fine, but half the of the problem is correctly implementing the tests which has been done incorrectly by others in the past.

The comments in this list seem from really smart people. I used to follow Jepsen https://jepsen.io/ closely. Those tests are quite comprehensive. But I think even those are not sufficient to quantify the quality of Paxos/Raft. Tools like TLA+ probably require a few months to learn and become proficient.

Oh, yeah, that's something I could put together. In the meantime, there's a mailing list and twitter you can subscribe to! https://jepsen.io

We need more tools like this. Following the Jepsen tests (http://jepsen.io) its obvious that very few systems take network partitions and non-optimal network conditions into account.

If the recipient fails after telling the deduping machine what it has seen, the recipient's failover will be in an unknown different state. And now the failover machine has to try to figure out what its state should be, what is going on, and so on. You can add ways to addressing each possibility here, and you'll get ever more obscure chains that again can result in failure. At the cost of ever increasing complexity, you'll make failures ever more complicated. Frequently developers who implement these manage to do two bad things. First they introduced a lot of complex code which rarely triggers except in disaster, and so whose bugs tend to survive. And second, they manage to convince themselves that they have accomplished the impossible, and make reliability promises that other developers unwisely believe. Exactly how unreliable most systems were and how much the documentation couldn't be trusted was underappreciated until https://jepsen.io/ came along and started proving how bad most distributed software was. Now it may seem bizarrely unlikely that you'll ever see this kind of situation. But failures often start from network congestion due to a packet storm. And the failures lead to chatty Byzantine fault tolerance protocols adding significant traffic. This causes cascading failures. And so a small, simple outage can escalate into a series of outages as ever more confused servers continue overwhelming the network with their futile attempts to discover what is…

I'm interested in the social-proof mechanisms we'll need to have for AI-assisted "serious projects" (where a database is a "serious project"). Third-party audits like https://jepsen.io/ are an example.

It's a nice intro! One thing I would respectfully suggest is using more precise terminology than "ACID". Jepsen has a great resource for looking at various consistency models (https://jepsen.io/consistency/models) and while every database these days says it is "ACID", very few can guarantee Strict Serializability in a distributed setting, like FoundationDB does.

Sort of related. Jepsen and Antithesis recently released a glossary of common terms which is a fantastic reference. https://jepsen.io/blog/2025-10-20-distsys-glossary

I don't understand this article and It's like the author doesn't really know what they're talking about. They don't want eventual consistency, they want read-your-writes, a consistency level that's stronger than EC yet still not strong. https://jepsen.io/consistency/models/read-your-writes Read-your-writes is indeed useful because it makes code easier to write: every process can behave as if it was the only one in the world, devs can write synchronous code, that's great ! But you don't need strong consistency. I hope developers learn a little bit more about the domain before going to strong consistency.

Rule of thumb: Medium = primers, Substack & corp blogs = mid-depth, hand-curated personal blogs = deep dives. Would love to see more Jepsen-grade explorations (<https://jepsen.io/consistency/models>) that include code snippets and highlight both implementation details and real-world usage caveats for different consistency models.

Thank you! And I agree haha. I love writing SQLite VFS's. In SQLSync, the major issue was precisely what you describe: pulling in the full snapshot to replay. This is the main driver behind the "partial" aspect of the Graft design. It means that clients only need to pull the portion of the incoming snapshot that they don't already have and that overlap with the read/write set of their transactions. So yes, to answer your question once SQLSync is powered by Graft, it will frequently be able to avoid downloading all state in order to reset and replay. Note that if a client is ok with relaxing to snapshot isolation (in particular this means clients may experience Write Skew[1]), and the r/w set of their local transaction does not intersect the snapshot changeset, Graft is able to perform an automatic merge. [1]: https://jepsen.io/consistency/phenomena/a5b

Although I'm not a fauna user, I like studying databases and query languages. I've followed them since Jepsen¹ reviewed FaunaDB 2.5.4 back in 2019. While their FQL looks useful, I can't imagine using a SaaS-only proprietary query language in this age for anything other than a research proof-of-concept to learn more about my problem before building a real product on an open platform. Curious that Bob Muglia (ex Snowflake) is on their board. I believe he's also on the board of Relational AI which is another SaaS with an interesting but proprietary query language. Snowflake is a huge success in spite of being proprietary because its SQL isn't hard to learn or use, but I can't say the same about rAI. Makes me wonder how they will turn out. ¹ https://jepsen.io/analyses/faunadb-2.5.4

Great link! On a related note, a difference in Postgres is that both Repeatable Read and the higher Serializable Isolation levels are both using Snapshot Isolation under the hood. I've been trying to grok this stuff for a few years and it's good to keep in mind that the implementations of various isolation levels between vendors can vary significantly. I've found the resources on Jepsen very helpful as well if you want to go down a rabbit hole :) https://www.postgresql.org/docs/current/transaction-iso.html https://jepsen.io/consistency https://jepsen.io/consistency/models/snapshot-isolation

The price of persistent storage and memory was the same for Mongo as it was for Postgres back when the NoSQL movement happened. Mongo wasn't made of magic scaling fairies and still needed resources. As soon as you cranked up the safety of Mongo to a nearly acceptable level it's performance fell through the floor. The only reason people though it performed amazingly was because of Mongo's rather deceptive marketing. I use the term "nearly acceptable" because for a long time Mongo's data consistency was absolutely crap (https://jepsen.io/analyses) Personally I think people used Mongo because it was new, shiny and exciting and it promised performance (though it didn't really deliver for most people).

> ensure consistency and reliability across two different database stores. This is main known known. And this is hard thing to attain. My favorite story on that is testing of tendermint consensus implementation [1]. The testing process found a way to break the consensus and the reason was that protocol implementation and KV store controlled by protocol used different databases. [1] https://jepsen.io/analyses/tendermint-0-10-2

I understand why the default transaction isolation level of most DBMS is weaker than serializable (it's for benchmark purposes), but I'd argue the best default is serializable. Most DBMS users don't even know there are many consistency models [1]. They expect transactions to "just work," i.e. to appear to have occurred in some total order, which is the definition of serializability [2]. And to some who know when to use a weaker isolation level for better performance can always set it per transaction [3]. --- [1] https://jepsen.io/consistency [2] https://jepsen.io/consistency/models/serializable [3] https://www.postgresql.org/docs/16/sql-set-transaction.html

They haven't read https://jepsen.io/analyses/mongodb-4.2.6 and therefore make the mistake thinking it's a good idea to use it for some reason. It is not. I also never found any feature which would want to make you use it in the first place, even if it weren't a bug-ridden mess that probably eats your data if you don't look, but maybe I didn't look long enough. (also discussed on HN at the time, for more examples of their great quality. That they haven't paid for a thorough follow-up analysis after their supposed fix is all one needs to know)

> Please avoid using distributed systems when non distributed solutions suffice. I would give the opposite advice. Every non-trivial production system is a distributed system. At the very least your database is a replica set and therefore a distributed system. So avoid learning distributed systems at your own peril. Check out https://jepsen.io/ and https://raft.github.io/

This lecture series by Martin Kleppman is excellent: https://youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB&a… Reading trough the Jepsen analyses is also invaluable (not for the faint hearted, you might find some software that is dear to you absolutely torn to shreds in there): https://jepsen.io/analyses Finally, all of the classic papers are worth a read: spanner, bigtable, google file system, dynamodb, chubby, dapper, the google paxos engineering paper, the main ones from Lamport

also jepsen report says that it is mongo marketing assessment, and they are not really acid: MongoDB’s claim of “full ACID transactions” which “maintain the same data integrity guarantees you are used to in traditional databases” could be misleading. We recommend that users who are accustomed to serializable behavior evaluate critical transactions carefully, to identify whether running at snapshot isolation could violate application-level constraints. MongoDB may wish to revise their marketing language to use “snapshot isolated” instead of “ACID”. https://jepsen.io/analyses/mongodb-4.2.6

Postgres by far. The inbuilt json support is great. In addition, Postgres as a whole has decades of bullet proofing behind it. MongoDB doesn't have many advantages, has many bugs and sharp edges, and doesn't have enough advantages to justify it's use. In [an old analysis by Jepsen](https://jepsen.io/analyses/mongodb-4.2.6), MongoDB 4.2.6 exhibited "read skew, cyclic information flow, duplicate writes, and internal consistency violations. Weak defaults meant that transactions could lose writes and allow dirty reads." Will note that json/text columns are better than jsonb. Lower serialization overheads and surprisingly smaller binary size. Or better yet, just normalize to a columnar format.

The biggest point of contention here seems to be over whether kafka can still be considered durable/safe when fsync is disabled. Seems like it'd be valuable to have a trusted third party like https://jepsen.io/ test it out! (not related, just a fan of their work)

Just ask Kyle Kingsbury[1] what kind of havoc comes with not having a strong model of causal order in distributed systems. [1] http://jepsen.io/

Your comment is not true for high quality databases like PostgrSQL. Check-and-set just works on the correct isolation level. There is no isolation, your transaction will fail and be rolled back. If you are new to the subject there are extensive reports available from Jepsten, one of the leading names in distributed system and ACID anomaly testing and research. https://jepsen.io/

Clojure is pretty sprightly, perhaps even prolific, surely. For example, if you're in any way interested in databases, you will have come across https://jepsen.io/ who do all their testing using Clojure.

Isolation levels in ACID are typically used for single instances. However, if you're talking about a distributed system then consistency diagram on Jepsen is more applicable[1]. Raft, for example, can ensure read consistency across a cluster by running reads through the consensus mechanism. LiteFS aims to be the analogue to Postgres/MySQL replication and those generally work great for most applications. [1]: https://jepsen.io/consistency

I disagree that the analysis articles from Jepsen, taken as whole, are arguing for a "perfect distributed system without trade offs". In my experience these articles are fairly even-handed, indeed, many products make some changes based on the analysis and then contract for a follow-up. That the "tone and implication is as if somehow there is some alternative panacea without tradeoffs..." is not something I observed in this analysis or the previous ones that I have read. In my opinion the Rethink DB 2.1.5 report went fairly well for that project. If the claims are aligned with the reality of the product, it's clear the Jepsen report will highlight that. https://jepsen.io/analyses/rethinkdb-2-1-5

Cloudflare etc that others have already posted are excellent by your definition. Uber blogs are surprisingly good. https://eng.uber.com/ Or more technical and less business goal focussed: https://jepsen.io/analyses https://lemire.me/blog/