Antithesis Blog

The test simulator/game starts at ~17:50 For those jumping to the comments wondering if it's worth watching: TigerBeetle[0] is a database for managing financial transactions. They extend the testing practices of FoundationDB (now spun out into Antithesis[1]), going a layer deeper to test for data integrity when there is disk io corruption (~20:30) In the demo, they have created a game-like simulation that runs in the browser. You can watch Joran "attack" database nodes and simulate corruption. The entire presentation is impressive and the simulator is very entertaining. [0] https://tigerbeetle.com [1] https://antithesis.com

Part of the FDB team (great folks) went on to create something quite incredible I have the pleasure of having early access to. If you’re into dependability check this out: https://antithesis.com/

Author here. Let me know if you have any questions! Let me take a shot at answering what I think will be the most common question: how can I trust this? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time. We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree. We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification. If you want to support the project, the easiest way is to give us a star on GitHub[3] [0] https://www.postgresql.org/message-id/19597-39c532e61d78dff6%40postgre… [1] https://antithesis.com/ [2] https://aretta.ai/ [3] https://github.com/malisper/pgrust

> Isolated QA should not exist because anything a QA engineer can do manually can be automated. I almost entirely agree. You'd certainly want a majority of QA responsibilities to fall to devs since that keeps the feedback loop tight. Also with the amount of compute we have you can trivially outpace a typical QA team with a single dev working on fuzz tests and property based tests. If you really still feel the need to outsource then there are cool tools like Antithesis [1] which let you trade money for compute in their complicated fuzz testing suite. Where that falls apart imo is when your software interacts with hardware in any non-trivial way. Fuzz testing doesn't really work when your system is spinning big and heavy things for example. Real hardware that interacts with the world will always create this gap where the devs don't know how to fully test it. Your systems engineers are best equipped to handle this and ensure certain controls follow the right curves, but they typically don't have the bandwidth or are very inexperienced with software. I think a QA organization _can_ fill this gap and deliver value. But I'd almost always prefer hiring another 1 or 2 system engineers who can work towards this problem full time instead. It's much easier to train someone in software than it is to train them in motor control or sealed systems. [1] https://antithesis.com/

Oh, I just realized I am guilty of the drive-by-free-association comment without actually saying anything about the subject of the post - sorry! Very cool to see a team use Jepsen for super early pre-release testing of the system. I wonder if you wish you had waited for the runtime to be a bit more stable, or you feel this was already well worth the effort, even with some of the identified failures being in "known incomplete" areas? (I could see either side of the argument - waiting longer might give you more valuable failures, but testing early gives you a chance to catch problems before they become baked into the foundation and become more difficult to fix...) Another tool that feels like sci-fi to me any time I hear a mention of it, is Antithesis [1] - written by the people who built FoundationDB. Could be another interesting integration to investigate in the future to help bulletproof the language runtime? [1]: https://antithesis.com

Seems like a great use case for https://antithesis.com

Antithesis | Software Engineers, ML Research Scientists | Northern VA | ONSITE | https://antithesis.com/ Antithesis is using deterministic simulation, fuzzing, and novel ML techniques to improve software testing and increase developer velocity. For an idea of what we're doing, see Will's talk here: https://www.youtube.com/watch?v=fFSPwJFXVlw or read the Simulation Testing section of https://www.foundationdb.org/files/fdb-paper.pdf. Fully funded medical, dental, and vision insurance. Unlimited vacation (with mandatory MINIMUM vacation time). Flexible hours. Incredible, cohesive team with the hardest and most interesting problems to solve. We work together onsite in a highly collaborative environment. We're doing things nobody has done before, so we don't need familiarity with any particular language, framework, domain, or anything like that. Don't care. (But if you love Nix...we should talk.) What we do care about: you're smart, get things done, a joy to work with, and are comfortable dealing with open-ended problems. Apply at https://antithesis.breezy.hr/ (Don't worry too much about which role you choose. We're looking for engineers who are ready to learn, not those that know everything already!) We're also hiring for professional services roles. Feel free to ask about those.

Thanks for a great question! When we made the decision: 1. We were thinking long term. For example, a distributed database is a significant investment. I love the spirit and orthogonality of C, but we didn't want to pay the safety tax of C over the next 20 years. At the same time, Zig has that same spirit—it's not only a perfect replacement, but a leap forward in developer velocity, power, performance and ergonomics. 2. We realized that TigerBeetle would take 2 years to get to a production release. This meant that our timeline would intersect with Zig's stability—like skating to where the puck is going to be at, or catching the swell as it breaks, rather than riding out the last of an old wave. The number of surfers (quantity) was not a concern. Rather, we were impressed by the sheer quality and early maturity of Zig's compilation story, not to mention the quality of the Zig community in general. For example, it would be hard to find a better std lib crypto than what Zig already has right now, thanks to Frank Denis of libsodium. 3. The design of TigerBeetle is also a single-threaded control plane. We use io_uring for the data plane to eliminate multi-threaded context switches, so multi-threading for async I/O is less of a necessary evil than it used to be. All memory is statically allocated at startup. We never call free() so there are no UAFs. You can start to see why Rust's borrow checker made less sense for our domain than it would have for others. Of course,…

This is an excellent article, I’ve seen almost all of the issues it calls out in production for various APIs. I’ll be saving this to share with my team. I’ve seen two separate engineers implement a “generic idempotent operation” library which used separate transactions to store the idempotency details without realizing the issues it had. That was in an organization of less than 100 engineers less than 5 years apart. One other thing I would augment this with is Antithesis’ Definite vs Indefinite error definition (https://antithesis.com/docs/resources/reliability_glossary/#definite-e… It helps to classify your failures in this way when considering replay behavior.

Some more links that are inside the article: - More info about skiplists: https://arxiv.org/pdf/2403.04582 - Performance comparison with B-tree ?: https://db.cs.cmu.edu/papers/2018/mod342-wangA.pdf - Other blog from Anthithesis about writing their own db: https://antithesis.com/blog/2025/testing_pangolin/ Also I find it a bit hard to understand the performance outcome of this setup. I know formats like parquet and databases like ClickHouse work better when duplicating data instead of doing joins. I guess BigQuery is similar. The article is great but would be also interesting to learn how performance actually worked out with this.

> A saner approach would be to start with a FFI-friendly language and create bindings. I don't think just being able to use an already written framework in Python is worth the trade-off. For what it's worth the devs say their "current long-term plan is to implement a second Hegel server in Rust" [0], so the current state of affairs is probably a compromise between getting something usable for end users out and something more "sane", as you put it. [0]: https://antithesis.com/blog/2026/hegel/#what%E2%80%99s-next

Loom does exhaustive search, with clever methods to prune it. On real world programs, you have to set a limit to that because it obviously grows extremely quickly even with the pruning. I've built something similar to Loom, except it's more focused on extensively modeling the C++11/Rust memory model (https://github.com/reitzensteinm/temper). My experience is that fairly shallow random concurrent fuzzing yields the vast majority of all concurrency bugs. Antithesis (https://antithesis.com/) are probably the leaders of the pack in going deeper.

I love that resource and reference it fairly frequently. There is also this [1] which Aphyr collabed on which you might find interesting if you haven’t seen it yet. [1] https://antithesis.com/resources/reliability_glossary/

I really enjoyed this, and following the link to the Antithesis talk was really interesting as well [0] as it goes in to some of the pitfalls of random testing and some of the properties that you'd like out of a good test system. This is a field I'm really interested in but it seems hard to grasp. [0] https://antithesis.com/blog/sdtalk/

https://antithesis.com/ was made to deal with this. You can think of its as a fuzzing but it has overall determinism for the whole system, so there is a time travel and interactive debugging.

Im pretty sure valgrind and friends can be used in zig. Zig is still not 1.0, theres not much stability guarantees, making something like Frama-C, even tho it is possible is simply going to be soo much pain due to constant breakages as compared to something like C. But it is not impossible and there have been demos of refinement type checkers https://github.com/ityonemo/clr Beyond that, tools like antithesis https://antithesis.com/ exist that can be used for checking bugs. [ I dont have any experience with it. ]

DST was i̶n̶v̶e̶n̶t̶e̶d̶ popularized at FoundationDB a little over a decade ago, and has been quietly gathering steam ever since. If you’re interested in the technique, the FDB paper has some good info in section 4 and section 6.2: https://www.foundationdb.org/files/fdb-paper.pdf (Disclosure: I am an author.) I also gave a talk overview of it at Strange Loop back in 2015, but don’t have the youtube link handy. If you’re interested in trying DST, we have a new company that aims to make it a much easier lift, especially for existing projects that can’t be retrofitted onto one of the many simulation frameworks: https://antithesis.com Happy to answer any questions about the approach, here or over email.

Aphyr didn’t test foundation himself, but the foundation team did their own Jepsen testing which they reported passing. All of this was a long time ago, before Foundation was bought by Apple and open sourced. Now members of the original Foundation team have started Antithesis (https://antithesis.com/) to make it easier for other systems to adopt this sort of testing.

> Put another way, automated tests don’t go far enough. We need yet another higher layer of abstraction. Computers are better at deciding what tests to run and when, and are also better at interpreting the results. Sounds like you might be interested in https://antithesis.com/ (no affiliation).

They are developing this simulation concept as a platform: https://antithesis.com/

They are famous for doing simulation testing. https://antithesis.com/ Have recently brought to market a simulation testing product.

For something that's actually practical for most engineers and at scale, I would look at: 1) https://www.stateright.rs and 2) https://antithesis.com

Two things: 1. After the recent successful growth of Antithesis [1], I'm diving into the topic of deterministic simulation testing. There are some cases (and ready-to-use libraries) where people are doing this in Rust, C++, Go, I'm interesting in this in Java. So I'm up to some experiments. I've also started an "awesome list" of resources about this topic [2] 2. I've made a generated serialization/deserialization library for the Kafka wire protocol in Rust, tested against the original Java implementation. I'll add 3.9 support once it's released and don't see much upcoming changes to the library, apart from maybe working on the Go version. [1] https://antithesis.com/ [2] https://github.com/ivanyu/awesome-deterministic-simulation-testing [3] https://github.com/ivanyu/kafka_wire_protocol

This is the central premise of Antithesis: https://antithesis.com (no affiliation)

P.S. A focus on soundness does not, I believe, help the cause of more correct software but distracts from techniques that have proven more fruitful in practice (and supported by theory) -- sometimes to the surprise of those who used to think that soundness is the only way, e.g. http://users.csc.calpoly.edu/~gfisher/classes/509/handouts/hoare-witho… This happens over and over: there is some important improvement in sound methods while at the same time unsound methods are making much bigger strides toward more correct software in practice. Here's just one very recent example (which isn't even necessarily state-of-the-art in unsound methods): https://antithesis.com Of course, this is a very complex topic, because there are some specific situations where sound techniques are more powerful and practical.

Just dropping my $0.02 cents on the Jepsen testing for Qdrant's distributed guarantees. While Jepsen is a solid choice, have you considered throwing Antithesis [1] into the mix as well? If it's already on your radar, then no worries, just figured I'd mention it. [1]: https://antithesis.com/

The FoundationDb team is working on distributed systems simulation testing via a new startup... Antithesis. See https://antithesis.com/ Stardog, enterprise knowledge graph platform, (stardog.com), is an early adopter of Antithesis and we've found it to be very helpful in HA Cluster, distributed consensus, etc. Not a stakeholder, just a satisfied early adopter.

Or something like https://antithesis.com/

You should team up with Antithesis. https://antithesis.com/

It is difficult to use by itself: the "foundation" in the name describes it quite well. It is a foundation that you build a database on. It fits my use case very well, for example, because I know my data model and usage patterns very well and I can integrate deeply with the database, but it's not a good match by itself for quick-and-dirty apps. It provides fantastic (strict serializable) consistency guarantees in a distributed database, which is extremely rare. It is a huge advantage, but sadly most people do not understand how badly most distributed databases are broken and don't even understand the concepts (https://antithesis.com/resources/reliability_glossary/) well enough to talk about the issues involved. See every discussion where someone mentions ACID. It's hard to compete for mindshare when the concepts are difficult and every other database has a warm-and-fuzzy-feeling website saying that everything will be great (it usually won't). Personally, I hope more people will start using it, and I hope to see more easy-to-use databases built on top of it (that's what it was designed for, really). In my experience with it, working with a fast distributed database that gives you strict serializable semantics right in your code is fantastic.