Jack Vanlightly

I highly recommend all of Jack's blog posts about RabbitMQ - https://jack-vanlightly.com/blog/tag/RabbitMQ Jack works with me on the RabbitMQ core engineering team. We've been hard at work to address a lot of the issues brought up in comments here. It's worth it to try out our latest releases. The engineering team is very active with the community and takes all constructive, helpful (i.e. reproducible) feedback seriously. Feedback is encouraged via the rabbitmq-users mailing list. Thanks.

Thanks for all your work on RabbitMQ, and for your great blog posts about it and other messaging systems. For anyone who wants to understand the potential complexities of HA RabbitMQ, spend some time reading https://jack-vanlightly.com/blog/2018/8/31/rabbitmq-vs-kafka-part-5-fa…

This is one of the best overviews on Pulsar with comparisons to Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pu…

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pu… > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working. Wish you success in your effort!

You got it! By approximate category and in no particular order: ## MISCELLANEOUS - [Low Tech Magazine](https://solar.lowtechmagazine.com), a solar-powered website showcasing the amazing low-tech things you can do with a little ingenuity, - [Stephen Wolfram's Blog](https://writings.stephenwolfram.com) on math, compsci, and everything else, - [Cool Macrophotography](https://www.barrywebbimages.co.uk/Images/Macro/Slime-Moulds-Myxomycete… for amazing pictures of tiny things (WARNING: Time Sink!), ## COMPUTER SCIENCE - [While-True-Do](https://blog.while-true-do.io) has lots of stuff on Linux administration and self-hosting, - [Backend Banter](https://www.backendbanter.fm) for a great podcast on backend development, - [Two Wrongs](https://two-wrongs.com) for statistics and functional programming, - [Math ∩ Programming](https://www.jeremykun.com/) where math and programming intersect! Written by a Googler... - [Paul Bourke's site](https://paulbourke.net/) where you can get happily lost exploring old pages on every topic CS touches, - [Jason Punyon's blog](https://jasonpunyon.com) for some fun algorithms and things, - [Yossi Kreinin's blog](https://yosefk.com/blog) for C, Python, and plenty of non-technical topics, - [Adrian Sampson's academic blog](https://www.cs.cornell.edu/~asampson/blog) for an academic's perspective, - [Cool polyhedra viewer project](https://georgehart.com/virtual-polyhedra/vp.html) if you like geometry, - [Another academic blog on…

Thanks for your reply and thanks for your insight and wisdom. 500 million requests per second. I think that's awesome for _intra thread communication_. I have a sharded bank simulation which naively sends money between accounts and it gets 700 million transactions a second which is the aggregate of 12 threads all transacting local to a thread, so 500 million across threads seems good (I split money up rather than accounts per thread). I see that some web frameworks can handle up to 600,000 requests per second on techempower benchmarks. So we need to combine the best of each! Especially the part about one writer. I have a journal entry called "Golden concurrency" inspired by someone's comment on HN (I really should have linked it!) talking about writing to a database from 3 processes and they decided to split the database into 3 databases rather than suffer contention from concurrency control. "Golden concurrency" is my term or desire that degradation caused by multiple producers/multiple consumers in different configurations could be optimised for and could be a scenario that is handled without performance degradation. I suspect you need a different design if you have 1 producer:1 consumer producers to consumers, or 1 producer:N consumers or N producers:1 consumer or N producers:N consumers. I enjoyed the following benchmarks with Kafka and RedPanda which made me think of this…

It just feels like two widely different scenarios we're talking about here. https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-nee… talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine. This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, losing data, and combing back up again, all while (c) ZooKeeper doesn't catch that the leader crashed and elects another leader. I think definitely the title/blurb should be updated to clarify that this is only in the "exceptional" case of >f failures. I mean, the following paragraph seems completely misleading: > Even the loss of power on a single node, resulting in local data loss of unsynchronized data, can lead to silent global data loss in a replicated system that does not use fsync, regardless of the replication protocol in use. The next section (and the Kafka example) is talking about loss of power on a single node combined with another node being isolated. That's very different from just "loss of power on a single node".

No. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1]. [0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-k… [1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-nee…

We fixed that particular issue: https://jack-vanlightly.com/blog/2023/8/17/kafka-kip-966-fixing-the-la…

They keep using the core Postgre while they touch the storage layer to works with S3. Can try ro read more here https://jack-vanlightly.com/analyses/2023/11/15/neon-serverless-postgr…

Thanks for the report - we just deployed fix for the same. No lock-in means it’s 100% open source (PromQL) compatible. You can swap out vendors or move to self-hosted open source solutions should you need to move away from Oodle. When you migrate out, you get to export all your data, dashboards and alerts. you don't need to make any code changes. We support bringing your own bucket (BYOB) for large enterprise customers however, you cannot bring your own compute at this time. Our thoughts are along the lines of how Snowflake approached the problem - everything fully managed to keep the operational overhead minimal. https://jack-vanlightly.com/blog/2023/9/25/on-the-future-of-cloud-serv…

I've been dreaming of Apple Notes and Obsidian doing what the author suggests. The approach seems similar to Delta Lake's consistency model, which is using object storage like S3, and yet allows concurrent writers and readers: https://jack-vanlightly.com/analyses/2024/4/29/understanding-delta-lak…

Thanks for feedback. The link is broken for me, but I think you are referring to this page? https://jack-vanlightly.com/analyses/2024/4/29/understanding-delta-lak… Indeed, the approach is similar. Especially the separation of assets (they call it "Data files") from the log ("Data log") is something I consider being a good choice.

Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer. Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie. Kafka as it's described in the post(): any number of nodes may be down, at most one power outage is allowed (loss of unsynced data), as soon as the majority is available a replicated system is available and doesn't lie. The counter-example simulates a single power outage () https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-nee…

Kafka, unlike Mongo DB, relies on recovery/replication instead of fsync: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-nee… Kafka has never tried to hide that fact and it does not, in any way, make Kafka unsafe.

It's a common misconception about Kafka and fsyncs. But the Kafka replication protocol has a recovery mechanism, much in the same way that Viewstamped Replication Revisited does (except it's safer due to the page cache), which allows Kafka to write to disk asynchronously. The trade-off is that we need fault domains (AZs in the cloud), but if we care about durability and availability, we should be deploying across AZs anyway. We've seen plenty of full region outages, but zero power loss events in multiple AZs in six years. Kafka and fsyncs: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-nee…

Thanks for the links! I was wondering if bcachefs is architectured with NAND-flash SSD hardware in mind (as recently highlighted on HN in the "Is Sequential IO Dead In The Era Of The NVMe Drive" article [1] [2]), to optimize IO and hardware lifecycle. Skimming through the "bcachefs: Principles Of Operation" PDF, it appears the answer is no. [1] https://jack-vanlightly.com/blog/2023/5/9/is-sequential-io-dead-in-the… [2] https://news.ycombinator.com/item?id=35878961

AWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems). It's not all hype either, according to this post https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-messages-o… it seems like a solid piece of tech.

It's a persistent store, so that would be different from Redis Pub-Sub. Compared to RabbitMQ, Pulsar seems to favor strong ordering and protection from message loss. This blog post offers some more info and leaders to other posts comparing Pulsar to RabbitMQ and Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pu…

> What is the relation between "start with why" and "databases are becoming a commodity"? My assumption is that the author is painting out a longer-term vision both for Confluent and its audience. It probably makes more sense in the context of other posts, e.g. > This trend towards object storage is not just happening at Confluent but across the data ecosystem. Many different types of cloud data systems are integrating object storage into their architecture, and I have covered a few of these such as Neon and ClickHouse Cloud. There is also a flurry of start-ups doing logs (such as the Kafka API) over object storage directly (six and counting). There is money being plowed into data systems that use object storage as the primary, and sometimes, only storage layer. https://jack-vanlightly.com/blog/2024/5/2/hybrid-transactional-analyti… In other words: managed Kafka is becoming a commodity and Confluent should "start with why" to figure out where to go. I figured the advice and perspective is of general enough interest.