WarpStream Blog

I think Ducklake[1] is a terrific example of this. They said "look, let's build a lake house over S3, but for the bit that needs strong consistency (the manifest of which S3 blobs are in play), let's use Postgres". Postgres as a metadata catalog or control plane is brilliant for this, since you get strong consistency and the scaling story around a metadata catalog is far different than the volume of data you need to store. Use S3 for volume, Postgres for consistent metadata. A similar pattern has spilled out of projects like Warpstream[2], which I suspect is using Postgres behind the scenes of their control plane. [1]: https://ducklake.select [2]: https://www.warpstream.com/

Reminds me of WarpStream: https://www.warpstream.com Similar idea but you have the power of S3 scale (if you really need it). For context, I do not work at WS. My company switched to it recently and we've seen great improvements over traditional Kafka.

This isn't appropriate for all use-cases, but one way to address your and GP's problem is as follows: 1. Aggregate (in-memory or on cheap storage) events in the publisher application into batches. 2. Ship those batches to S3/alike, NFS that clickhouse can read, or equivalent (even a dead-simple HTTP server that just receives file POSTs and writes them to disk, running on storage that clickhouse can reach). The tool you use here needs to be idempotent (retries of failed/timed out uploads don't mangle data), and atomic to readers (partially-received data is never readable). 3. In ClickHouse, run a scheduled refresh of a materialized view pointed at the uploaded data (either "SELECT ... INFILE" for local/NFS files, or "INSERT INTO ... SELECT s3(...)" for an S3/alike): https://clickhouse.com/docs/en/materialized-view/refreshable-materiali… This is only a valid solution given specific constraints; if you don't match these, it may not work for you: 1. You have to be OK with the "experimental" status of refreshable materialized views. My and other users' experience with the feature seems generally positive at this point, and it has been out for awhile. 2. Given your emitter data rates, there must exist a batch size of data which appropriately balances keeping up with uploads to your blob store and the potential of data loss if an emitter crashes before a batch is shipped. If you're sending e.g. financial transaction source-of-record data, then this will not work for…

For Rust-based Kafka alternatives, I like Tansu[1]. It at least provides Kafka API parity, and critically also gives users a pluggable backend (embedded SQLite, S3 for low cost diskless type workloads and Postgres because just use Postgres) It’s nice to try and out innovate Kafka, but I fear the network effect can’t be beaten unless the alternative is 10x better. Something like Warpstream’s architecture[2] had a shot at dethroning Kafka, but critically even they adopted the Kafka API. Sure enough, Apache Kafka introduced a competing feature[3] within two years of warpstreams launch too. [1] - https://github.com/tansu-io/tansu [2] - https://www.warpstream.com/ [3] - https://topicpartition.io/blog/kip-1150-diskless-topics-in-apache-kafk…

Update: Just read this article[1] which clarifies S3 Express One Zone. Yes, performance is greatly improved, but actually storage costs are 8x more than a standard S3 bucket. The naming S3 Express One Zone is terrible and a bit misleading on pricing changes. [1] https://www.warpstream.com/blog/s3-express-is-all-you-need

It's a bit sad that many modern databases were recently acquired. They had the potential to bring a lot of innovations. 1. https://www.warpstream.com/ 2. https://www.orioledb.com/ 3. https://quickwit.io/

Off topic: warpstream's calculator on the pricing page is pretty cool https://www.warpstream.com/pricing That breakdown switch is a lovely touch.

This is clever. And as I understand it, one of the tricks WarpStream (https://www.warpstream.com) use to reduce the costs of operating a Kafka cluster.

Not to be confused with https://www.warpstream.com/

Awhile I made something really dumb: https://www.warpstream.com/etc/terminal you can enter a 'gibson' command.

Reminds me of FoundationDB and Antithesis[0]. [0]: https://www.warpstream.com/blog/deterministic-simulation-testing-for-o…

It's my understanding that the newer generation of data lakes still make use of a tiny, strongly consistent metadata database to keep track of what is where. This is orders of magnitudes smaller than what you'd have by putting everything in the same database, but it's still there. This is also the case in newer data streaming platforms (e.g. https://www.warpstream.com/blog/kafka-is-dead-long-live-kafka). I'm curious to hear if you have examples of any database using only object storage as a backend, because back when I started, I couldn't fin any.

Short answer is that MSK has (almost) all the same problems as OSS Kafka. Kinesis streams is a different beast that would require its own blog post. RE:numbers: https://www.warpstream.com/blog/warpstream-benchmarks-and-tco

I am eager to hear how it will affect their latency numbers: > Engineering is about trade-offs, and we’ve made a significant one with WarpStream: latency. The current implementation has a P99 of ~400ms for Produce requests because we never acknowledge data until it has been durably persisted in S3 and committed to our cloud control plane. In addition, our current P99 latency of data end-to-end from producer-to-consumer is around 1s via https://www.warpstream.com/blog/kafka-is-dead-long-live-kafka

Co-founder of WarpStream here. We wrote a short article explaining how the economics of this new storage class work for a system like ours: https://www.warpstream.com/blog/s3-express-is-all-you-need