Redpanda Blog

In most cases, a single process app sending data to a file and a small HTTP server serving it will perform much better with less downtime. As I understand it, this is where Redpanda enters the picture. Maybe others here disagree? https://softwareengineeringdaily.com/2021/01/22/redpanda-kafka-alterna… https://github.com/vectorizedio/redpanda https://vectorized.io/

All this talk about NATs as an alternative to Kafka with no mention of Redpanda’s Zookeeper-less alternative (written in modern C++ - https://vectorized.io/)

Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/

indeed. that's why we continuously _empirically_ prove that we are in fact safe - https://vectorized.io/validating-consistency/ there is no substitute to testing tho there are 2 levels here. 1) raft has a proof (and a great phd dissertation from diego), but what matters is if we actually implemented it correctly. so 2) is we need to continuously test it. Denis did a lot of similar work at CosmosDB (microsoft) and has spent his career working on consensus. Hopefully these eases some concerns.

See smf-rpc (fastest rpc in the west) https://github.com/smfrpc/smf Created in seastar-framework (scylladb's core) created at Akamai , used in Redpanda (https://vectorized.io/) that is doing scylladb for apache kafka.

I’m not too convinced of the approach. I’ve been anxiously waiting for https://vectorized.io/ to release their message queue. It is built in modern C++, uses ScyllaDB Seastar framework to do IO scheduling in userspace, with better mechanical sympathy. And like Hashicorp’s Nomad and Vault, which I’m a fan of, it has built-in distributed consensus and easy operation.

Redpanda [1] is fantastic if you want a solid Kafka clone. It's written in C++, with no external dependencies; it has Raft consensus built in. The API is Kafka-compatible. There are some missing features compared to Kafka (no dynamic rebalancing of partitions being the top one), but they're also rolling out some new features that Kafka does not have, such as transformation pipelines [2]. NATS's brand new JetStream [3] feature is also looking very promising. It uses Raft internally. NATS itself is rock solid, and JetStream builds on that foundation. NATS also has very interesting support replication topologies, meaning you can build graphs of streams that each feed in other streams, so you can do the publishing and consumption in different locations, with different availability constraints. [1] https://vectorized.io/redpanda [2] https://vectorized.io/blog/wasm-architecture/ [3] https://docs.nats.io/jetstream/jetstream

> Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place? Very few people roll out of bed and think "my application needs a high performance distributed event stream". For every problem that can be solved with kafka, there are other solutions that don't involve kafka. Many of the alternate solutions don't involve event streams at all. Are they better? I don't know. That question is unknowable without looking into the problems in particular, and looking at what other technology is out there. And I say this with a lot of fondness for kafka. I really like it. I just think there are always technical tradeoffs to make. > any more than MongoDB competes with PostgreSQL MongoDB and Postgres do compete with each other though! They work differently - so part of the competition isn't around features; its around how your people think about your data. "Think of your data in a way that makes sense for mongodb!" say mongodb sales reps. "Think of your problem as a message queue" say fans of rabbitmq. They aren't wrong in the general case. But there are lots of problems where I could use kafka, or I could use redis, or I could use a naked postgres instance. You need a much more complex heuristic to decide what technology to adopt than "Should I use kafka?" -> "Yes."[1] And I say all that as a big fan of kafka. I think its API is delightful and just today I was plagerizing its API design to sort out backpressure in a system…

Kafka is (IME) pleasant to use for people writing application code. It can handle high message volumes. It has good documentation. Its core abstraction is easy to understand and use. OTOH, the author is correct that it's more work to set up and maintain than non-distributed software. Don't introduce it unless you have carefully considered alternatives. I became de facto responsible for running a small Kafka cluster at a small company that could have gotten away with something less shiny. That was not fun. (Especially since the company started with Kafka 0.8 and encountered many changes along the way to modern releases). Eventually the company ponied up for Confluent to take over running Kafka for us as a service. Now I'm back to enjoying the nice Kafka APIs without having to deal with the annoying Kafka servers. I like the Kafka API/capabilities but dislike running actual Kafka servers. If I were to start a one-man venture that needed event streaming I'd probably try Redpanda [1]. Its developer hangs out here, which is how I first learned of it. It's supposed to implement Kafka service but with no Zookeeper or JVM knowledge needed to run it. I wouldn't even consider running the original Apache Kafka for a solo/small team. [1] https://vectorized.io/

> Cassandra (https://scylladb.com) and Kafka (https://vectorized.io) have already been rewritten once in C++, with massive latency and throughput improvements. It is not a single dimension problem as they try to frame it. Performance is a single dimension of systems. Java with GC is a memory safe solution compare to C++ (hello segmentation faults). The reason why the C++ implementation is faster is not that it is written in C++ but it is written using a different approach. There are plenty of Java HFT systems that run circles around C++ systems yet we cannot draw a conclusion that Java is better suited than C++ for HFT. Implementation details matter, code quality matters and performance does not come purely because you pick a particular programming language. Btw. most of the systems which I was working on using Cassandra instead of ScyllaDB because the companies were not ok switch up for some performance gains (because Cassandra was performant enough for their business use case). I think what you receive as superiority is not as clear cut as you try to make it.

Redpanda https://vectorized.io/

Similar vibe to redpanda vs. kafka: take a reasonable idea and reimplement it so it isn't terrible. https://vectorized.io/blog/fast-and-safe/

https://vectorized.io/redpanda/ is a more useful link, since the main domain appears to have some JS errors right now.

- [RedPanda|https://vectorized.io/redpanda/]? - [Apache Pulsar|https://pulsar.apache.org]?

Ok, thanks. Does redpanda do some kind of auto anti-affinity on hosts for partition group to spread across remote cores? ps. redpanda link from article is broken, goes to https://vectorized.io/blog/tpc-buffers/vectorized.io/redpanda 404

According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space): https://vectorized.io/dataengineering-podcast-interview/ The throughput is 4x, the latency is 10x.

I helped early on to review this plan for Alex and the Vectorized team, so I’m very excited to finally see it be released to the world. Vectorized consists of quite the lineup of engineers (https://vectorized.io/team) and anyone who gets to be mentored by them will be gaining a huge advantage when it comes to Distributed System design and implementation!

For a drop-in Kafka replacement, check out: https://vectorized.io/redpanda/

This sounds really similar to some of the work coming out of Vectorized on Redpanda [0]. They're building a Kafka API compatible system in C++ that's apparently achieving significant performance gains (throughput, tail latency) while maintaining operational simplicity. [0]: https://vectorized.io/redpanda

Vectorized | Remote | Full-time | https://vectorized.io/careers Come shape the future of streaming! Vectorized is the startup behind Redpanda, a Kafka-compatible event streaming platform. With a compelling combination of no data loss, operational simplicity, and consistent performance at scale, Redpanda takes advantage of modern hardware to deliver the modern streaming platform for mission critical workloads. We are a team of doers, seasoned engineers, hackers and builders, working on the future of streaming. A fully distributed, diverse team, with members in Medellin - Colombia, London, San Francisco, St. Petersburg - Russia, NYC, San Diego, Germany, Canada, Poland and growing! Vectorized is looking for the following roles: * Customer Success Manager - https://vectorized.io/careers/customer-success-manager/ * Product Manager - https://vectorized.io/careers/product-manager/ * Sales Development Representative - https://vectorized.io/careers/sales-development-representative/ * Support Engineer - https://vectorized.io/careers/support-engineer/ * Solutions Engineer - https://vectorized.io/careers/solutions-architect/ * Site Reliability Engineer - https://vectorized.io/careers/sre/ * Kubernetes Engineer - https://vectorized.io/careers/kubernetes-engineer/ * Technical Content Developer - https://vectorized.io/careers/tech-content-dev/ * C++ Core Engineer - https://vectorized.io/careers/cplusplus-core-engineer/ To apply, visit the above job-specific URLs, or send an…

> Applications of Java and Rust do not overlap. Both are general purpose languages, so of course they overlap. > Good luck rewritting all the big data tools from Java (Elasticsearch, Spark, Kafka, Hadoop, Neo4j, Deeplearning4j, Cassandra, Solr, Arrow, OrientDB) Cassandra (https://scylladb.com) and Kafka (https://vectorized.io) have already been rewritten once in C++, with massive latency and throughput improvements. No reason why they couldn't get their superior Rust clones in the observable future. Materialize (https://materialize.com), Noria (https://github.com/mit-pdos/noria), and Sled (https://github.com/spacejam/sled) are just some of the Rust database projects that are aiming at unseating the de facto standard implementations in the space. InfluxDB (https://www.influxdata.com) is now doing major Rust development as well. The future is almost here. It's just not evenly distributed yet.