Confluent Blog

This will be interesting to try out. I've used all the major cloud event/logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance. Only downside with Google Pubsub can be latency (which I'm working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it'll be some good competition to Google's option. Also want to note that Jay Kreps who helped build Kafka at LinkedIn is now behind http://www.confluent.io/ which is like a better/enterprise version of Kafka.

Confluent | Remote (US and Europe) | Full Time | https://www.confluent.io/ Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Our cloud-native offering is designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. We’re looking for backend engineers to join our cloud infrastructure networking team and help us improve connectivity, performance, reliability, and security. We are a team of passionate engineers who love solving complex distributed systems and infrastructure problems. You’ll be able to have a big impact almost immediately and you will have the opportunity to own solutions end-to-end with a lot of autonomy. We’ll support your growth and actively look to help you develop your technical skills, ensure you’re doing work you enjoy, and make sure you’re advancing your career. To apply in US go to https://www.linkedin.com/jobs/view/2736465947/. To apply in Europe go to https://www.linkedin.com/jobs/view/3189560718/.

To stimulate insightful thinking and discussion, I'd like to present an opinion. I believe that tiered storage cannot save Kafka. 1. First, tiered storage does not fundamentally solve the issue of Kafka's cross-Availability Zone (AZ) traffic costs. This part of the traffic cost can cause a significant expenditure in the cloud. For reference, you can check out this link: https://www.confluent.io/blog/understanding-and-optimizing-your-kafka-… 2. Essentially, tiered storage only alleviates the disadvantage of Kafka's partition data movement, but it does not eradicate it. Kafka requires the last logsegment of a partition to be on local storage. If this segment is large, a substantial amount of partition data still needs to be replicated during scaling.

This is essentially the idea behind CDC (Change Data Capture) [0]. Martin Kleppmann has some great blogs about this as well [1]. [0] https://en.wikipedia.org/wiki/Change_data_capture [1] https://www.confluent.io/blog/turning-the-database-inside-out-with-apa…

I guess they need funding to better publitize their offering. It's almost one-click activation in Google Cloud (I guess in AWS/Azure would be te same). IMHO their biggest competitors are the clouders themselves (with pubsub, SQS, etc..), MQTT, ZeroMQ, and the open source Kafka (which is surprisingly easy to setup). For me, the best idea they had was the Kafka schema registry (https://www.confluent.io/product/confluent-platform/data-compatibility… : it's too easy to inject bytes into kafka, but for using that stuff you need to know the format (and no, dunping a JSON string is not a good subtitute for a schema). Instead of guessing, and pulling your hair when there are changes, they provide a way to insert the data with the correct types and format. (disclaimer I don't have shares in confluent, nor interest in buying some)

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. There are Kafka clusters running in production with over a petabyte of stored data." [1] https://www.confluent.io/blog/okay-store-data-apache-kafka/

> Can you guarantee ordering? Can you guarantee exactly once? How do you get transactional semantics? How do you handle cascading failures? It's all a bit overwhelming. I don’t advocate DDD/microservices but if you absolutely must go down this road, Kafka Streams really neatly answers all these questions. Ben Stopford’s book[0] and blog posts[1] are really great if you want to know more. [0] http://www.benstopford.com/2018/04/27/book-designing-event-driven-syst… [1] https://www.confluent.io/blog/building-a-microservices-ecosystem-with-…

Pretty great talk about Kafka at Tesla: https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-s…

It is curious indeed. The first time I noticed this curiosity is when I saw Martin Kleppman’s Turning the Database Inside Out [1]. It’s a great watch (or read) and I really recommend it! [1] https://www.confluent.io/blog/turning-the-database-inside-out-with-apa…

Whoa! somebody actually try to implement the log as the central component of a db! After read https://www.confluent.io/blog/turning-the-database-inside-out-with-apa… I wondered about that. I think that make the log first class and "plug" relational tables (optionally) will make a amazing database engine. In short, you PERSIST your commands: POST /City .. PUT /City .. DELETE /City .. and put listeners that decide if persist or not the commands, this allow to do: POST /SendMail (to:...) and have the flexibility to bundle the domain logic on top of the data logic in a single lang. This is my long term goal..

Related read - here is an explanation, coming from a Clojure folk, of why Kafka (Streams) might play a crucial role in your next architecture. https://www.confluent.io/blog/toward-functional-programming-analogy-mi… Has been a great influence in my jobs over the last year.

Surprised there wasn't any mentioned of [1] in the paper's citations. This notion of articulating the internal sub-systems of a DB as first class elements of a distributed realization has been emerging (from various quarters) for the past few years. I would also include Apache Pulsar in this category. Aurora itself seems very interesting from an architectural point of view and definitely worth studying. [1]: https://www.confluent.io/blog/turning-the-database-inside-out-with-apa…

This is a great question. Many people think the AGPL solves this problem but it absolutely does not. This is why MongoDB, which was AGPL licensed, just changed to a custom license. The other reason we don't want to use AGPL is that it can be quite aggressive in making you open source your own code if you want to embed our code in a proprietary application. We actually think building proprietary applications is a fine use of our software, and is one of the major use cases for this kind of infrastructure. This is addressed in more depth along with many other questions in this FAQ: https://www.confluent.io/confluent-community-license-faq

For a more general introduction to how materialized views fit into a larger ecosystem of derived data, I always recommend Martin Kleppmann's post/video on "Turning the Database Inside Out": https://www.confluent.io/blog/turning-the-database-inside-out-with-apa…

For me, the best way to think about Kafka is as a distributed log. That said, it does feature rather robust transformations of log entries - does KSQL count as ”queries or whatever”? https://www.confluent.io/product/ksql/

Disclaimer: I'm one of the creators of Kafka and founders of Confluent The best way to get the power, throughput, latency of Kafka without the operations is to use a hosted service. The one created and supported by the Kafka team is Confluent Cloud https://www.confluent.io/confluent-cloud/

This is the best explanation I've found about Kafka and Samza: https://www.confluent.io/blog/turning-the-database-inside-out-with-apa…

This is an excellent explanation - http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/

Interesting that LinkedIn is an investor in a startup consisting of its former employees. http://confluent.io

So much that we presume in the modern cloud wasn't a given when Apache Kafka was first released in 2011. kevstev wrote just above about Kafka being written to run on spinning disks (HDDs), while Redpanda was written to take advantage of the latest hardware (local NVMe SSDs). He has some great insights. As well, Apache Kafka was written in Java, back in an era when you were weren't quite sure what operating system you might be running on. For example, when Azure first launched they had a Windows NT-based system called Windows Azure. Most everyone else had already decided to roll Linux. Microsoft refused to budge on Linux until 2014, and didn't release its own Azure Linux until 2020. Once everyone decided to roll Linux, the "write once run everywhere" promise of Java was obviated. But because you were still locked into a Java Virtual Machine (JVM) your application couldn't optimize itself to the underlying hardware and operating system you were running on. Redpanda, for example, is written in C++ on top of the Seastar framework (seastar.io). The same framework at the heart of ScyllaDB. This engine is a thread-per-core shared-nothing architecture that allows Redpanda to optimize performance for hardware utilization in ways that a Java app can only dream of. CPU utilization, memory usage, IO throughput. It's all just better performance on Redpanda. It means that you're actually getting better utility out of the servers you deploy. Less wasted / fallow CPU cycles — so better…

Delivery is communication between two actors. Processing is what one actor (the receiver) does with a message. Communication is when two actors exchange a message. Communication is generally done over an unreliable medium because in practice there is no way to communicate without the potential of failure. 1. Exactly once communication between two actors over an unreliable medium is impossible. At the very least you have to account for the possibility of failure of the medium, so you might need to re-send messages. 2. At least once communication between two actors is possible -- just re-send a message until the receive acknowledges the message. 3. Because a message might be re-sent, the receiver must be able to cope with duplicate messages. This is what you're describing. This might be done by making message processing idempotent or tracking which messages you've seen before. In either case, you have achieve exactly once processing. That is, if a receiver is given the same message multiple times, only the first receive of the messages changes the state of the receiver. --- > But exactly-once delivery is not only possible, it's trivial. Considering that many in the field consider this problem to be impossible (or, at best, extremely difficult, e.g. https://www.confluent.io/blog/exactly-once-semantics-are-possible-here… this should be a huge red flag to yourself that you're missing something. Everyone has blind spots and that's okay, but…

I think it depends but to add some noise to the discussion: People really abused kafka: https://www.confluent.io/en-gb/blog/publishing-apache-kafka-new-york-t… like really abused it. Kafka is hard to use, has lots of rough edges, doesn't scale all that easily, and isn't nice to use as a programmer. but you can make it do lots of stupid shit, like turn it into a database. People tried to use message queues for synchronous stuff, or things that should be synchronous, and realised that queuing those requests is a really bad idea. I assume they went back to REST calls or something. Databases are much much faster now, with both SSD, design and fucktonnes of ram. postgres isn't really the bottleneck it once was. SQS and NATS cover most of the design usecases for pure message queues (as in no half arse RPC or other feature tacked in) and just works. Message queues are brilliant, I use them a lot, but only for data processing pipelines. But I only use them to pass messages, not actual data. so I might generate a million messages, but each message is <2k could I use a database? probably, but then I'd have to make an interface for that, and do loads of testing and junk.

As someone who’s spent a lot of time working on data pipelines, I think this is a great breakdown of the complexity most data engineers are facing. However, I think there’s two more keys to tidying up messy pipelines in practice: 1. You need to colocate both stream processing for the data pipeline and real-time materialized view serving for the results. 2. You need one paradigm for expressing both of these things. Let me try to describe a bit why that is. 1. You virtually always need both stream processing and view serving in practice. In the real-world, you ingest data streams from across the company and generally don’t have a say about how the data arrives. Before you can do the sort of materialization the author describes, you need to rearrange things a bit. 2. Building off of (1), if these two aren’t conceptually close, it becomes hard to make the whole system hang together. You still effectively have the same mess—it’s just spread over more components. This is something we’re working really hard on solving at Confluent. We build ksqlDB (https://ksqldb.io/), an event streaming database over Kafka that: 1. Let’s you write programs that do stream processing and real-time materialized views in one place. 2. Let’s you write all of it in SQL. I see a lot of people on this post longing for bash scripting, and I get it. These frameworks are way too complicated today. But to me, SQL is the ideal medium. It’s both concise and deeply expressive. Way more people are…

I prefer to see the whole system as a succession of stream transformations (https://www.confluent.io/blog/turning-the-database-inside-out-with-apa… If you view the sequence of input events as first-class and the "current state of the world" as derived, then a lot of problems go away. You need a datastore that can give you a consistent answer as to what order events occurred in, but it's a lot easier to make appending to an append-only list atomic than to make arbitrary state computations ACID. For downstream derived computations, basically you either make the causal relationship explicit, or accept that you have eventual consistency. The only case where you can have inconsistency is where you have a "diamond" in your computations (i.e. you compute B that's derived from A and C that's also derived from A, and you compute D that's derived from B and C). So you figure out the business implications and either accept it or eliminate the diamond (by computing (A, B) from A and (B,C) from (A, B) instead of computing B and C separately from A). You will also get inconsistency if you do some totally ad-hoc query that's not part of your existing pipelines, but usually that's the kind of reporting query that doesn't need to be 100% consistent; if you do need a consistent version of that then the best approach is to take a regular snapshot, which can be consistent. Basically you have a lot more precise control, you have causal relationships where you have explicit…

> I'm confused by what you mean by that. By large, I mean 10+ MByte. > Completely agree. Having hacked on and patched the code inside Celery, it's really quite a bummer. I don't understand what the point of celery is. Literally everything I do requires /some/ persistent state in the workers, and there's no way to do that with celery. > Are you talking about publishing connections? Consuming connections? One used for both? What does "stuck" mean? I'd be interested in hearing more about this. TCP connections. As in, a connection to the server from a consumer. High latency connections seem to exacerbate the issue. I think the issue is the state machines server-side and client-side get out of sync, and things just stop until the keep-alives/heartbeat cause the connection to reset, but that's a bunch of time to wait with no messages. I also ran into the issue that basically every python library had at least one or two locations where `read()` was called without a timeout, but that was at least easier to fix. > Kinda pedantic, but exactly once delivery is possible in some very restricted situations (see Kafka's implementation of this guarantee: https://www.confluent.io/blog/exactly-once-semantics-are-pos...). Exactly once processing is what's tough-née-impossible. So yeah, idempotence is great. Well, it isn't really a thing, so you at least shouldn't depend on it being a thing for your architecture if possible.

> If you have large messages and use keepalives (and you'll need keepalives), you need to write your own message fragmentation. I'm confused by what you mean by that. Do you mean "large" as in "take a long time to process in the consumer"? If so, and if your consumer is not issuing heartbeats concurrently with message processing, then that is true. > There are no python libs that just work. Completely agree. Having hacked on and patched the code inside Celery, it's really quite a bummer. I think this is because the Python libs try to abstract over things that ... just straight up can't be abstracted away given the semantics of AMQP: specifically connection-drop-detection, "resumption" of a consume (not really possible; this isn't Kafka), and the specific error code classes (connection-closed vs channel-closed vs information). > If you have a single open connection, it will get stuck from time-to-time. Are you talking about publishing connections? Consuming connections? One used for both? What does "stuck" mean? I'd be interested in hearing more about this. > exactly-once delivery isn't a thing Kinda pedantic, but exactly once delivery is possible in some very restricted situations (see Kafka's implementation of this guarantee: https://www.confluent.io/blog/exactly-once-semantics-are-possible-here… Exactly once processing is what's tough-née-impossible. So yeah, idempotence is great.

Before I proceed, let me just state that I have no particular knowledge of either RDBMS' internals or Kafka, so this is nothing but my own amateur musings on the subject. Now, I wholeheartedly agree that Kafka is more scalable, but I think the key point here is that there is no particular law of nature as to why that is the case. It may just be an historical accident of how RDBMS - and PosgreSQL in particular - have evolved. Further: many of the properties of Kafka are in fact also desirable properties for the PostgreSQL transaction log. My take on the inopinatus observation, together with the Samza article [1] mentioned on this discussion, are as follows. You can think of Postgres as two "products" (bounded contexts, if you like): - a stream-based, possibly replicated, transaction log; - a projection of that transaction log into relational calculus, plus all of the associated machinery. Thus far we never had the need to think of these as clearly separate "products", but Kafka makes it obvious that they are. In truth, the amount of tools processing WAL outside of PostgreSQL were already hinting in this direction; Kafka just made it obvious. From this perspective, it seems a tad expensive to take the original transaction log, convert it to a RDBMS representation, then convert it to events and, in some cases, then store it as an event stream in Kafka. It would be much more efficient to simply use the original transaction log directly - and this is why, to me, even…

SQS is only meant for multiple publishers and a single subscriber or set of subscribers that do the same thing. A message in an SQS queue is either delivered or in the queue. SQS operates on a strictly polling model. Meaning you’re not going to put a message directly in an SQS queue and have it processed by multiple types of subscribers. Are you expecting some generic system to make sure your subscribers are up and running and polling? Better question is, do you know what SQS is and have you ever done anything with it? This is one definition of “exactly once”. https://www.confluent.io/blog/exactly-once-semantics-are-possible-here… Exactly once semantics: even if a producer retries sending a message, it leads to the message being delivered exactly once to the end consumer. This is done with AWS FIFO queues by: Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue. So exactly once is handle by the producer, the queueing service, and the subscriber.

M U INC | Senior Data Engineer (aka "Director of I/O Bound Stuff") | Rust / Kafka | Austin, TX | ONSITE | Full Time | 100K-130K w/ EQUITY We are a small independent startup (fully funded by our MVP's proceeds) focused on managing, enriching and presenting large and complex Energy Data. Our technical capabilities translate DIRECTLY into commercial success, we are thus obsessed with coupling Engineering Excellence with Pragmatism. We strive to continuously "co-optimize" architecture and technical decisions with the company's long term business strategy. Our MVP has been extremely successful. We are actively looking for someone ready to develop and take charge of our next generation Data Platform. We've fallen in love with Martin Kleppmann's "Building Data-Intensive Applications" book and his "Turning the database inside-out..." [0] talk. You will be building a Kappa Architecture from scratch. This platform will become the data backbone of our business, supporting most of what we do. This is the stack we already have and/or envision: - Apache Kafka - Rust - LMDB/RocksDB - Docker/Kubernetes - Python You will report directly to the CEO (aka Chief Nerd Officer) and become a member of our small core team. Our success and your success will be extremely aligned as you will be making substantial contributions to our business (data is critical for us). This is an ONSITE position only here in Austin, TX. Let's talk! Email us at [email protected] [0]…

Following the BottledWater-Pg link through to the Confluence blog [0], there's a hilarious illustration of what we're doing to ourselves. The first flow chart [1] is simple. It shows a user, an app, and three separated data volumes that each serve a separate use case (db, cache, and index [assuming for OLAP workloads]). The chart is headlined with the adamant imperative "Stop doing this". Instead, Confluence suggests that we start doing this [2]: user -> app -> DB -> extraction -> Kafka -> (Index <-> Cache <-> HDFS) -> monitoring -> samza. Ehhh, no thanks. I like the other option. We need to understand that good engineering is not about making more work for ourselves. It's about simplicity and elegance, and being able to accomplish complex tasks WITHOUT wrapping ourselves up into some intractable mega-contorsion. More moving parts means more fragility and more waste. Simplicity means beauty, power, and flexibility. Now, I'm not suggesting that such architectures are never justified. I just want to highlight that the complexity should be eschewed, not celebrated. If you find yourself writing a blog post that converts a simple 3-step process into a complex 5-step, 9-destination process, alarm bells should be ringing, and you should be talking about why your organization (see Conway's Law) and/or the state of computer science sucks so bad that the 3-step process isn't good enough. [0]…