Redpanda Blog

I think a few things explain these kinds of projects 1. There are a lot of Agentic Data Plane startups for knowledge workers(not really for coders[1] but for CFOs, Analysts etc) going up. e.g https://www.redpanda.com/ For people to ask "Hey give me a breakdown of last year's sales target by region, type and compare 2026 to 2025 for Q1". This can be done entirely on intranet — by agents or humans — but as someone pointed out the intranet can also be a dangerous place. 2. People who have skills — like the one OP has with wasm (I guess?) — are building random infra projects for enabling this. 3. All the coding people are getting weirded out by its security model because it is ofc not built for them. [1] As I have commented elsewhere on this thread the moment a coder does webfetch + codeexec its game over from security perspective. Prove me wrong on that please.

It's interesting how Redpanda went with Benthos. I'm curious to see how this helps integrate with Meroxa. (I remember them talking about that integration when their serverless offering came out back in March.) Edit: Nice! Found this in the related press release: "Meroxa is excited to partner with Redpanda to bring multi-language SDK support for the Redpanda Connect ecosystem via our popular open source project Conduit," said DeVaris Brown, founder and CEO of Meroxa. "As an existing Redpanda customer, we are excited to deepen our partnership to bring to the world the best developer experience for the connector ecosystem with Meroxa and Redpanda Connect." https://redpanda.com/press/redpanda-acquires-benthos

I've been looking at RedPanda[0] for a new project. Any ideas on that? [0] https://redpanda.com/

I've heard good things about Red Panda[1], but I haven't tried it. [1] https://redpanda.com/

Following up - https://redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka… Try this on your laptop see global data loss - hint: multi-az is not enough Regardless of the replication mechanism you must fsync() your data to prevent global data loss in non-Byzantine protocols.

Check out http://redpanda.com for a simpler to manage, and 10x faster Kafka

Right, and with all due respect I'd like to leave those in the past which is why I didn't mention not forgetting them! NATS is a lively project, has never had anything to do with mules or camels, and isn't hard-tied to concepts like ESB. It deserves to be mentioned in 2022. But here are some projects I left out that do deserve to be mentioned I think: - RedPanda[0] - Apache Pulsar[1] [0]: https://redpanda.com/ [1]: https://pulsar.apache.org/

Went searching for https://redpanda.com/ and couldn't find it in the DB, fyi :) Not affiliated, just saw the Jepsen report posted here a few weeks ago.

A complete nit, but the testimonial from the CTO of The Seventh Sense on https://redpanda.com/ spells Redpanda as "Redpand".

Redpanda We have lot of open roles in engineering. We're fully remoo. We have people in 17 countries right now. I'm personally hiring SRE in APAC (we need coverage in those timezones) and Infosec. Check us out: https://redpanda.com Jobs page: https://redpanda.com/careers/

For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be interesting to test them together on the performance benchmarks. I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus). It's the very architecture of Kafka itself due to the notorious head of line problem (check the top most comments [4]. [1] Redpanda – A Kafka-compatible streaming platform for mission-critical workloads (120 comments): https://news.ycombinator.com/item?id=25075739 [2] Redpanda website: https://www.redpanda.com/ [3] Kafka vs. Redpanda performance – do the claims add up? (141 comments): https://news.ycombinator.com/item?id=35949771 [4] What If We Could Rebuild Kafka from Scratch? (220 comments): https://news.ycombinator.com/item?id=43790420

I hope you like this blog post illustrating how Generative AI can be integrated into streaming data environments using just 45 lines of code, leveraging DataSQRL and Redpanda Data. DataSQRL is a tool that simplifies building data applications on streaming data. What's remarkable is how it enables handling complex streaming data processes in a surprisingly concise manner - the entire implementation showcased in the blog is only 45 lines of code! Additionally, it explores Redpanda Data, a streaming data platform that serves as an efficient alternative to Apache Kafka. The post talks about the technical implementation but hints at broader applications in predictive analytics and real-time decision-making across various industries. For those who are into AI, data streaming, or just love efficient coding, this might be a compelling read. If anyone here has streaming data and AI, please let me know your thoughts. Originally posted on RedPanda's website https://redpanda.com/blog/real-time-clickstream-gen-ai-aws-datasqrl

Redpanda Data (formerly Vectorized) | Remote | Full-time | https://redpanda.com/careers/ Come shape the future of streaming! Redpanda, a Kafka-compatible event streaming platform. With a compelling combination of no data loss, operational simplicity, and consistent performance at scale, Redpanda takes advantage of modern hardware to deliver the modern streaming platform for mission critical workloads. We are a team of doers, seasoned engineers, hackers and builders, working on the future of streaming. A fully distributed, diverse team, with members of Medellin - Colombia, London, San Francisco, St. Petersburg - Russia, NYC, San Diego, Germany, Canada, Poland and growing! Redpanda is looking for the following roles: C++ Core Engineer Developer Support Engineer Distributed Systems Engineer Information Security Engineer Lead Engineer, WASM Site Reliability Engineer Test Infrastructure Engineer Director, Product Marketing Sr. Engineer, Developer Content Sr./Principal Product Marketing Manager Product Manager To apply, visit https://redpanda.com/careers/ click on the role, then APPLY FOR THIS JOB

No. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1]. [0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-k… [1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-nee…

Redpanda has been a superior wire-compatible alternative to Kafka for years. https://www.redpanda.com/compare/redpanda-vs-kafka

Maybe a good time to consider alternatives https://www.redpanda.com/compare/redpanda-vs-kafka

Redpanda community edition operates under the BSL license - https://www.redpanda.com/blog/open-source IANAL, but it looks pretty open source to me.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpanda.com/blog/write-caching-performance-benchmark 2 - https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-up-after-a… 3 - https://kafka.apache.org/documentation/#brokerconfigs_log.flush.interv…

Most critical to me seems to be the integration relicensing/de-open-sourcing (and the article seems generally to feel the same), > Started relicensing some of the most critical integrations and connectors as proprietary2 under a completely different license But left unsaid is which integrations got relicensed. I'm very curious to know! Ok, from the Redpanda announcement, seems to be Splunk & Snowflake connectors that they have moved to enterprise plan features. I'm not sure this is exhaustive but I tend to think it is. Source: https://redpanda.com/blog/redpanda-connect It does make me wonder & think, perhaps there's too monolithic an architecture if moving two connectors out of core & having bentho-snowflake and bentho-splunk forked off is too hard. Does the entire project really need a fork?

Disclaimer, I work at Redpanda. Redpanda Community Edition is licensed with the Redpanda Business Source License (BSL). The core features are free and source-available. This license was inspired by MariaDB and CockroachDB and "for 99.999% of users, restrictions will not apply". The big restriction is that users "cannot provide Redpanda as a commercial streaming or queuing service to others" which is primarily to deter large cloud vendors from taking our work and impacting our ability to operate as a business. Here are some links for more details on how we came to this decision and how we license all our products and features. https://redpanda.com/blog/bsl-source-available-license https://docs.redpanda.com/docs/get-started/licenses/

My interest in it isn't with websites. Redpanda supports using wasm (https://redpanda.com/blog/wasm-architecture) so that with Sqlite can make for very fast streams enrichment. Similar for using DuckDB depending on your data.

And RedPanda uses WebAssembly as an execution engine to keep with the WASM theme here. https://redpanda.com/blog/wasm-architecture

I think a big use case is running kernels/UDFs directly inside databases or data infrastructure systems. For example, [0]. Imagine you need a custom function in postgres, but instead of implementing it in C and shipping a shared library, you would implement it in WASM. [0] https://redpanda.com/blog/wasm-architecture

The blog for a deeper dive into results, fixes, and discussion on the write behavior in the kafka protocol. https://redpanda.com/blog/redpanda-official-jepsen-report-and-analysis…

Redpanda Data | REMOTE | Full-time | https://www.redpanda.com/careers Redpanda is pioneering the Agentic Data Plane (ADP) - a new category in AI infrastructure that makes it simple and secure to connect AI agents with enterprise data and systems. Built on a multi-modal data streaming engine, Redpanda empowers agentic applications that reason and act in real-time with speed, autonomy, and precision. Global leaders including Activision Blizzard, Cisco, Moody's, Texas Instruments, Vodafone and 2 of the top 5 banks in the U.S. rely on Redpanda to process hundreds of terabytes of data a day. Backed by premier venture investors Lightspeed, GV and Haystack VC, Redpanda is a diverse, people-first organization with teams distributed around the globe. Software Engineer, Core Enterprise - https://job-boards.greenhouse.io/redpandadata/jobs/4652843005 Software Engineer, Core SQL - https://job-boards.greenhouse.io/redpandadata/jobs/4659296005 AI Agent Engineer - Poland - https://job-boards.greenhouse.io/redpandadata/jobs/4678043005

I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion: * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol. * The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie… To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support: -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/; -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka); -- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco); -- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/); -- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/). * WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered…

Redpanda | Full-time | Remote | https://redpanda.com/careers Come shape the future of streaming! We are the startup behind Redpanda, a Kafka-compatible event streaming platform. With a compelling combination of no data loss, operational simplicity, and consistent performance at scale, Redpanda takes advantage of modern hardware to deliver the modern streaming platform for mission critical workloads. Redpanda is looking for the following roles within engineering: * Analytics Engineer - https://jobs.lever.co/redpanda.com/6f52eb53-b4ca-4763-8c68-dba7279bc77… * Backend Software Engineer, Cloud - https://jobs.lever.co/redpanda.com/36d3b98e-533e-41be-822b-f3f0c093480… * C++ Core Engineer - https://jobs.lever.co/redpanda.com/b2f26147-9b9f-4088-9867-ccfdb97cc78… * Developer Experience Engineer - https://jobs.lever.co/redpanda.com/913d6556-d827-4c1a-9a4f-2b37e9b095c… * Developer Productivity Engineer - https://jobs.lever.co/redpanda.com/c9780ee7-036b-4ccc-975f-388fd4c42c7… * Developer Support Engineer - https://jobs.lever.co/redpanda.com/5eb54061-289b-4e38-8855-4fe5b08514a… * Engineering Manager, Cloud - https://jobs.lever.co/redpanda.com/59ab1690-8469-45a1-9b95-aafc5c82f64… * Engineering Manager, Core - https://jobs.lever.co/redpanda.com/efac50ea-a925-4fe6-badc-84c1199bf59… * Information Security Engineer - https://jobs.lever.co/redpanda.com/9f45fd56-443b-4887-a8b8-fdf5be6c484… * Site Reliability Engineer -…