Apache Flink Blog

You should let the Apache Flink team know, they mention exactly-once processing on their home page (under "correctness guarantees") and in their list of features. [0] https://flink.apache.org/ [1] https://flink.apache.org/what-is-flink/flink-applications/#building-bl…

Yeah, you definitely need backpressure, and doing that well is had. There's a great blog post about how Flink manages this [0] and it involves a complex credit-based flow control system. But in practice the performance advantages of allowing operators to do work as they have capacity (rather than being potentially starved by the capacity of their downstreams) outweighs this. [0] https://flink.apache.org/2019/06/05/a-deep-dive-into-flinks-network-st…

Along similar lines, I found Flink's blog posts to be good reading. Some examples: https://flink.apache.org/news/2015/12/04/Introducing-windows.html https://www.ververica.com/blog/session-windowing-in-flink https://flink.apache.org/news/2020/07/30/demo-fraud-detection-3.html

https://github.com/apache/incubator-seatunnel https://flink.apache.org/

You should check out Apache Flink. It does a bunch of those things that Spark doesn't, though it's also missing a few things that Spark has. https://flink.apache.org/

> Somewhat like Map/Reduce was quickly replaced by Spark, which has no real competitors. I wouldn't say that - what about Flink? (https://flink.apache.org/)

For stateful processing you need something like: * https://kafka.apache.org/documentation/streams/ - Kafka Streams (and its KTable) * https://flink.apache.org/ - Flink * https://spark.apache.org/streaming/ - Spark Streaming * https://github.com/asavinov/bistro - Bistro Streams The focus of Benthod seems to be performance and resilience (implemented without persistence).

Flink's SQL implementation actually follows ANSI SQL - which I think is very important. There is a way of interpreting streams and tables to make ANSI SQL meaningful in the presence of streams, which we follow [1]. The big advantage is (besides not having to learn another syntax and retaining compatibility with SQL tools and dashboards) that this seamlessly handles the batch (bounded/static input) and streaming (unbounded/continuous) use cases with literally the same SQL statement. [1] http://flink.apache.org/news/2017/04/04/dynamic-tables.html

Flink's event-time support is coming along nicely. Their first round of true event-time support came in November (https://flink.apache.org/news/2015/11/16/release-0.10.0.html), and much more is on the way. Flink will be an excellent platform for Beam, both batch and streaming. As I understand it, Spark has event-time support coming soon as well. I think basic stuff is landing in 1.7. Not sure precisely what they have planned, but I can only imagine that Spark will also become an excellent platform for executing streaming Beam pipelines in due time. In the meantime, the streaming runner for Spark can either target those features which Spark does support well (i.e., processing-time windowing, in this case), or try to emulate those it doesn't (such as how it was done in the article).

Restate is built as a sharded replicated state machine similar to how TiKV (https://tikv.org/), Kudu (https://kudu.apache.org/kudu.pdf) or CockroachDB (https://github.com/cockroachdb/cockroach) are designed. Instead of relying on a specific consensus implementation, we have decided to encapsulate this part into a virtual log (inspired by Delos https://www.usenix.org/system/files/osdi20-balakrishnan.pdf) since it makes it possible to tune the system more easily for different deployment scenarios (on-prem, cloud, cost-effective blob storage). Moreover, it allows for some other cool things like seamlessly moving from one log implementation to another. Apart from that the whole system design has been influenced by ideas from stream processing systems such as Apache Flink (https://flink.apache.org/), log storage systems such as LogDevice (https://logdevice.io/) and others. We plan to publish a more detailed follow-up blog post where we explain why we developed a new stateful system, how we implemented it, and what the benefits are. Stay tuned!

I'm in the same boat - looking to consolidate a few smaller data pipelines that have organically grown over the years into a single pipeline and taking the opportunity to try and get the org to rethink how we emit events in various parts of the stack. As teams operate fairly independently, we need something that enables a flexible data structure whilst still enabling the ability to join data on common well-defined fields. The current path we are going down is looking at using a 'nested' JSON structure that enables sub-processes in the various systems to inherit values from the parent. Something similar to: { "type": "schemaX", "version": 1, "payload": { "k1": "v1", "type": "schemaY", "version": 3, "payload": { "kk1": "v1" } } } The structure itself will be documented using JSONSchema and hopefully we will be able to verify the validity of events as they are processed, though this might be too expensive to do in high volume scenarios. The goal is to then build a low latency router that takes in routing policies to forward subsets of events to further data pipelines. The policies themselves will be defined using some kind of DSL (possibly using JSONPath?). As a whole, this seems to be a fairly common problem[0] that other companies are trying to solve, but a lot of the low level details are not spoken about. One thing that does seem to be a common component to a service like this is Apache Flink.[1] [0] Netflix Keystone -…

Apache Flink does provide end-to-end exactly-once guarantees when coupled with data sources and data sinks that participate in its checkpointing mechanism. See: - An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!) — https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-… - Flink's Fault Tolerance Guarantees — https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connec…

flink uses akka for control plane kind of stuff. data plane stuff, like piping data between tasks, doesn't. check out https://flink.apache.org/2019/06/05/flink-network-stack.html for some (maybe outdated) details.

> why Flink and not Kafka ? I assume you mean "Kafka Streams" when you say this, as Kafka is just a event bus and Flink can be used to read messages from it. The biggest advantage of Flink (IMO) is that you can write the Flink logic once, and reuse it for both batch and stream processing. So if I write a Flink job that consumes a Kafka stream and produces some aggregated outputs, that same job can be run against my data lake in S3/GCS/Azure Blob Storage, etc. Kafka Streams does not support batch processing, or working on top of anything other than Kafka. Flink supports building on top of other message buses like Pulsar as well: https://flink.apache.org/news/2019/11/25/query-pulsar-streams-using-ap…

I think this is the equivalent to Flink's statefun: https://flink.apache.org/stateful-functions.html These are comparable to triggers on distributed tables under the hood, with a function addressing scheme and an ingress/egress story. Compared to Flink, on the surface, I'd say these lack the integration with regular stream processing operators. But anyway this is very refreshing to see! These are the future of large scale backend apps, with a clear and simple scaling path.

How is this different from Apache flink real time SQL support? https://flink.apache.org/2020/07/28/flink-sql-demo-building-e2e-stream…

Not sure why the Flink github site got linked. Here's a list of companies using it in production, with links to details. https://flink.apache.org/poweredby.html

I always hate when announcements (especially for a 1.0 release) don't start off with "This is what this is!" http://flink.apache.org/index.html

Hi DannoHung, Maybe this article could be of some value to you: http://flink.apache.org/news/2015/02/09/streaming-example.html Btw. handling DataStreams and DataSets in Flink is done via almost similar API calls. In DataStreams you only have to additionally insert the Time-Window parameters. Everything else (transformations like filter, map, flatMap etc.) remain the same. Kind regards, Harris

I'm sorry, this hasn't been my experience, or the experience of anyone I know writing stream processing systems. Having written one each over a GC and not (Naiad, and the timely dataflow in Rust linked above), dev time on the former was disproportionately spent fighting GC perf issues. The folks I know doing stream processing over the JVM have garbage collection (and avoiding it) as perf issue #1. For example, you can read about Flink's approach to avoiding GC at https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.… You can read a more scientific discussion of GC vs region allocation here: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-g…

Many commenters seems to state ah, if company makes $25M it can afford to pay. First, certain amount of revenue, even if it looks high for you, does not mean it is able to pay ANY amount vendor require. Paying Oracle $50.000 per CPU Core to run Oracle (for 100s of thousands of CPU cores) would be steep even for Facebook/Meta This idea what there are those large evil companies do not contributes to Open Source, and there are same large VC funded copyright holders are just struggling to survive is quite wrong - it is not that simple and it is Open Source which is one of the ways which create balance of power Second, it is not about money. Having any restriction on who can use it makes it impractical for using as component of other Open Source software, which Is where I think Open Source power really shines. Check out this article for example https://flink.apache.org/news/2022/09/08/akka-license-change.html

Don't forget about the Java machine learning & deep learning "renaissance": Deeplearning4j https://deeplearning4j.org/, Konduit Serving https://serving.konduit.ai/, Amazon's open-source DJL https://djl.ai/ Apache Mahout https://mahout.apache.org/ Apache Arrow https://arrow.apache.org/ TensorFlow can run on any JVM for building, training and running machine learning models. They have created recently https://github.com/tensorflow/java PyTorch also supports inference on the JVM but you still need to train with Python. https://www.infoq.com/news/2020/02/pytorch-releases-java-bindings/ I think something like Scala will be used in the future instead of Python even for prototyping. Streaming is also big business on the JVM: Apache Kafka https://kafka.apache.org/ (80% of Fortune 100 companies use it) Apache Flink https://flink.apache.org/ Apache Storm https://storm.apache.org/ Hard to think that .NET ecosystem will ever reach the ecosystem of Java when it comes to open-source.