data Artisans Blog

data Artisans | ONSITE (Berlin, Germany) Hiring Process: application, remote interview, on-site interview. data Artisans is building the next-generation platform for programming data-intensive applications. We are tackling some of the biggest technical challenges in big data and data streaming today. We love open sourcing our code, and working together with the open source community that has formed around Apache Flink, one of the biggest projects of the Apache Software Foundation in its category. Positions: - Software Engineer (https://data-artisans.workable.com/jobs/241346) - Support Engineer (https://data-artisans.workable.com/jobs/241347) - Community Evangelist (https://data-artisans.workable.com/jobs/241348) Benefits: - Pick your own tech gear - Competitive salary and stock options - Flexible working hours and vacation More details here: http://data-artisans.com/careers

data Artisans | ONSITE (Berlin, Germany) and REMOTE Hiring Process: Application, Phone interview, and On-site interview What: Data processing, Big Data data Artisans is building the next-generation platform for programming data-intensive applications. We are tackling some of the biggest technical challenges in big data and data streaming today. We love open sourcing our code, and working together with the open source community that has formed around Apache Flink, one of the biggest projects of the Apache Software Foundation in its category. Positions: - Software Engineer (https://data-artisans.workable.com/jobs/241346) - Senior Data Engineer (https://data-artisans.workable.com/jobs/325667) - Software Engineer (Support) (https://data-artisans.workable.com/jobs/241347) Benefits: - Competitive salary and stock options - Flexible working hours and vacation - Pick your own tech gear More details here: http://data-artisans.com/careers

"data Artisans Streaming Ledger [...] [extends] Apache Flink [...] to perform serializable transactions from multiple streams across shared tables and multiple rows of each table." [1] This sounds like a hybrid approach if you need streaming to handle the speed and volume of data, but still want to use ACID on a small subset of events. LMAX [2] architecture may be another approach to consider for your application. [1] https://data-artisans.com/wp-content/uploads/2018/08/2018-08-31-dA-Str… [2] https://martinfowler.com/articles/lmax.html

Counting views/impressions in combination with Apache Kafka sounds like the ideal use case for a stream processor like Apache Flink. It supports very large state which can be managed off-hand. This should enable you to count the exact number of unique views in real time with exactly once semantics. Here is a blog post on large scale counting with more details. It also includes a comparison with other streaming technologies like Sanza and Spark: https://data-artisans.com/blog/counting-in-streams-a-hierarchy-of-need… Also check out this blog post by a Twitter engineer on counting ad impressions: https://data-artisans.com/blog/extending-the-yahoo-streaming-benchmark

Google's take: https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-googl… Data Artisan's take: http://data-artisans.com/why-apache-beam/

True. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.

Here: https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-c… http://data-artisans.com/extending-the-yahoo-streaming-benchmark/

Note that this proposal is being back not only by Google, but also Cloudera, Data Artisans, Talend, Cask, PayPal, ... Some other posts on the announcement: http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and-open-sour… http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-googles-dat… http://data-artisans.com/dataflow-proposed-as-apache-incubator-project… http://blog.cask.co/2016/01/cask-anticipates-googles-dataflow-to-flour…

This is pretty cool stuff. I have been working with Apache Flink where the computational model is exactly once (which is still hugely useful) but it still comes with caveats about duplicate messages downstream, which makes writing to downstream db or having multiple flink processes tricky. Often times this is done by windowing data and producing complete snapshots of that window (i.e. you need to keep all the state for a given window) such that the message downstream is idempotent and then aggregating windows together (which can be non trivial for certain aggregations). Once flink (and other systems) support this, I think it will really be a game changer. It will allow for doing things like sending an increment downstream rather than needing to keep all that state. Instead of mostly seeing stream processing as a thing we do for analytic use cases, it can really become the backbone for streaming applications. Event Sourcing/CQRS has been an idea for a while, but in practice, is difficult to do because to the inherent difficulty in dealing with message semantics and consistency (see https://data-artisans.com/blog/drivetribe-cqrs-apache-flink for a good write up about such an app). The ability to independently optimize for both reads and writes while also not having to make all messages be idempotent, in my mind, will make this feasible for a broad range of teams and won't require a huge amount of work in thinking up clever message formats or working through as many failure…