Feldera Blog

Oh this very much reminds me of [feldera](https://feldera.com) — they do incremental loads and computations using some novel approaches (most of which i am too dumb to follow). Really nice folks too.

Feldera (https://feldera.com | REMOTE (US) | Full-Time RELIABILITY AND PERFORMANCE ENGINEER: https://jobs.ashbyhq.com/feldera/709c14e4-1fa9-46b4-9ff8-0783e137ed78 - Strong background in systems engineering, performance testing, or site reliability engineering. - Fluency in Python and Linux fundamentals. Rust experience is a plus. - Experience with distributed systems and database concepts (consistency, fault tolerance, transactions). - Experience with CI/CD/Infrastructure as Code: GitHub Actions, Docker, Kubernetes. - Hands-on experience running large-scale and long-running workloads, preferably in a cloud-native environment. - Curiosity, rigor, and the ability to design experiments that simulate messy real-world conditions. SOLUTION ENGINEER: https://jobs.ashbyhq.com/feldera/544aff74-263f-4749-a4d0-af07d75cd404 - 5+ years experience in solution architect, customer engineering or solution engineering roles. - Strong background in distributed systems, databases, cloud infrastructure, and modern data platforms. - Experience with data-intensive systems in production (e.g., Kafka, Delta Lake, Iceberg, Kubernetes, monitoring/observability stacks). - Exceptional debugging and problem-solving skills, especially in customer-facing contexts. - Excellent communication skills, both for customer-facing and internal interactions. - Ability to write and maintain high-quality technical docs and playbooks. https://jobs.ashbyhq.com/feldera Feel free to email…

The Differential Datalog team founded Feldera: https://www.feldera.com/ They switched from differential Datalog to differential SQL, I think because they realized Datalog is a really tough sell.

The correct way to think about the problem is in terms of evaluating joins (or any other queries) over changing datasets. And for that you need an engine designed for *incremental* processing from the ground up: algorithms, data structures, the storage layer, and of course the underlying theory. If you don't have such an engine, you're doomed to build layer of hacks, and still fail to do it well. We've been building such an engine at Feldera (https://www.feldera.com/), and it can compute joins, aggregates, window queries, and much more fully incrementally. All you have to do is write your queries in SQL, attach your data sources (stream or batch), and watch results get incrementally updated in real-time.

We use an in-product profiler (discussed here: https://www.feldera.com/blog/introducing-feldera's-visual-profiler… along with CPU profiles to identify where in the code we're spending time.

This triggered some associations for me. Strongest was Cells[0], a library for Common Lisp CLOS. The earliest reference I can find is 2002[1], making it over 20 years old. Second is incremental view maintenance systems like Feldera[2] or Materialize[3]. These use sophisticated theories (z-sets and differential dataflow) to apply efficient updates over sets of data, which generalizes the case of single variables. The third thing I'm reminded of is Modelica[4], a language where variables are connected by relations (in the mathematical sense). So while A = B + C can update A on when B or C change, you also can update just A and B, then find out what C must have become. [0] https://cells.common-lisp.dev [1] https://web.archive.org/web/20021216222614/http://www.tilton-technolog… [2] https://www.feldera.com [3] https://materialize.com [4] https://modelica.org

Rust has long compilation times because it doesn't prioritise compilation times. What does [0] have to do with the borrow checker? [0] https://www.feldera.com/blog/cutting-down-rust-compile-times-from-30-t…

There are lots of options to stream data out of Postgres, including: - https://electric-sql.com (disclaimer: co-founder) - https://feldera.com - https://materialize.com - https://powersync.com - https://sequinstream.com - https://supabase.com/docs/guides/realtime/broadcast - https://zero.rocicorp.dev Etc.

Definitely not a skill issue if you read the details in their post - https://www.feldera.com/blog/the-pain-that-is-github-actions - they're clearly very experienced with using GitHub Actions and have run into legitimate challenges due to the complexity of what they're using it for.