Dagster Blog

The most time and energy has been getting my head around the source data [0] and industry-specific nuances. In terms of stack I have a self-hosted Dagster [1] data pipeline that periodically dumps the data onto Cloudflare R2 as parquet files. I then have a self-hosted NodeJS API that uses DuckDB to crunch the raw data and output everything you see on the map. [0] Mostly from https://bmrs.elexon.co.uk/ [1] https://dagster.io/

I have a side project at the moment to use LangChain to help build a Q&A system from domain specific documentation on a product I own. Its around 500 different PDFs. I was going to follow this guide* which looks credible and solves for how to deal with large corpus of texts. Curious what others think as to approach. * https://dagster.io/blog/chatgpt-langchain

I've had good success with https://dagster.io, which is much more opinionated about your pipelines, including properly typing inputs and outputs.

Data pipeline / ETL orchestration. Prefect: https://www.prefect.io Dagster: https://dagster.io

I have worked with Airflow during the past three years, but recently we adopted Dagster and I have been using it for the past 3 months. I have found it quite joyful to use and the experience has been very positive. Its main advantages compared to Airlfow (IMO): - A great UI - It forces you to clearly define inputs, outputs and types. - Separation of concerns: Between configuration and data, between processing and IO, and between code and deployment options. - It allows you to define flexible dags which you can configure at runtime, which makes it easiy to run locally or in k8s, or to switch the storage backend depending on the environment. This blog post by the founder outlines the differences between the two in much more detail: https://dagster.io/blog/dagster-airflow

Author here. Agreed — I think adding some comparisons to other database partitioning strategies would be helpful. My 2 cents, specifically about manually partitioning Postgres/MySQL (rather than using something like Citus or Vitess): SQLite-on-the-server works similarly to Cassandra/DynamoDB in how it partitions data. The number of partitions is decoupled from the number of databases you're running, since data is automatically rebalanced for you. If you're curious, Dagster has a good post on data rebalancing: https://dagster.io/glossary/data-rebalancing. With manual partitioning, compared to automatic partitioning, you end up writing a lot of extra complex logic for: - Determining which database each piece of data lives on (as opposed to using partitioning keys which do that automatically) - Manually rebalancing data, which is often difficult and error-prone - Adding partitions manually as the system grows - (Anecdotally) Higher operational costs, since matching node count to workload is tricky Manual partitioning can work fine for companies like Notion, where teams are already invested in Postgres and its tooling. But overall, I think it introduces more long-term problems than using a more naturally partitioned system. To be clear: OLTP databases are great — you don’t always need to reach for Cassandra, DynamoDB, or SQLite-on-the-server depending on your workload. But I do think SQLite-on-the-server offers a really compelling blend of the developer experience of…

What is the difference to https://dagster.io/ ?

Also not to be confused with https://dagster.io/

We used Copybara to setup a hub-and-spoke model at Dagster where public-facing repos can live within our larger internal monorepo, and while it worked we had to do a lot of hacky things. https://dagster.io/blog/monorepos-the-hub-and-spoke-model-and-copybara

* https://dagster.io/blog/fake-stars (2023) - Tracking the Fake GitHub Star Black Market with Dagster, dbt and BigQuery * https://arxiv.org/abs/2412.13459 (2024/2025) - Six Million (Suspected) Fake Stars in GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware

For bigger workflows, this declarative pattern is already implemented by orchestrators like Dagster, Flyte, and recently Airflow; e.g., https://dagster.io/blog/declarative-scheduling [fixed]

We have been using pex to deploy code to running docker containers in ECS to avoid the cold start delay. Cuts down the iteration loop time for development significantly. https://dagster.io/blog/fast-deploys-with-pex-and-docker

Dagster Labs | Software Engineers, Product Managers, Designers | SF, Minneapolis, NYC or Remote | Full-time | Open-source Dagster is the data orchestration platform built for productivity. Data engineers all around the world, in every industry, use Dagster to build scalable and maintainable data applications like ETL pipelines, ML training pipelines, data integrations and similar systems. Data applications are the true core of AI and ML training systems, and our goal is to make Dagster the de facto standard for structuring these systems. We’re an early stage, well-funded startup team with a proven track record of shipping open source software with global adoption. We put a premium on respectful, clear, and complete communication, and we expect each other to be creative, curious, effective, and empathetic. More info: https://www.dagster.io/careers

On Dagster’s website there are comparisons to dbt. Is it complementary (as positioned in this stack) or competitive? Dagster versus dbt https://dagster.io/vs Migrating off dbt cloud https://dagster.io/blog/migrate-off-dbt-cloud

Snowflake isn't local, you have to pay for cloud ... Airflow is Airflow, complexity, steep learning curve, there is a whole industry trying to be as/more powerful than Airflow with less complexity and cleaner integration with modern things you might want to do like k8s, although of course Airflow is still super popular and powerful. the Dagster folks have some comparisons, of course there are popular modern alternatives other than Dagster https://dagster.io/blog/dagster-airflow https://dagster.io/vs/dagster-vs-airflow

SEEKING WORK | Los Angeles | Remote I'm a senior Python engineer and developer marketing content writer for data and infra companies. Most recently I wrote this piece on building an outbound email pipeline with Dagster https://dagster.io/blog/outbound-reporting-pipeline.

On a related note, the dagster team did a similar search for fake stars: https://dagster.io/blog/fake-stars.

At least it's better than buying fake stars https://dagster.io/blog/fake-stars

If anyone is tempted to do this, here is a good blog post on how to build your own retrieval-based LLM powered conversational agent. https://dagster.io/blog/chatgpt-langchain

I've already built this and you can use this as a guide, being PII we just need to swap openai for a model on huggingface.co, this let's you do the question answering part run locally. https://dagster.io/blog/chatgpt-langchain

You can actually train the model to have access only to a specific body of text, like a documentation. Very useful article explaining this approach: https://dagster.io/blog/chatgpt-langchain

Dagster already offers Dagster Cloud which came out a month or so ago[1]. It's a fully hosted solution for teams that don't want to run their own. [1] https://dagster.io/cloud

https://www.dagster.io/blog/software-defined-assets Of course, nothing stops Airflow or other tools fron thinking this way as well.

Funny enough this post mirrors quite a bit of our thinking over at Dagster! https://dagster.io/blog/rebundling-the-data-platform

We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless. There are a number of blog posts echoing a similar critique[1]. Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages. According to their README: https://github.com/apache/airflow#approach-to-dependencies-of-airflow > Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server? [1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-a… - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-fl… -…