Prefect Blog

https://www.prefect.io Not the most SEO-friendly choice of name. Great product though.

The article resonates a lot with me. I've been seeing the transition from Jenkins to Azure DevOps / GitHub Actions (same thing more or less) in the company I'm working at and came to very similar conclusions. The single big Jenkins machine shared by 10+ teams mixing UI configuration from 20 plugins with build systems and custom scripts wasn't great, so it was the right decision to move away from it. However, neither great is the current workflow of write->commit->wait->fail->write... while figuring out the correct YAML syntax of some third party GitHub Action that is required to do something very basic like finding files in a nested folder by pattern. Take a look at Prefect - https://www.prefect.io/ - as far as I can see, it ticks a lot of the boxes that the author mentions (if you can live with the fact that the API is a Python SDK; albeit a very good one that gives you all the scripting power of Python). Don't be scared away by the buzzwords on the landing page, browsing the extensive documentation is totally worthwhile to learn about all of the features Prefect offers. Execution can either happen on their paid cloud offering or self-hosted on your own physical or cloud premises at no extra cost. The Python SDK is open source. Disclaimer: I am not affiliated with Prefect in any way.

Data pipeline / ETL orchestration. Prefect: https://www.prefect.io Dagster: https://dagster.io

Prefect | Software Engineers & Solutions Engineers| Remote, full-time | https://www.prefect.io/about/careers/ I'm Chris from the Prefect team. We're hiring senior software engineers on the open source and platform teams, as well as solutions engineers on our go to market team and would love for you to join us! Ideal candidates will be familiar with the data ecosystem, and more specifically with Python, Kubernetes, and working with large open source communities. We have an engaging remote-first culture and a variety of excellent benefits including competitive salary, free lunch every day, $300 per month in remote work expenses, $1k in education expenses per year, 401(k), health insurance, and much more. Apply at https://www.prefect.io/about/careers/

Want to shout out an alternative Python open source workflow orchestration tool, Prefect https://www.prefect.io/ It has a server/UI component that you can deploy relatively easily on something like Kubernetes. It then makes it easy to configure your flows to run on varying amounts of compute resources.

Agreed, Glue turned out to be very underwhelming. We ended up moving ETL flows to self-hosted Prefect (https://www.prefect.io/) instead. In general, AWS excels at a lot of core features (EC2, S3, the databases) but the higher-level services feel very thrown-together, with awful documentation. They are launching a large amount of half-baked services these days, (intending to capitalize on vendor lock-in?), but it's making the ecosystem start to look like a confusing mess. A lot of these wouldn't survive if they were marketed as standalone products.

Airflow is still a very good option, though there are more competitors on the market. Two options are: Prefect - https://www.prefect.io/ Flyte - https://flyte.org/

I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few: http://airflow.apache.org/ https://www.prefect.io/ https://dagster.readthedocs.io/en/0.6.7/ https://www.getdbt.com/

https://www.prefect.io/ is also pretty easy.

Prefect can also run on Dask https://www.prefect.io/

It definitely feels like there is Airflow inspiration though poking through the site and docs, it seems that the devil is in the details with respect to the differences between the two... kinda like the relationship between Airflow and Prefect [1]. It looks like the barrier to entry is higher with Flyte but that there are benefits as well. One example I see is that one can run Airflow simply without containers if desired with just simple Python functions whereas Flyte seems to be much more concerned with managing the execution environment for you (pros/cons to that). Flyte also seems to be more "Kubernetes native" by default [2][3] vs with Airflow this is more of a choice amongst several executors. I'd be curious to see a performance benchmark using comparable workflows vs Airflow with the Kubernetes Executor or Kubernetes Operator. [1]: https://www.prefect.io/ [2]: https://kccncna19.sched.com/event/UaYY/flyte-cloud-native-machine-lear… [3]: https://www.youtube.com/watch?v=KdUJGSP1h9U

- Yes. it is a service and at least the runner will stay like that for the time being. - We are not quite live yet, but the pricing model is based on compute capacity and it is divided in tiers (e.g. small=50GB for concurrent scans=$1500/month, large can get up to a TB). infinite queries, infinte jobs, infinite users. The idea is to have a very clear pricing with no sudden increases due to volume. - You do not have to swap your storage - our runner comes to your S3 bucket and your data never ever have to be anywhere else that is not your S3. - You do not have to swap your orchestrator either. Most of our clients are actually using it with their existing orchestrator. You call the platform's APIs, including run from your Airflow/Prefect/Temporal tasks https://www.prefect.io/blog/prefect-on-the-lakehouse-write-audit-publi… Does it help?

Disclaimer: I work for Prefect. It looks like we added cron and other schedule types to the deployment CLI just under a month ago[1]. Over the last couple of releases, we've also made it easier to pull deployments from GitHub or bake your flow code into Docker images instead of needing S3-like storage. As with any product, there's always more to do, so I appreciate you sharing your thoughts. More than anywhere else I've worked, community feedback is a huge driver of product enhancements and feature development. Feel free to join our Slack community[2] if you'd like to share more feedback or ask questions. [1] https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#210 [2] https://www.prefect.io/slack/

I'm also curious about this. The folks I hear about Temporal from seem to be very disjoint from Airflow users, and Temporal's python client is still alpha-stage. It seems notable to me that the big Prefect rewrite mentioned elsewhere [0] leans into the same "workflow" terminology that Temporal uses. I have to wonder if Prefect saw Temporal as superceding the DAG tools in coming years and this is them trying to head that off. That post's discussion of DAG vs workflow also sounds a lot like why PyTorch was created and has seen so much success. Tensorflow was static graphs, pytorch gave us dynamism. [0] https://www.prefect.io/blog/announcing-prefect-orion/

I think you mean prefect orion/v2[0]. I'm curious too. [0] https://www.prefect.io/orion/

I'm working with some researches on about ~40k lines of Python disease modelling code and these are the things that helped us becomes more productive: - tests (unit, smoke, acceptance) (eg. https://mattsegal.dev/alternate-test-styles.html, https://understandlegacycode.com/approval-tests/) - continuous integration (via GitHub actions eg: https://mattsegal.dev/pytest-on-github-actions.html) - 10x performance improvements via cProfile (https://julien.danjou.info/guide-to-python-profiling-cprofile-concrete… - getting fast feedback on code changes by visualizing outputs with Streamlit (https://www.streamlit.io/) - input parameter validation (eg. https://pydantic-docs.helpmanual.io/, https://mattsegal.dev/cerberus-config-validation.html) - total automation of data processing job (eg. https://www.prefect.io/ - we don't use this but that's the kind of thing we have)

Did not know this ETL that produce GIFs by using Prefect calls: https://examples.dask.org/applications/prefect-etl.html -> https://www.prefect.io/about/company

Sorry for the confusing example. So, the AWS lambda in the data product example is a bit of a red herring, and it's used as the outer process to create branches and launch bauplan pipelines through the Python client (https://github.com/BauplanLabs/data-products-with-bauplan/blob/ba94a51… It can be your laptop, an Airflow task, a prefect flow or a step function or a cron job on a VM - it's the "host" process (for the data product we picked lambda because it's the easiest way for people to "run small Python stuff every 5 minutes" - this is a prefect example: https://www.prefect.io/blog/prefect-on-the-lakehouse-write-audit-publi… When you interact with the Bauplan lakehouse, all the compute happen on bauplan, nothing happens in the lambda: think of launching a Snowflake query from a lambda - the client is in the lambda but all the work is done in the SF cloud. Unlike many (all?) other lakehouses, Bauplan is code-first, so you can program the entire branching and merging patterns with a few lines of code, offloading the runtime to the platform. The platform itself runs on standard EC2, which contains the dockerized functions needed for execution - typically we manage Ec2 in single tenant, private link, soc2 compliant account we own for simplicity, but nothing prevents the VMs to be somewhere else (given connectivity is ok etc.). It is our philosophy that you should not worry about the…

The Prefect guys wrote this https://www.prefect.io/guide/blog/why-not-airflow it's something but far from comprehensive. There's not a lot of writing about this. Folks seem content to fight Airflow's deficiencies. Most of them are too young to know any better. The critiques you'll find are generally written by competitors, or folks adopting a competitor. Here's the big one I see lots of folks get wrong: do NOT run your code in Airflow's address space. Airflow was copied from Facebook Dataswarm and comes with a certain set of operational assumptions suitable for giant companies. These assumptions are, helpfully, not documented anywhere. In short, it is assumed that Airflow runs all the time and is ~never restarted. It is run by a team that is different from the team that uses it. That ops team should be well-staffed and infinitely-resourced. Your team is probably not like that. So instead of deploying a big fleet of machines, you are probably going to do a simple-looking thing: make a docker container, put Airflow in it, then add your code. This gives you a single-repo, single-artifact way of deploying your Airflow stuff. But, since that's not how Airflow was designed to work, you have signed yourself up for a number of varieties of Pain. First, you are now very tightly coupled to Airflow's versioning choices. Whatever version of Python runs Airflow, runs your code. Whatever versions of libraries Airflow uses, you must use. This is bad. At one point I supported a data…