Astronomer Blog

https://github.com/astronomer https://astronomer.io Open source Airflow platform on Kunernetes with commercial support. Best Airflow product on the market by far.

Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS [1][2], and also contribute to Airflow. Our experience was similar to boulous' — Airflow is awesome once it's running but getting it running in an environment that scales to the point that you can deploy your first production DAG can take some effort. That's what led us to trying to do that work once for everyone. Reusability and composability of components are some of my favorite aspects of working with Airflow. [1]: https://www.astronomer.io/ [2]: https://github.com/astronomerio/helm.astronomer.io

The fact that most new programming services and products are oriented towards C-suite types, and not to the people who actually use them. Most websites looks like [1] or [2] which are full of corporate-friendly buzzwords but don't help me understand what they actually do or how they work. To get a concrete understanding I need to go to github and find repos that actually use the product to even understand what it's for. [1] https://www.astronomer.io/ [2] https://tailscale.com/

Astronomer | Senior Software Engineer | US REMOTE | Full-time | https://www.astronomer.io/ At Astronomer, our R&D team is dedicated to providing an exceptional experience in managing Apache Airflow at scale. As a leading player in the industry, we are seeking two experienced Software Engineers to work on our data observability product - Astro Observe. For more details and to apply: https://jobs.ashbyhq.com/astronomer/da3426d9-9da9-47e5-aadc-76102254fb… https://jobs.ashbyhq.com/astronomer/0435e8cd-03a8-4253-ac1f-cb5a8f61ea…

Or you know, this little company called astronomer https://www.astronomer.io ...

I love SMIL animations. If you're curious about what you can do, here's a few examples: https://www.pcmaffey.com/roll-your-own-analytics https://www.pcmaffey.com/finally-i-closed-my-linkedin https://www.astronomer.io/404.html (let this play > 1 minute) You can open any of these images and View Source to dive in to the implementation... except for Edge, SMIL plays exactly the same in all browsers, whereas I've found using CSS to drive the animations has varied results (especially in Safari) and worse performance.

We use airflow mainly for ETL work, backfills, and batch processing. We do a lot of work with clickstream type data, be it taking data from analytics.js and loading it into redshift, or taking analytic data from redshift and loading into Google analytics. We have developed many pipes and connectors for airflow that allow us to connect to many data sources, both at the source and sink ends. I mainly work on the DevOps team, running our infrastructure and working on back end systems, so my knowledge of airflow is more high level. I just keep the system up so it can run ;P FYI I work for a startup out of Cincinnati, OH called Astronomer, you can find us at https://www.astronomer.io Also, we weren't like all down, we just saw lots of time out issues when reading/writing to S3.

Yeah that ending is a bit mysterious. In the end, Romeo decided not to move forward. It wasn't price; just not a great fit. The experience did prompt us to change our back-end from a heavy reliance on AWS services to more standard big data technology: http://www.astronomer.io/blog/sneak-peak-at-our-new-backend

i have a really good friend who did the whole UI/UX design bootcamp during the explosion of UX/UI jobs. he did okay, he's probably hopped jobs 2-3 times now and is now without a job. i actually feel for him, it's definitely one of the career paths that's looked at as excess/waste now while companies slim up to reappropriate money for AI. but i do think there was something there, he was genuinely passionate about what he did and it's just really hard to find work doing it now. i feel guilty saying this but i've let him talk me through some of what he does, show me how he sees and approaches design (the bulk of what he did was design the interfaces for publicly used webapps and mobile apps) and... idk. i feel like it's all acquired taste and almost a "good app developer will think of these things when they design the front end" and a lot of his insight to me broadly looked like a lot of stuff i would've considered myself as a mere sidequest and my general thought process to deliver a good app. the difference is im building the app and designing the user experience, but his entire career is silo'd to just building the user experience. im not against breaking out the design to a dedicated resource whether thats one designer or one team who wants to try and maintain a consistent language for a company. i think this has upside to make the design experience not locked to a single developer or developer team, and opens it up to a lot more channels of input. but on the other hand,…

DBT in simplest terms is just a way to orchestrate transformations in dependency order. Table "A" needs to be updated before table "B," because "B" selects from "A." It works well for that. I've seen it used to get wild west SQL logic into version control. To replace scheduled SQL workbooks running entire data pipelines. I've also seen over-engineered integrations with other orchestration tools in the "stack." Once a DBT project grows to a certain size, roughly 500-1000 .SQL files, it gets hard to manage. Not impossible, though it takes intentionality about how to group things together and scale them out operationally. "Slim CI" is a recent buzzword that has some nice ideas about build automations. Cosmos is supposed to solve a lot of the automation gripes with dbt. I haven't tried it yet but would like to. https://www.astronomer.io/cosmos/

At Airbnb, we were using SubDAGs to try to manage large number of tasks in a single DAG. This allowed organizing tasks and drilling down into failures more easily but came with its own challenges. In more recent versions of Airflow, TaskGroups (https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.ht… https://www.astronomer.io/guides/task-groups/ ) were made to help this a little bit. Hopefully that helps a bit. At ~1k nodes in the graph introspection becomes hard anyway, as others have suggested, breaking it down if possible might be a good idea.

astronomer.io | Everything | Fully Remote | Full Time We're the commercial developers behind Apache Airflow! If you are looking to build on a K8s native stack around one of the most popular open source projects around, please shoot us a note! https://www.astronomer.io/careers/

Here's an article from the same source: https://www.astronomer.io/blog/introducing-airflow-2-0/

Astronomer | https://astronomer.io | Remote | Full-Time Astronomer is the commercial developer of Apache Airflow, a community-driven open-source tool that’s leading the market in data orchestration. We’re a globally-distributed and rapidly growing venture-backed team of learners, innovators and collaborators. Our mission is to build an Enterprise-grade product that makes it easy for data teams at Fortune 500’s and startups alike to adopt Apache Airflow. As a member of our team, you will be at the leading edge of the industry and build solutions for some of the world's toughest data problems. All roles are listed at https://jobs.lever.co/astronomer?lever-via=pXKhCU4CVs and the three below are specific to improving data engineer's developer experience. Feel free to reach out via email at julian [at] astronomer.io. - Senior Software Engineer, Infrastructure: https://jobs.lever.co/astronomer/eb8cf141-a6f2-4207-b299-ee6b27aa0967?… - Senior Software Engineer, Product Development: https://jobs.lever.co/astronomer/62250bee-2727-4941-8e2d-8065734a9712?… - Software Engineer, UI/UX: https://jobs.lever.co/astronomer/cefedf9a-91fa-490b-87df-9e0abae5c538?…