Robinhood Engineering
robinhood.engineering · Company blog · on HN since 2017
Robinhood's engineering team writes about its data infrastructure, including Airflow, Kafka, Kubernetes, and Faust, plus mobile development.
What HN saysDuring an outage, commenters used the blog to infer Robinhood's stack and cited its Kubernetes deployment post when guessing at scalability causes.
19HN people shared it
4linked it in comments
15stories submitted
36story points
4 years agolast shared
2017people per year2026
Most-shared posts
- 1Why Robinhood Uses Airbnb's Airflow2 people · 4 points · first shared 2017 · discussion on HN
- 2Building an application deployment platform on kubernetes f6af367d1e052 people · first shared 2020
- 3Faust: Stream Processing for Python1 people · 11 points · first shared 2018 · 1 comments on HN
- 4Beautiful animations using Android ConstraintLayout1 people · 3 points · first shared 2017 · discussion on HN
- 5How We Built a Better News System at Robinhood1 people · 3 points · first shared 2018 · discussion on HN
- 6We scaled Robinhood’s brokerage system for greater reliability1 people · 2 points · first shared 2021 · discussion on HN
- 7Upgrading and Scaling Airflow at Robinhood1 people · 2 points · first shared 2019 · discussion on HN
- 8Robinhood – Scaling Confidently with the Load and Fault Team1 people · 2 points · first shared 2021 · discussion on HN
- 9Robinhood: The old guard of investing is at it again1 people · 2 points · first shared 2021 · 2 comments on HN
- 10Fresher Data Lake on AWS S31 people · 1 points · first shared 2022 · discussion on HN
- 11Increasing Efficiency and Confidence with Project Apollo at Robinhood Engg1 people · 1 points · first shared 2020 · discussion on HN
- 12Robinhood: Scaling and Load Testing1 people · 1 points · first shared 2022 · discussion on HN
- 13Elk at Robinhood1 people · 1 points · first shared 2021 · discussion on HN
- 14Tracking Temporal Data at Robinhood (2017)1 people · 1 points · first shared 2022 · discussion on HN
- 15Engineering Onboarding – Built to Scale1 people · 1 points · first shared 2021 · discussion on HN
What HN people wrote when they linked it
ceohockey60 · 2020-03-03
Since it's Day 2, def appears to be an architectural scalability issue, not some bugs. Quick scan of their engineering blog shows the following stack:
-Zookeeper, Kafka, Spark, Airflow, Faust (Kafka Streams in Python)
-Kubernetes (was/is SALT + Terraform)
-AWS Aurora, PG, ES, Influx
-Presto via AWS Athena, Redshift
-homegrown solutions for a data lake & managing k8s microservices
Fwiw, their team does appear to have a rigorous internal post-mortem process (SEV reviews) [0] drawn from industry best practices.
Let's hope a public one will be made when issues are resolved. (FINRA will be asking for one anyway.)
Not piling on. I genuinely feel for all the infra engineers staying up all day/night fixing this.
[0] https://robinhood.engineering/creating-a-sev-process-that-scales-with-…
astannard · 2020-03-10
you might be right:
https://robinhood.engineering/building-an-application-deployment-platf…
takeda · 2020-03-10
They recently had a blog about moving to kubernetes
https://robinhood.engineering/building-an-application-deployment-platf…
tedmiston · 2018-08-29
Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS, and also contribute to Airflow.
IMO the pros / cons are really relative to what you're comparing it to and what your workflow needs are.
We've written some guides on "Airflow vs ___" [1] (currently AWS Glue and Oozie). Feel free to email me (in profile) if you'd like to see a comparison to something else.
I like these two posts on Airflow vs Luigi, Pinball, Azkaban, etc [2][3].
[1]: https://www.astronomer.io/guides/
[2]: http://bytepawn.com/luigi-airflow-pinball.html
[3]: https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8