U
Uber Engineering Blog
eng.uber.com · Company blog · on HN since 2015
Uber's engineering team writes about its infrastructure, datastores, spatial indexing, and machine learning systems.
What HN saysCommenters call the articles surprisingly good and recommend them for production ML and engineering topics, and cite a paper on Uber's merge queue.
525HN people shared it
223linked it in comments
547stories submitted
11Kstory points
10 months agolast shared
2015people per year2026
Most-shared posts
- 1Why Uber Engineering Switched from Postgres to MySQL32 people · 876 points · first shared 2016 · 294 comments on HN
- 2H3: Uber’s Hexagonal Hierarchical Spatial Index (2018)17 people · 346 points · first shared 2018 · 74 comments on HN
- 3Why Uber Engineering Switched from Postgres to MySQL (2016)14 people · 140 points · first shared 2020 · 129 comments on HN
- 4Rethinking GPS: Engineering Next-Gen Location at Uber13 people · 97 points · first shared 2018 · 18 comments on HN
- 5Designing Schemaless, Uber Engineering’s Scalable Datastore Using MySQL (2016)12 people · 142 points · first shared 2016 · 33 comments on HN
- 6Genetic algorithms for training deep neural networks (2017)12 people · 123 points · first shared 2017 · 20 comments on HN
- 7Fast and Reliable Schema-Agnostic Log Analytics Platform12 people · 19 points · first shared 2021 · 1 comments on HN
- 8Uber Engineering Blog12 people · 2 points · first shared 2015 · discussion on HN
- 9How We Built Uber’s Highest Query per Second Service Using Go10 people · 508 points · first shared 2016 · 129 comments on HN
- 10The Road to UChat: Building Uber’s Internal Chat Solution10 people · 38 points · first shared 2017 · 12 comments on HN
- 11Cherami: Uber Engineering's Durable and Scalable Task Queue in Go10 people · 29 points · first shared 2016 · discussion on HN
- 12Evolving Distributed Tracing at Uber Engineering9 people · 5 points · first shared 2017 · discussion on HN
- 13Data Race Patterns in Go8 people · 313 points · first shared 2022 · 186 comments on HN
- 14Uber Open Sources Pyro, a Deep Probabilistic Programming Language7 people · 155 points · first shared 2017 · 22 comments on HN
- 15Large-scale, semi-automated Go GC tuning7 people · 149 points · first shared 2021 · 104 comments on HN
What HN people wrote when they linked it
angryGhost · 2022-04-11
https://eng.uber.com/
has great articles covering many topics
jpgvm · 2022-01-01
Cloudflare etc that others have already posted are excellent by your definition.
Uber blogs are surprisingly good. https://eng.uber.com/
Or more technical and less business goal focussed:
https://jepsen.io/analyses
https://lemire.me/blog/
fsavard · 2018-10-25
It's a theme I like to read about (I mean "practical issues around ML in production settings"). I find company blogs and some research publications are great resources. Examples:
- https://eng.uber.com/
- https://code.fb.com/
and many more. Google also publishes papers on various engineering practices obviously, some ML-related, but I can't find a blog where they focus on that specifically.
Also it's not "to keep up to date", but there's a great paper (from Google) that's often cited:
Machine Learning: The High Interest Credit Card of Technical Debt
https://ai.google/research/pubs/pub43146
It talks about issues you face over the long run (I've experienced some of those). It
also provides interesting pointers for further reading, e.g. about "pipeline jungles".
If others have pointers, I'm curious to hear about them as well.
Gwypaas · 2022-06-18
People feel like they have a short learning curve and they master Go. Greenfield programming in Go is an amazingly empowering feeling. The issue is they don't see the issues they create until something implicit, a default value after a refactor or any of Go's paper cuts lead to a production issue.
One central issue is that anything touching concurrency and Go is a tangle of nastiness with implicit effects. But apparently Go is "natively concurrent" and everyone simply prays that the race detector is good enough.
https://eng.uber.com/data-race-patterns-in-go/
jd__ · 2022-04-21
We know Uber has some great internal tooling (there's even a paper published on their merge queue: https://eng.uber.com/research/keeping-master-green-at-scale/). We had various Uber engineers using Mergify on open source repositories those last years. I didn't check recently if that was still the case, I'll make sure our lis is not too much outdated.
I definitely agree with your view on how pipelines can be different. I think none of our customers has something that is exactly the same. However, many of them don't have the workforce to build in house or even to optimize as far as you would do in a (very) large company.
jeremy_k · 2022-04-21
It is probably a huge leap to think about using Bazel, but I found Uber's blog post about how they manage this issue after setting up a monorepo a really good read - https://eng.uber.com/go-monorepo-bazel/
anshumankmr · 2021-12-21
There blog seems to be up:https://eng.uber.com/
Perhaps that can suffice in the mean time
nojito · 2021-11-03
The biggest issue with comparisons and “benchmarks” is that almost no one uses clickhouse like this.
Good example is what Uber is doing
https://eng.uber.com/logging/
Or alinity’s great materialized view tutorials.
Clickhouse is unmatched with these workflows.
amzans · 2020-11-23
Hey that's great! I think everyone should do what works for them.
Using a tool "just because" it worked for others, seems to be what often drives many decisions in this industry. For example, would you go for the 2,200+ microservice architecture from Uber[1] just because they say it solved all their problems? They probably have their reasons, and they are unlikely to be the same as everyone else.
I know my stack as a solo founder is unconventional, but it's what has worked best for me. And I'm happy with my choice.
If serverless does the trick for you, then focus on that, and keep building the features that your customers actually care about.
[1] https://eng.uber.com/microservice-architecture
throwaway189262 · 2020-10-19
Sure! Uber uses it for failiver instead of load balancing but same concept https://eng.uber.com/eng-failover-handling/
simonkafan · 2020-07-02
I recently visited Uber's engineering blog (https://eng.uber.com/) and I can't help but it seems they have engineers goofing around with new technology completely useless for their core business ("Look, we now also use meta learning like all the hip kids!")
A company that is swamped with ad revenue money may can afford this but for the rest (especially when they are VC backed) there will come the time when they have to ask themselves if they really need 4000 engineers to maintain the service that brings in the money (= a few apps and a cloud based backend)
jmalicki · 2020-06-12
postgres works great for `n` in ranges in excess of (10k, 10B, 1k) at least, which not many would consider small.
The only real justification I've heard is postgres' story on replication still lags mysql, and the write amplification bit which they've done a lot of work on for pg13 (see [0])
Even for many very large companies I've seen, noSQL databases are mostly used as caches that can afford to be delayed or a bit out of sync, not as the database of record.
[0] https://eng.uber.com/postgres-to-mysql-migration/
datadem · 2020-05-19
Uber actually has some of the best technical writing in [datasci/causal inference](https://eng.uber.com/causal-inference-at-uber/).
stared · 2020-03-07
The lottery ticket hypothesis is IMHO the single most interesting finding for deep learning. It explains why does deep learning works (vs shallow neural nets), why is initial over-parametrization is often useful, why deeper is often better than shallow, etc.
I recommend for an overview:
- the original paper "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", https://arxiv.org/abs/1803.03635
- "Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask" https://eng.uber.com/deconstructing-lottery-tickets/ showing that if we remove "non-winning tickets" before the training, the trained network still works well
paxys · 2019-08-08
Uber as a product is very technologically complex, mainly because of the scale it operates at. You can read about some of their tech at https://eng.uber.com.
__exit__ · 2019-05-22
The Uber Engineering Blog[0] is the first thing that that comes to my mind.
It has posts describing optimizations and approaches they took in certain subsystems.
[0] https://eng.uber.com/
bitforger · 2019-05-17
Follow up from ICLR really elucidates some of the questions raised by this:
https://eng.uber.com/deconstructing-lottery-tickets/
https://arxiv.org/abs/1905.01067
slack3r · 2019-02-19
Yes. Deep Learning definitely needs a new programming language. Uber's Pyro is worth checking out.
(https://eng.uber.com/pyro/)
krosaen · 2019-02-18
Very cool!
If you like hexagons, and boy do I like hexagons, I recommend also checking out Uber’s Hexagonal Hierarchical Spatial Index
https://eng.uber.com/h3/
PaulHoule · 2019-02-06
I love the fanatical approach to high speed over a bad network -- fitting messages into 1 MTU is a real protip.
This article
https://eng.uber.com/new-rider-app/
talks about their main app and the talk about architecture is fascinating because it has the "factory factory" that builds the app and then the part that interacts with the API is kept separated from the UI code. I keep converging on solutions like that whenever I write GUI code because otherwise you will lose control sooner or later.
goodmachine · 2018-09-29
Me too! Some followup here on why not much happened:
https://www.reddit.com/r/MachineLearning/comments/2t5ozk/what_ever_hap…
Incidentally, GA's (specifically, novelty search/NEAT) are creeping back into the picture again:
https://eng.uber.com/deep-neuroevolution/
Groxx · 2018-07-26
Worth noting that mattermost has made pretty significant improvements around this in the past several months, now that there's a large customer: https://eng.uber.com/uchat/
web/apps have been pretty much completely rewritten too. it's noticeably better, if you haven't used it for a while.
ofrancon · 2018-03-22
A few links you can look at if you're interested in neuroevolution, from the same group of researchers:
Ken Stanley and Risto Miikkulainen original NEAT (NeuroEvolution of Augmenting Topologies) paper: http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
Ken Stanley's novelty search page, and a link to his book, "Why Greatness Cannot Be Planned: The Myth of the Objective": http://eplex.cs.ucf.edu/noveltysearch/userspage/
Risto Miikkulainen's Evolving Deep Neural Networks paper: https://arxiv.org/abs/1703.00548
Ken Stanley & team's work at Uber, with links to some recent papers: https://eng.uber.com/deep-neuroevolution/
noncoml · 2018-01-05
> Lol. If that's true (and I'm not disputing it) then how would switching to MySQL fix that problem?
It's explained in the blog the great-grandparent linked:
https://eng.uber.com/mysql-migration/
gregdoesit · 2017-12-01
Uber | Back-end, Android, iOS | Amsterdam, Netherlands | FULL-TIME ONSITE VISA
Uber's Amsterdam engineering office is looking for back-end, Android and iOS engineers for the following teams:
* Payments: building the future of payments for on-demand services. We own a very large chunk of the payments stack end-to end in this office, which opens exciting opportunities.
* Mobile platform: build tooling that will make our engineers more productive (and happier!)
Learn more about our team at http://t.uber.com/amsterdam
Interested? Shoot [email protected] an email if interested.
--
About me: I'm an engineering manager started as an engineer here at Uber, in Amsterdam. My first project was rewriting the Uber app (https://eng.uber.com/new-rider-app/) and the last interesting piece of code I committed was open sourcing the architecture behind it (https://github.com/uber/RIBs).
Diederich · 2017-11-03
Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling. It was designed with these key principles:
Universal: Pyro can represent any computable probability distribution.
Scalable: Pyro scales to large data sets with little overhead.
Minimal: Pyro is implemented with a small core of powerful, composable abstractions.
Flexible: Pyro aims for automation when you want it, control when you need it.
Check out the blog post for more background or dive into the tutorials.
https://eng.uber.com/pyro/
jamesmishra · 2017-07-24
Hey frik, former Uber engineer here. I never worked on Jaeger, but I loved using it at Uber.
If you want to get started, the documentation is at http://jaeger.readthedocs.io/en/latest/
We also have a blog post that describes the history of Jaegar and some of the design decisions. https://eng.uber.com/distributed-tracing/
Jaegar is compatible with the OpenTracing standard ( http://opentracing.io/documentation/pages/supported-tracers ) and is also compatible with the Zipkin backend and wire protocol ( http://zipkin.io/ ).
LegNeato · 2017-03-14
I think the best people to speak to this would be the Uber folks, as I believe they still use both via okbuck (https://github.com/uber/okbuck):
https://eng.uber.com/ios-monorepo/
jeevand · 2017-01-21
If you are an engineer, go interview with them.
https://eng.uber.com/
616c · 2016-07-26
Love them or hate them (I am in the latter category), but observe even Uber mentions some of their heavier backend stack for internal usage runs on Dropwizard.
"The highest-volume service in this area is Gurafu, which provides a set of utilities for working with road map data and improves efficiency and accuracy by providing more sophisticated routing options. Gurafu is fronted by µETA, which adds a business logic layer on top of raw ETAs (things like experiment group segmentation). Both Gurafu and µETA are web services built on the DropWizard framework."
https://eng.uber.com/tech-stack-part-one/
https://news.ycombinator.com/item?id=12152346