Engineering at Meta
engineering.fb.com · Company blog · on HN since 2019
Meta's engineering blog, where its engineers describe infrastructure, messaging, open-source releases, and large-scale systems work.
What HN saysHN links it for outage post-mortems, the F14 hash map writeup, and WhatsApp and Messenger engineering. Commenters also use it to argue about Meta's AI spending.
657HN people shared it
365linked it in comments
528stories submitted
16Kstory points
4 months agolast shared
2019people per year2026
Most-shared posts
- 1Fighting spam with Haskell at Meta (2015)14 people · 101 points · first shared 2020 · 82 comments on HN
- 2Making messaging interoperability with third parties safe for users in Europe14 people · 28 points · first shared 2024 · 1 comments on HN
- 3Code Verify: An open source browser extension for verifying code authenticity12 people · 72 points · first shared 2022 · 28 comments on HN
- 4How WhatsApp enables multi-device capability11 people · 192 points · first shared 2021 · 137 comments on HN
- 5Project LightSpeed: Rewriting the Messenger Codebase11 people · 5 points · first shared 2021 · 1 comments on HN
- 6Homepage10 people · first shared 2019
- 7Open-sourcing a more precise time appliance9 people · 498 points · first shared 2021 · 182 comments on HN
- 8Scaling Mercurial at Facebook (2014)9 people · 10 points · first shared 2021 · discussion on HN
- 9Threads: Meta’s Newest Social App9 people · 10 points · first shared 2023 · 1 comments on HN
- 10Sapling: A new source control system with Git-compatible client8 people · 1.0K points · first shared 2022 · 524 comments on HN
- 11ZippyDB: Facebook's general purpose key value store8 people · 15 points · first shared 2021 · discussion on HN
- 12Reducing Instagram’s basic video compute time by 94 percent8 people · 14 points · first shared 2022 · 1 comments on HN
- 13Building Meta's GenAI infrastructure7 people · 664 points · first shared 2024 · 303 comments on HN
- 14More details about the October 4 outage7 people · 470 points · first shared 2021 · 285 comments on HN
- 15Programming languages endorsed for server-side use at Meta7 people · 243 points · first shared 2022 · 298 comments on HN
What HN people wrote when they linked it
tencentshill · 2026-06-12
They might do a post-mortem on https://engineering.fb.com. They have before.
https://engineering.fb.com/2021/10/05/networking-traffic/outage-detail…
conradkay · 2026-05-23
The numbers for Meta are pretty misleading, I assume the $3 billion is something like direct generative AI revenue?
Sure they're torching money on building consumer LLMs, but they seem to be doing very well optimizing things like ad ranking
https://engineering.fb.com/2025/11/10/ml-applications/metas-generative…
https://engineering.fb.com/2026/03/31/ml-applications/meta-adaptive-ra…
woadwarrior01 · 2025-07-19
I'd recommend reading the Swiss table design notes[1] in the Abseil documentation. You might also like F14 maps[2] from Folly.
[1]: https://abseil.io/about/design/swisstables
[2]: https://engineering.fb.com/2019/04/25/developer-tools/f14/
JohnScolaro · 2025-04-04
I was wondering that too!
Surely Airbnb - a company that runs a website - has the capability to put a text post on their own website. Then they'd own the content, and people looking for it could find it easier? It's not a revolutionary concept either, Facebook has one:
https://engineering.fb.com/
Wait, Airbnb already has one?
https://airbnb.io/
Genuinely confusing.
rockwotj · 2025-03-31
> Anytime Meta, Oracle, Microsoft, or anyone else announces another AI bit barn, a juicy power purchase agreement usually follows. Meta's mega DC being birthed in the bayou was announced alongside a 2.2GW gas generator plant — so much for those sustainability and carbon neutrality pledges.
Ouch, I grew up where Meta created its first data center and while it is very efficient (https://engineering.fb.com/2011/04/14/core-infra/designing-a-very-effi… I remember a more recent article about how they are zero net emissions (https://tech.facebook.com/engineering/2021/11/10-years-world-class-dat… I wonder if they have kept that in this AI rush (along with most of the datacenter companies).
EDIT: a direct link to net zero for Meta: https://tech.facebook.com/ideas/2020/9/facebooks-path-to-net-zero/
ipsum2 · 2025-03-07
> You’ll note that nobody is open sourcing their ad recommender
Actually... (2019) https://ai.meta.com/blog/dlrm-an-advanced-open-source-deep-learning-re…
Source code:
https://github.com/facebookresearch/dlrm
Paper:
https://arxiv.org/abs/1906.00091
Updated 2023 blog post, but solely for content recommendation, but ads recommendation is ~90% the same:
https://engineering.fb.com/2023/08/09/ml-applications/scaling-instagra…
It's a little out of date, but the internal one is built with the same concepts, just more advanced modeling techniques and data.
gruez · 2025-02-21
Map and more details here: https://engineering.fb.com/2025/02/14/connectivity/project-waterworth-…
ChrisArchitect · 2025-02-18
ForHackernews · 2025-02-11
> Facebook creates tech to mitigate the pathologies their past continuously present.
Remember when they hacked a running Android Dalvik machine because their organizational constraints were such that they could never remove code or delete unused classes?
https://archive.is/nIPlg
https://engineering.fb.com/2013/03/04/android/under-the-hood-dalvik-pa…
Facebook seems like a place where they do amazing engineering to temporarily stave off the disastrous consequences of their previous feat of amazing engineering.
8fingerlouie · 2024-10-22
It's such a great little protocol, and besides being "small enough" for hobby projects, it also scales enough to be used for something like Facebook Messenger [1]
[1]: https://engineering.fb.com/2011/08/12/android/building-facebook-messen…
crabmusket · 2024-10-15
Classic article on this: https://engineering.fb.com/2015/06/26/security/fighting-spam-with-hask…
otterley · 2024-08-25
> In the last 48 hours, Facebook also released a paper of it being excellent at RCA.
Do you have a link? The most recent information (not a paper) I could find was back in June: https://engineering.fb.com/2024/06/24/data-infrastructure/leveraging-a…
This is the HN discussion to which you may be referring: https://news.ycombinator.com/item?id=41326039
er4hn · 2024-08-15
One interesting thing is that if you look at what companies that want to prepare for Store Now Decrypt Later are doing (see links at bottom) they're pretty much all using the non-production ready OQS. If you believe in hybrid encryption this is mostly okay since a failure in the PQC portion should not cause a breakage in the classical portion. Assuming that OQS has implemented the hybrid protocol correctly.
- https://www.microsoft.com/en-us/research/project/post-quantum-tls/ - https://engineering.fb.com/2024/05/22/security/post-quantum-readiness-… - https://blog.cloudflare.com/kemtls-post-quantum-tls-without-signatures
zem · 2024-05-07
i can't, unfortunately, it was youtube internal code. but by the same token a lot of the performance stuff was specific optimisations for that code and would not really generalise.
one of the current "extremely large python codebase" projects is instagram, and they do have some public repos, notably monkeytype (https://github.com/Instagram/MonkeyType) which youtube did have its own analogue of, and cinder (https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagra… in general facebook's engineering blog is a great place to read about this sort of thing.
riffic · 2024-04-10
recently shipped and it'll be complete when you can follow fedi accounts from Threads but my feeling is it will be much more restricted in that direction:
https://engineering.fb.com/2024/03/21/networking-traffic/threads-has-e…
lelandfe · 2024-03-25
Great story.
> Once all the thumbnails were pulled... it would start downloading full sized images
LQIP, Low-Quality Image Placeholder – that was pretty new stuff circa 2014. Facebook also popularized fetching a tiny version, and blurring it: https://engineering.fb.com/2015/08/06/android/the-technology-behind-pr…
mackid · 2023-12-31
Mark Russinovich gives a good talk most years on the internals of Azure and the systems that run it. [1] is an example. Look for talks from other years as well.
Meta also publishes a number of papers/blogs/OSS projects on their engineering site [2]
James Hamilton of AWS gives a talk most years on their infrastructure. Worth watching multiple years [3].
[1] https://youtu.be/69PrhWQorEM?si=u7vh_Um6SQNoyeFH
[2] https://engineering.fb.com/category/data-center-engineering/
[3] https://youtu.be/AyOAjFNPAbA?si=nFRJVcQI4EiamC-O
btown · 2023-09-12
Instagram's backend is a Django monolith "with several million lines of code and a few thousand Django endpoints" as of 2019, and the same architecture was used for Threads! They now run Django on a JIT-enabled fork of CPython, and leverage a ton of internal tooling for static analysis and strong typing.
https://instagram-engineering.com/types-for-python-http-apis-an-instag…
https://engineering.fb.com/2023/09/07/culture/threads-inside-story-met…
https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagra…
https://engineering.fb.com/?s=django
Django is also excellent for e-commerce with projects like https://github.com/saleor/saleor - or you can roll your own e-commerce system quite easily, which we've done at my travel startup.
anayak · 2023-07-06
While that is technically correct, they haven't been running vanilla PHP for quite some time - it's jitted into x86 and run natively on the machine. It's also pretty extensively optimized. And the VM that does the jitting is all C++. Their MySQL deployment is also an internal fork that's been heavily modified for scalability (storage/compute separation, sharding, RocksDB based storage engine instead of InnoDB, Raft for synchronous replication, etc.). Lots of really fantastic engineering work has gone into making their backend infra scale.
Some worthwhile references:
[1] https://engineering.fb.com/2016/08/31/core-data/myrocks-a-space-and-wr…
[2] https://research.facebook.com/file/529018501538081/hhvm-jit-a-profile-…
[3] https://research.facebook.com/file/700800348487709/HHVM_ICPE_2023-(1).…
jitl · 2023-05-19
This seems like offshoot tech from the Facebook messenger rewrite that claimed to lower as much as possible to the database for cross-platform sharing - https://engineering.fb.com/2020/03/02/data-infrastructure/messenger/
The proposed architecture seems cool, until you realize that instead of writing the languages your mobile devs know, instead you end up writing a lot of C (or generating a lot of C) to be cross platform, and the overlap of C developers and mobile / consumer-oriented developers in Silicon Valley is quite slim. I heard through the grapevine that the success of that rewrite was greatly overstated and it turned into a pain quite quickly.
All that said, I have a cross-platform app using SQLite… and I’m okay with C… maybe I can con the mobile team at my company into using this to ensure more job security?
dgl · 2023-03-05
I’d really like to see something like https://engineering.fb.com/2022/03/10/security/code-verify/ become a more standard part of browsers. i.e. a standard way to go from sub-resource integrity to “full integrity”.
Having it as an extension just moves the weak point to the extension update mechanism.
Of course apps can just distribute themselves as an extension which is probably the best compromise right now and something most web based password managers for example already offer.
theobeers · 2023-01-08
As of last year, Meta recommends internally that CLI tools be implemented in Rust:
https://engineering.fb.com/2022/07/27/developer-tools/programming-lang…
samora · 2022-12-26
This is great! I think Facebook Lite already uses something like this. Great work.
https://engineering.fb.com/2016/03/09/android/how-we-built-facebook-li…
jillesvangurp · 2022-11-22
The article actually addresses Kotlin. They'd love to switch to it but they just can't do it overnight because they have so much mission critical Java code. So, this is a stop gap solution for legacy code. They published another article some time ago how they are switching to Kotlin: https://engineering.fb.com/2022/10/24/android/android-java-kotlin-migr…
Migrating millions of lines of code is a non trivial effort. They'll be stuck with bits of Java for quite some time. So, this helps make that less painful.
KronisLV · 2022-11-18
> At Meta we call an individual set of changes made to the codebase a “diff.”
I don't work at Meta and I still don't have any idea which of you is right.
A commit would satisfy the description above.
As would a pull/merge request, which the image in the article perhaps fits better: https://engineering.fb.com/wp-content/uploads/2022/11/Code-Reviews-at-…
Of course, it's possible to review each individual commit as well, but who does that?
I know that some folks love to mess around as much as they need in their local branches, commit often and then do an interactive rebase, to maybe end up with one or just a few commits that can then be the basis for a pull/merge request, then the difference matter a little bit less.
hu3 · 2022-10-25
Not quite. According to a more recent article, Facebook's PHP, Hack, is still a primary language:
> For business logic and relatively stateless applications, the Hack ecosystem has the highest level of automation and support at Meta and is the recommended language.
On a related note, their Rust usage has grown significantly:
> For performance-sensitive back-end services, we encourage C++ and Rust. Rust is a new addition to this list. There’s a rapidly increasing Rust footprint in our products and services, and we’re committing to Rust long-term and welcome early adopters. For CLI tools, we recommend Rust. This is a new recommendation for this year.
https://engineering.fb.com/2022/07/27/developer-tools/programming-lang…
lopkeny12ko · 2022-10-16
Have you never seen https://engineering.fb.com/? Engineers there blog about their tech tools all the time. "Legal peril" sounds like a bit of a stretch.
WookieRushing · 2022-10-09
It wasn’t when I last saw it. Rocksandra ended up being a stepping stone to fbs most common distributed db, zippydb https://engineering.fb.com/2021/08/06/core-data/zippydb/
Zippydb is honestly one of the best parts of fb infra. It let you select levels of consistency vs latency
hobofan · 2022-07-28
Which makes sense given yesterdays announcement that Rust is now an endorsed server-side language at Meta and Rust the recommended language for CLI tools: https://engineering.fb.com/2022/07/27/developer-tools/programming-lang…
vlovich123 · 2022-03-15
The flakiness recommendation isn't tenable at reasonable scale. All tests are flaky to some degree. People frequently set timeouts - if your CI infra is overloaded tests, the assumptions about the timeouts will fail. Additionally, the number of non-flaky tests trends to 0 as the number of tests & engineers increases.
FB did a fantastic job here (had first hand experience before & after this system):
https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-fl…
Wish there were products out there to be able to plug this into existing CI.