LinkedIn Engineering
engineering.linkedin.com · Company blog · on HN since 2011
LinkedIn's engineering team writes about the company's infrastructure, data systems, and web and mobile development.
What HN saysCommenters single out Jay Kreps's post "The Log" as one of the best engineering blog posts and an influential piece on event streaming and Kafka.
415HN people shared it
163linked it in comments
413stories submitted
3.7Kstory points
10 months agolast shared
2011people per year2026
Most-shared posts
- 1The Log: Real-time data's unifying abstraction92 people · 424 points · first shared 2013 · 17 comments on HN
- 2Blazing fast node.js: performance tips from LinkedIn10 people · 126 points · first shared 2011 · 55 comments on HN
- 3TCP Over IP Anycast – Pipe Dream or Reality?8 people · 88 points · first shared 2015 · 30 comments on HN
- 4A comparison of client-side templating solutions at LinkedIn8 people · 21 points · first shared 2012 · 1 comments on HN
- 5LinkedIn Intro: Doing the Impossible on iOS7 people · 487 points · first shared 2013 · 306 comments on HN
- 6Leaving JSPs in the dust: moving LinkedIn to dust.js client-side templates5 people · 137 points · first shared 2011 · 41 comments on HN
- 7LinkedIn IPhone App's Infinite, Swipe-Able View Implementation5 people · 120 points · first shared 2012 · 16 comments on HN
- 8Linkedin open-sources their distributed object store – Ambry5 people · 35 points · first shared 2016 · 4 comments on HN
- 9Engineering at LinkedIn5 people · 7 points · first shared 2011 · discussion on HN
- 10Shared Dictionary Compression (SDCH) for HTTP at LinkedIn5 people · 4 points · first shared 2015 · discussion on HN
- 11A Brief History of Scaling LinkedIn5 people · 4 points · first shared 2015 · discussion on HN
- 12Blog5 people · first shared 2016
- 13Open-Sourcing Pinot: Scaling the Wall of Real-Time Analytics4 people · 78 points · first shared 2015 · 10 comments on HN
- 14Garbage Collection Optimization for High-Throughput and Low-Latency Java Apps4 people · 58 points · first shared 2014 · 10 comments on HN
- 15Running Apache Kafka at Scale4 people · 50 points · first shared 2015 · 9 comments on HN
What HN people wrote when they linked it
simonw · 2025-12-23
This post by Jay Kreps that introduced Kafka to the world remains one of my favorite pieces of engineering blog content of all time: https://engineering.linkedin.com/distributed-systems/log-what-every-so…
sewen · 2025-01-24
That blog post is a great read as well. Truely, the log abstraction [1] and "Turning the DB inside out" [2] have been hugely influential.
In a way this article here suggests to extend that
(1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo)
(2) have a way to link the events related to a broader operation (handler execution) together
(3) make the log aware of handler execution (better yet, put it in charge), so you can automatically fence outdated executions
[1] https://engineering.linkedin.com/distributed-systems/log-what-every-so…
nithayakumar · 2022-08-17
Facebook definitely does it. Thats where we first learned about this way of working. Some bigger tech companies have actually written about it (see links below) but a list of ones that we've seen with this kind of infra include LinkedIn, Shopify, Dropbox, Stripe, Slack, DataDog, and Palantir - to name a few.
Eng blog links:
LinkedIn: https://engineering.linkedin.com/blog/2021/building-in-the-cloud-with-…
Shopify: https://shopify.engineering/modelling-developer-infrastructure-teams
Slack: https://slack.engineering/development-environments-at-slack/
Doximity: https://technology.doximity.com/articles/github-codespaces-is-great-bu…
itsdrewmiller · 2022-01-01
Most of my favorite blog posts are from individuals rather than companies, but here are a few that are fully or basically from tech companies:
- https://engineering.linkedin.com/distributed-systems/log-what-every-so… (pretty long but very lucid)
- https://web.archive.org/web/20170612053125/https://blogs.dropbox.com/t… (can't find it on their blog anymore)
- https://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-do-deploy… (personal blog but clearly about Stack Overflow)
alexkreidler · 2021-06-18
I've seen many comments here along the lines of "but when should Kafka be used?"
If you want a deep dive into event streaming systems (logs) that also gives some examples of big data systems they interact with, I highly recommend "The Log: What every software engineer should know about real-time data's unifying abstraction," from LinkedIn, the creators of Kafka.
Their engineering blog also has many other interesting articles on their data systems.
[0] https://engineering.linkedin.com/distributed-systems/log-what-every-so…
macksd · 2021-06-01
Yeah I would have loved to see HDFS get really scalable metadata management. I remember hearing about LinkedIn's intentions to really do some significant work there are the last community event I attended, but from their blog post this week it doesn't sound like that's happened since the read-from-standby work [1].
Kerberos (quite popular on big enterprise clusters) is really what makes it hard to get data in / out IMO. I see generic Hadoop connectors in A LOT of third party tools.
[1] https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-…
bem · 2021-05-04
One powerful way to deal with these problems is event sourcing. It's a reasonably elegant way to materialize a single application-specific cache based on many different data sources. Two great resources:
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
https://queue.acm.org/detail.cfm?id=3321612
nchammas · 2021-02-22
That talk by Martin Kleppmann is fantastic.
Another, closely related article is "The Log: What every software engineer should know about real-time data's unifying abstraction" by Jay Kreps.
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
gkop · 2021-02-01
In case you haven't already read it, do read the classic LinkedIn "Log" article: https://engineering.linkedin.com/distributed-systems/log-what-every-so…
I just learned Kafka this year to augment an existing system at work. One thing I have found is, there's a good amount of smarts in the Kafka clients. Look carefully at the Kafka clients available for the runtimes you work with, and evaluate them for robustness, features, and development activity (sadly, for example, kafka-node seems to have lost all momentum).
natn · 2020-10-13
This is one of the best and most accessible pieces I've read about the underlying principles of how such systems work.
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
adamkl · 2020-08-21
That is exactly the premise behind Kafka and LinkedIn's (creator of Kafka) systems architecture. Here's an excellent article on how that architecture looks in practice:
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
iblaine · 2020-07-28
There seems to be a trend where companies are creating tools to monitor data quality...
* Great Expectations: https://github.com/great-expectations/great_expectations
~2 years old, does data profiling
* Data Sentinel: https://engineering.linkedin.com/blog/2020/data-sentinel-automating-da…
It's not open sourced however.
* Data Anomaly Detector: https://www.slideshare.net/iblaine/using-airflow-for-tools-development
Something I built at One Medical that is not open sourced...but this deck has some interesting concepts in it. The approach here is to minimize the effort, lines of code, number of clicks needed, to test your data.
* Amundsen: https://github.com/lyft/amundsen
This is open sourced, and some contributes are building features for data quality analysis. Amundsen seems like it has a bright future.
Terr_ · 2020-01-30
> now they're stumped as to how they're supposed to quickly get data out
I'd argue that (given a large enough business) "reporting" ought to be its own own software unit (code, database, etc.) which is responsible for taking in data-flows from other services and storing them into whatever form happens to be best for the needs of report-runners. Rather than wandering auditor-sysadmins, they're mostly customers of another system.
When it comes to a complicated ecosystem of many idiosyncratic services, this article may be handy: "The Log: What every software engineer should know about real-time data's unifying abstraction"
[0] https://engineering.linkedin.com/distributed-systems/log-what-every-so…
Darkstryder · 2019-10-18
Sometimes no. For instance a database migration script is not equivalent to the diff between two database creation scripts.
While it should be possible to take a database creation script (state A) and a database migration script (edge A->B) and infer the new resulting database creation script (state B), the reverse is not true.
This is the tables vs events duality described in this great article : https://engineering.linkedin.com/distributed-systems/log-what-every-so…
725686 · 2019-05-22
I've never used CQRS or Event Sourcing, but I found it very interesting after watching some of Greg Young's videos.
He also has this: https://leanpub.com/esversioning
This blog by Jay Kreps (Linked-in) is also a good read: https://engineering.linkedin.com/distributed-systems/log-what-every-so…
chrisjc · 2019-05-16
This is worth a read.
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
fancy_pantser · 2019-05-13
The next obvious step after message queues and distributing work would be streams.
Here is an excellent introduction to unified logs and stream processing by the author of Kafka at LinkedIn:
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
geodel · 2018-07-17
So Go set Heap to 2X and CPU 25% as GC overhead. In Java side may be due to generational/copying GC Heap is recommended 3-4X and CPU ~20%. This is good read about Java GC tuning:
https://engineering.linkedin.com/garbage-collection/garbage-collection…
tschellenbach · 2018-05-25
The web part of it is pretty easy to scale. You simply add more web servers. The problem is in the storage/db layer. I imagine the feed was the primary challenge scaling Twitter.
Other components such as the search were probably also quite tricky. One thing i've never figured out is how Facebook handles search on your timeline. That a seriously complex problem.
Linkedin Recently published a bunch of papers about their feed tech:
https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s-f…
And Stream's stackshare is also interesting: https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-mil…
jto1218 · 2018-05-23
love this article about the log abstraction and some of the motivation behind Kafka. It's a great resource to get a feel for what it is and how you can use it:
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
indogooner · 2018-02-18
Depends on my interests at the time but I prefer the blogs where the tech is open source so mostly Twitter[1] and LinkedIn[2] for me. I also follow Airbnb[3] primarily for their blogs on Data and ML (the area in which I work). Have occasionally looked at Uber and Netflix for some of the posts related to Big Data and ML.
[1] https://blog.twitter.com/engineering/en_us.html
[2] https://engineering.linkedin.com/blog
[3] https://medium.com/airbnb-engineering
tomaskafka · 2017-12-31
You will be happy to read this famous post if you haven't yet - log instead of database is a great abstraction!
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
manigandham · 2017-11-02
Nothing in that diagram has anything to do with Kafka, it's just showing that you can connect different systems together using a single message bus. If you are new to that concept then I recommend reading the seminal blog post about using logs as the data backbone: https://engineering.linkedin.com/distributed-systems/log-what-every-so…
After that, you can choose whichever implementation you want, of which Kafka is just one option. And yes it does require some work to setup and run, as most software does, but it's no more or less than any other distributed system at this point.
Gigablah · 2017-10-02
Surely any discussion about the concept of logs and mention of Kafka should have a reference to this excellent article by Jay Kreps on LinkedIn:
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
davidgl · 2017-07-13
Strongly recommend this as a good place to start: https://engineering.linkedin.com/distributed-systems/log-what-every-so…
SEJeff · 2016-10-03
Two must-read documents for those who want to learn more about this method of building reactive applications:
https://engineering.linkedin.com/distributed-systems/log-what-every-so…
http://martinfowler.com/eaaDev/EventSourcing.html
Note that Martin's blog is what inspired the event bus in https://home-assistant.io, an open source home automation project I occasionally contribute to.
aidos · 2015-12-08
That's really silly. LinkedIn produce some really interesting engineering content.
EDIT I'm not sure what Pulse is, but it looks like aggregated content. Anyways, here's an article from the LinkedIn engineering team that's well worth a read https://engineering.linkedin.com/distributed-systems/log-what-every-so…
fintler · 2015-05-19
Another engineering blog that people might not know about:
https://engineering.linkedin.com/
yminsky · 2015-04-11
Here's an example of someone from Linked-In describing a "low latency" setup with a 6GiB heap.
https://engineering.linkedin.com/garbage-collection/garbage-collection…
I've heard similar things from folks at Twitter, IIRC. But I do find the whole thing kind of mysterious, I have to admit. I'd love to learn that I was wrong.
jjHey · 2014-10-25
No. I only had boxes with SATA drives. I don't see how you could get below 1ms latency with spinning disks. You would have to use in-memory queue to get this sort of latency. Secondly, I had Storm consuming off Kafka every millisecond, so in the worst case scenario a message would have to wait one millisecond in the queue anyway. Here's a great article about throughput/latency of Kafka https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-mil…