Werner Vogels
allthingsdistributed.com · Personal blog · on HN since 2007
Werner Vogels, Amazon's CTO, writes about distributed systems, AWS services, the Dynamo paper and engineering leadership.
What HN saysHN links it for the Dynamo paper, which commenters call a great read, and for the AWS Lambda PR/FAQ document.
297HN people shared it
141linked it in comments
286stories submitted
5.0Kstory points
3 months agolast shared
2007people per year2026
Most-shared posts
- 1Dynamo: Amazon’s Highly Available Key-Value Store (2007) [pdf]29 people · 4 points · first shared 2011 · discussion on HN
- 2Working Backwards16 people · 19 points · first shared 2009 · 1 comments on HN
- 3Amazon's Dynamo14 people · 8 points · first shared 2007 · discussion on HN
- 4Design Considerations for High-Throughput Cloud-Native RDBMS [pdf]12 people · 92 points · first shared 2017 · 19 comments on HN
- 5Modern Applications at AWS10 people · 89 points · first shared 2019 · 42 comments on HN
- 6Building and operating a pretty big storage system called S3 (2023)10 people · 42 points · first shared 2023 · 3 comments on HN
- 7The Different CTO Roles (2007)10 people · 5 points · first shared 2011 · discussion on HN
- 8The Story of Apollo – Amazon’s Deployment Engine9 people · 9 points · first shared 2014 · discussion on HN
- 9Homepage8 people · first shared 2010
- 10Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS7 people · 298 points · first shared 2012 · 126 comments on HN
- 11Eventually Consistent - Revisited7 people · 21 points · first shared 2008 · discussion on HN
- 12AWS Lambda PR/FAQ After 10 Years5 people · 46 points · first shared 2024 · 12 comments on HN
- 13Under the Hood of Amazon EC2 Container Service4 people · 226 points · first shared 2015 · 34 comments on HN
- 14Lessons from 10 Years of Amazon Web Services4 people · 170 points · first shared 2016 · 24 comments on HN
- 15Diving Deep on S3 Consistency4 people · 132 points · first shared 2021 · 49 comments on HN
What HN people wrote when they linked it
throwaw12 · 2026-07-14
part of it
https://muratbuffalo.blogspot.com
https://www.allthingsdistributed.com/
https://brooker.co.za/blog/
https://surfingcomplexity.blog/
jsty · 2025-08-04
The AWS Lambda PR/FAQ was released late last year - worth a read
https://www.allthingsdistributed.com/2024/11/aws-lambda-turns-10-a-rar…
packetlost · 2025-04-19
If you're looking for an application of these, the DynamoDB paper is a really great read: https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
HDThoreaun · 2025-03-16
The problem with postgres scaling is that you have to have a single master which means horizontal scaling really only gives you more reads and a failover. Eventually you wont be able to find a server big enough to handle all the writes, and if you get enough reads with even a small number of writes single master setups fall over. Distributed computing gets complicated very quickly but the gist here is basically that you need to be able to have multiple instances that can accept writes. Lots of literature on this but good starting points imo would be the paxos paper https://lamport.azurewebsites.net/pubs/time-clocks.pdf and dynamo db paper https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
LAC-Tech · 2024-08-28
I think a lot of things. CRDTs predate the term "CRDT", they seem to be a mathematical construct/pattern found in the wild, that smart people codified and formalised.
IME CRDTs just lay out the rules you need to follow for syncing eventually consistent nodes properly, ie without weird surprises or data loss.
As for concrete examples:
- in the famous Amazon Dynamo Whitepaper, they perfectly describe a CRDT in section 4.4. (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)… though I don't think this made it to modern Dynamo DB
- I believe CouchDB follows all the CRDT laws
- TomTom uses them
I think the TL:DR is if you're going to sync two data stores that can be written to without coordination, learn the CRDT laws and make sure you follow them. They're fairly simple mathematical properties at the end of the day.
baldeagle · 2023-10-26
AWS just KILLED EC2!!! Well, EC2-classic. And they did so with two years of notice and support to transition all customers. Really a class act of migration. I'm not sure Google won't just send out an email on a Friday before a long weekend and then kill the servers the following Monday.
https://www.allthingsdistributed.com/2023/09/farewell-ec2-classic.html
ambry · 2023-04-09
That's right: You upload an object, and the HTTP 200 response includes a header that is the object ID. Your imagination is almost certainly correct: creating a mapping from user-defined key to storage location is really best done by a metadata store so that data doesn't need to be reshuffled with topology changes.
A classic distributed key-value store (such as Dynamo[1]) spends a lot of its design on how to consistently map user-keys to storage locations. Most of these solutions are focused on performance (so avoiding a middle-layer lookup) and accept the tradeoff that data will need to be re-balanced when doing things like expanding the cluster. This isn't an acceptable tradeoff for object stores that have much higher data density (100+ TBs per NIC)
[1]: https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
dshah · 2022-10-07
Thanks for the kind words. I don't consider myself a "brilliant" engineer. More of an experienced engineer. I've been building software products for 30+ years.
I think the role of the CTO varies based on the scale of the company.
This article on the types of CTOs is useful framing:
https://www.allthingsdistributed.com/2007/07/the_different_cto_roles.h…
ignoramous · 2022-06-20
> Anthony had long come to the same conclusion I had, that highly visible corporate incubators do a good job of shaping culture and getting great press, but most often their biggest products were demos that never get deployed to the field.
Cisco is pretty good at this: https://news.ycombinator.com/item?id=8348900 (2014)
> As we were finishing my coffee Anthony said, "I'm going to let a few of the execs know I'm not out for turf because I only intend to be here for a few years."
One ought to know, 'tis but a game of poker, mister.
More on the role of a CTO: https://www.allthingsdistributed.com/2007/07/the_different_cto_roles.h… (2007) and https://news.ycombinator.com/item?id=20642423 (2019)
rubiquity · 2022-02-16
Disclaimer: ex-AWS as well, worked very closely with MemoryDB, but not paid to shill!
You're entitled to your opinion but your line of reasoning for how MemoryDB for Redis came to exist or the reasoning about why it isn't in upstream Redis is not factual. MemoryDB's architecture uses Amazon's home grown log replication services as pointed out by Werner Vogel in his blog post about MemoryDB[0]. This architecture is fundamentally incompatible with upstream Redis. The real reason why MemoryDB for Redis exists is far less juicy: MemoryDB came about by meeting customers where they were. Customer's love Redis, especially for its non-caching features, but replication is a headache with all the existing solutions today.
Also as far as I know one of the lead committers to Redis is from AWS.
0 - https://www.allthingsdistributed.com/2021/11/amazon-memorydb-for-redis…
anonymousDan · 2022-01-20
Can anyone recommend me a good paper (or other resource) describing the designs of DynamoDB and S3? Ideally something in the spirt of the original Dynamo paper (i.e. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf )
vishnu_ks · 2022-01-01
A couple of my favorite technical blogs
- https://openai.com/blog/
- https://ai.googleblog.com
- http://antirez.com/
- https://research.facebook.com/blog/
- https://martin.kleppmann.com/archive.html
- https://devblogs.microsoft.com/oldnewthing
- https://www.allthingsdistributed.com/
- https://jvns.ca/
- https://www.johndcook.com/blog
- http://feross.org/
- https://ridiculousfish.com/blog/
- https://blog.andrewcantino.com/
- https://nickcraver.com/blog/
- https://blog.codinghorror.com
I have been also building an engineering blog aggregator called https://diff.blog to make it super easy to disover and follow blogs. Would love if you folks can give it a try :)
Communitivity · 2021-02-01
Not papers or books, but I'd recommend you look at the blog articles and such that Basho published about their design of Riak. In particular, they wrote a great article explaining vector clocks as opposed to other synchronizing mechanisms, and they wrote several good articles on CRDTs.
https://riak.com/category/technical/ - Riak blog
https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf - Dynamo paper that Riak was in part based on
jedberg · 2020-06-07
> The people actually doing the work don’t have time to talk about it and the people spending all their time talking about it aren’t doing the work.
I think that's a really sad and cynical view. I assume you don't spend 16 hours a day writing code. I assume you have hobbies and other interests. For some people, educating others about their favorite topics is their hobby. Writing about their projects is their hobby.
Here are entire blogs by people who are "doing":
http://www.brendangregg.com
https://perspectives.mvdirona.com
https://netflixtechblog.com
https://www.allthingsdistributed.com
https://airbnb.io
https://engineering.fb.com
Those are just a few I can think of in one minute. If I went through my RSS feeds and bookmarks I could find more.
nivertech · 2020-03-23
The EC2 project was started in Cape Town.
Also Amazon's CTO has some links to South Africa (or maybe he is from Amsterdam).
https://www.allthingsdistributed.com/
carlsborg · 2019-09-28
Werner Vogels has a recent blog post [1] entitled "Modern Applications at AWS" where he says "To succeed in using application development to increase agility and innovation speed, organizations must adopt five elements, in any order: microservices; purpose-built databases; automated software release pipelines; a serverless operational model; and automated, continuous security."
I see the devops role at large cos evolving into dev-sec-ops for the "automated, continuous security" tooling and processes. (Besides the usual tooling for delivering and supporting a reliable network service, cloud engineering, system-level issue resolution in dev and test, load and performance and containerized test automation in the pipelines, failover fire-drills, etc.)
[1] https://www.allthingsdistributed.com/2019/08/modern-applications-at-aw…
colemorrison · 2019-09-23
This makes me think of the Amazon CTO's analysis from Modern Applications on AWS where they push Serverless as the future:
"...we anticipate that there will soon be a whole generation of developers who have never touched a server and only write business logic."
Honestly, which sounds more like a nightmare than a dream. Knowing what your software runs on and how it works is critical if you want to create REALLY GREAT software.
(reference article: https://www.allthingsdistributed.com/2019/08/modern-applications-at-aw…
PakG1 · 2019-07-31
Looking at this now, it's amazing to compare with the public criticism that Werner Vogels gave to the face of the corporate blogging movement back in 2006.
https://www.allthingsdistributed.com/2006/03/naked_answers.html
Back then when he claimed that Amazon did a very good job of being customer focused, I was thinking: maybe, but whatever, I'm not sure what's special about their customer experience compared to others. Today, I'm thinking: wow, now that's a real example of audacious vision both executed and realized.
davidjnelson · 2019-04-05
The big innovation of aurora is they decoupled the compute from the storage [1]. Both compute and storage need to auto scale, but compute is the really important one that is hardest. Aurora serverless scales the compute automatically by keeping warm pools of db capacity around [2]. This is great for spiky traffic without degraded performance.
1. https://www.allthingsdistributed.com/files/p1041-verbitski.pdf
2. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-se…
wanghq · 2018-06-30
This is similar to Amazon's "working backwards" based product development. Before writing the manual, they do another important writing - press release. It's amazing to see they talked about this idea a decade ago and still use the same approach to drive the development today.
https://www.allthingsdistributed.com/2006/11/working_backwards.html
not_kurt_godel · 2018-06-23
DynamoDB is very much serverless :) I'd highly recommend to anyone remotely interested in serverless and/or databases to read the Dynamo whitepaper to get an idea of how it works: https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
cavisne · 2018-05-11
The marketing message could be clearer here but if you read the aurora paper
https://www.allthingsdistributed.com/files/p1041-verbitski.pdf
And the docs
There are actually two backup/restore methods.
1) Point In Time - streamed to S3. Guaranteed restore to a new cluster with less than 5 minutes data loss
2) Aurora Backtrack - On cluster buffer, up to 24 hours retention (no guarantee). "Instant" restore to the same cluster
So you should feel safe using the service (up to the 5 minute worst case scenario).
Obviously no one would ever want to use either backup method. However previously disaster recovery would involve spinning up a new cluster, deploying application changes with the new endpoint (or in a best case repointing a dns entry), waiting for the cache to warm up again. Now it becomes a 1 click fix through the UI with no application changes.
zackelan · 2017-12-06
If you want to read about the theoretical underpinnings, Cassandra is derived from the original Dynamo paper[0] from Amazon, and Scylla is a drop-in replacement for Cassandra written in C++ instead of Java. Cockroach follows more closely the Google Spanner[1] approach.
For a more practical summary, compare the architecture overviews of Cassandra[2] and Cockroach[3].
0: http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
1: https://static.googleusercontent.com/media/research.google.com/en//arc…
2: https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/arch…
3: https://www.cockroachlabs.com/docs/stable/architecture/overview.html
zaptheimpaler · 2017-11-25
DynamoDB paper [1] - one of the big AP, KV stores that kicked off the NoSQL wave.
Paper about Akamai[2] - "Nuggets in content delivery"
Also, acolyers blog "the morning paper". He reviews and explains one paper every day. its great because you get a sense of whats current.
[1]http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
[2]https://www.akamai.com/cn/zh/multimedia/documents/technical-publicatio…
hodgesrm · 2017-11-17
Amazon Aurora does something similar in the sense that it decouples InnoDB functions including buffer cache and storage access from the overlying MySQL engine. The decoupled InnoDB lives on separate VM(s) and includes optimized data replication to support DBMS high availability. For example, with Aurora you can failover a large DBMS in seconds to a new site and have it come up with a heated buffer cache since that replicates too.
This approach makes sense for Aurora because it leverages cloud storage to provide data high availability across availability zones. Just putting InnoDB in a file system seems less interesting unless you add features that are strikingly different from what it does now.
See http://www.allthingsdistributed.com/files/p1041-verbitski.pdf for a full description.
phamilton · 2017-08-24
The Dynamo Paper. http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
One of the best practical "How can this improve our business?" technical papers, and an excellent introduction to reading papers.
neduma · 2017-04-21
http://www.allthingsdistributed.com/ - By Amazon CTO Werner Vogels
Thinking further on this.. Apple is totally not in the picture or i do not know.
benrhughes · 2014-11-07
Amazon's CTO Werner Vogels blogs pretty regularly at http://www.allthingsdistributed.com/
dantiberian · 2014-10-20
I've learned heaps from reading the papers that these companies publish. Google, Facebook, and Amazon all publish well written, readable, papers. If you're looking for somewhere to start, then the Dynamo paper is a classic: http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
asrowe · 2013-05-27
I liked the article a lot. For more detail its worth looking at:
Join Processing in Relational Databases, Priti Mishra and Margaret H. Eich, ACM Computing Surveys (CSUR) Surveys, Volume 24 Issue 1, March 1992, Pages 63 - 113
via http://www.allthingsdistributed.com/2013/04/join-processing-relational…