James Hamilton

nice read. It's also worth mentioning that James Hamilton's personal website [1] has been hosted by Godaddy ever since, including the domain's name servers. [1] http://mvdirona.com

If you are interested in datacenter design, distributed systems or just large services, I would recommend following James Hamilton's blog: http://perspectives.mvdirona.com/. James covers a very wide range of topics, and is great about linking to source material and related articles.

James Hamilton's blog is excellent for anyone interested in data centres: http://perspectives.mvdirona.com

Amazon is expanding their EC2 feature set so rapidly... The pace is mind blowing to me. Last year, Randy Bias estimated EC2 was pulling $220M revenue/yr: http://cloudscaling.com/blog/cloud-computing/amazons-ec2-generating-22… And he estimated an overly conservative 10-20% annual growth. But given the EC2 buzz this year, and personal anecdotes from my friends and colleagues using it, my gut feeling tells me the 2010 revenues will have increased 50-100% over 2009 revenues. Is EC2 profitable to Amazon? Likely very profitable if you want my opinion. It is well accepted in the industry that the dominant cost in large scale datacenters is power and cooling --not hardware, not human resources-- and I keep running numbers in my head and the hourly prices of all instance types are well above power & cooling. Just as an example, we know that this new GPU instance has two 95W Xeon X5570 and two 247W Tesla M2050; assuming (1) a max TDP of 50W for the motherboard and rest of the server, (2) instances are run under full load 100% of the time and always reach these max TDP numbers (unlikely, but follow me for the sake of the argument), (3) Amazon uses servers with 80PLUS power supplies (80% efficient or more), (4) a rather good datacenter with a PUE of 1.3 (power usage effectiveness, which includes overhead from power distribution and cooling; numbers in the range of 1.2-1.4 are often quoted by James Hamilton from the AWS team: http://perspectives.mvdirona.com/), and (5)…

The architect of AWS and author of Dynamodb has very good thoughts on formal methods. They’re worth a read: https://perspectives.mvdirona.com/2014/07/challenges-in-designing-at-s…

AWS' James Hamilton lives pretty much on a boat around the world most of the time [0]. I was at AWS 2008-2014 and had a few chats with him. Super nice, very smart, humble. For a little while he used to be a car mechanic for Italian cars in his younger years. [0]: https://mvdirona.com/

VP and Distinguished Engineer at Amazon Web Services and his personal blog about boats https://mvdirona.com/ (also some stuff about tech on https://perspectives.mvdirona.com/)

> The people actually doing the work don’t have time to talk about it and the people spending all their time talking about it aren’t doing the work. I think that's a really sad and cynical view. I assume you don't spend 16 hours a day writing code. I assume you have hobbies and other interests. For some people, educating others about their favorite topics is their hobby. Writing about their projects is their hobby. Here are entire blogs by people who are "doing": http://www.brendangregg.com https://perspectives.mvdirona.com https://netflixtechblog.com https://www.allthingsdistributed.com https://airbnb.io https://engineering.fb.com Those are just a few I can think of in one minute. If I went through my RSS feeds and bookmarks I could find more.

Some great reading on the topic from James Hamilton's blog. https://perspectives.mvdirona.com/2013/02/the-power-failure-seen-aroun… https://perspectives.mvdirona.com/2017/04/at-scale-rare-events-arent-r… https://perspectives.mvdirona.com/2017/12/when-you-cant-afford-not-to-…

James Hamilton of AWS https://perspectives.mvdirona.com/ Jordan Mechner's journals about making Prince of Persia http://www.jordanmechner.com/backstage/journals/ WARNING: You may find yourself staying up all night to page through one of them in entirety. I did.

I am just going to drop a few quick comments. It's 6:39AM in NY, and I just woke up. Maybe not relevant at all, but... I love reading about biography like these. I own a couple books, for example, "Where Wizards Stay Up Late: The Origins Of The Internet" did keep me awake for a few days on an early train. This one, absolutely easy to digest, thank you for sharing. 1. So do the four code bases share code they reuse (think of a common repository they depend on)? 2. Are the query planners all different because they needed to be platform specific? 3. Is DB2 still on four code bases? Would be awesome if we have one central catalog of stories like this one. BTW, I didn't know much about the author, so I did some quick googling: http://mvdirona.com/jrh/work/ (I hope HN won't flood it to 404! Mr. James, is this on AWS!? It looks like it might be.)

Coincidentally, a couple months ago one of my professors told me about James Hamilton, apparently, they met when he was studying at Waterloo. I started reading his blog. This guy is brilliant. And his comments are often gems. The articles are good too but think of them as conversation openers. The real deal is in the comments imho. I recommend it. A somewhat funny article is about Zynga, the "prodigal kid" who left AWS for on-premises, only to come back later: http://perspectives.mvdirona.com/2015/05/the-return-to-the-cloud/ In the comments, he debunks (or take a shot at it) the idea that cloud providers like AWS aren't a good fit for organizations which have massive, but stable workloads. This guy is so cool. I only wish he had more time to write.

James Hamilton's blog is not to be missed: http://perspectives.mvdirona.com/

Amazon's James Hamilton estimates the north Carolina 171 acre solar array only generates %4 of the energy required for the datacenter http://perspectives.mvdirona.com/2012/03/17/ILoveSolarPowerBut.aspx

Best hair ever http://mvdirona.com/jrh/work/JamesHamilton.jpg

Unknown is relative, but here are some of the rss feeds I'm subscribed too: http://nosql.mypopescu.com/ http://lambda-the-ultimate.org/ http://eli.thegreenplace.net/ http://www.stefankrause.net/wp/ http://mechanical-sympathy.blogspot.com/ http://www.cowtowncoder.com/blog/blog.html http://highlyscalable.wordpress.com/ http://hunch.net/ http://perspectives.mvdirona.com/ http://www.scottaaronson.com/blog/ http://www.softwarebyrob.com/ http://blog.mikemccandless.com/

>If you are speaking about server applications: ARM doesnt really have a shot here. The guy who is in charge of datacenter power efficiency at Amazon is super-enthusiastic about ARM servers: http://perspectives.mvdirona.com/2010/11/20/VeryLowCostLowPowerServers… James writes a lot about power efficiency in server computing, and his blog is a must-read for anyone who wants to have a qualified, researched opinion on the subject of power management on the server.

This is a lovely little article that speaks of the unscalability of distributed transactions. However, to my great surprise, Google is apparently putting distributed transactions in the Spanner project, the next generation of BigTable. See Jeff Dean's talk here: http://perspectives.mvdirona.com/2009/10/17/JeffDeanDesignLessonsAndAd…

James Hamilton, VP at Amazon and author of the popular blog "Perspectives" has some interesting insights into and great things to say about the work SeaMicro and other new startups are doing to revolutionize the server industry. http://perspectives.mvdirona.com/2010/06/14/SeaMicroReleasesInnovative…

To anybody questioning what the next level of DNS could be, think of this way: infrastructure services like EC2 or Slicehost enable us to move/copy/build out images quite easily. But if this flexibility is not matched by API-driven networking surrounding our servers, there will always be a ceiling of flexibility that our servers cannot realize. I don't quite know what dnsworks has in mind, but anything that aims to make DNS as flexible as other stuff is becoming, will be quite useful. Related reading (not by me): http://perspectives.mvdirona.com/2009/12/19/NetworkingTheLastBastionOf… http://www.bretpiatt.com/blog/2009/12/20/public-clouds-and-their-featu… http://www.rationalsurvivability.com/blog/?p=1603

James has put out a lot of great material. He's given a talk series and wrote a conference paper called "On Designing and Deploying Internet-Scale Services" which is a trove of valuable design insights: https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/h… (somewhat condensed) Slides from a talk: http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf He writes some of the more practical and useful advice I've seen about how to run a successful business based on high-scale distributed systems. One of his mnemonics that's stuck with me is the "Four Rs" of recovery-oriented computing: Restart, Reboot, Reimage, Replace (slide 6 in PDF). A human shouldn't be engaged to troubleshoot a problem until the platform has first tried in successing to restart the software, reboot the OS, re-image the machine, and finally replace the hardware entirely. After these auto-recovery steps have failed, only then is it time to engage a human. He describes the connection between these techniques and significantly lower operations costs.

Recovery-Oriented Computing has hit HN before. Instead of surfing the overview page, check out the toplevel with links to a LOT of fantastic and accessible papers and presentations: http://roc.cs.berkeley.edu/ Then, move on to James Hamilton[1]'s "On Designing and Deploying Internet-Scale Services" which is a 10 minute read that's HOWTO on putting Recovery-Oriented Computing into practice: http://static.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilt… That paper changed the entire way I design and evolve systems. And I keep seeing the lessons from it come up in less-distilled forms in other places. For example, Zach Holman of Github's recent and great "How To Build A Github" http://zachholman.com/talk/how-to-build-a-github [1] Vice President and Distinguished Engineer on the Amazon Web Services team where he is focused on infrastructure efficiency, reliability, and scaling. http://www.mvdirona.com/jrh/work/

I have helped scale (non web) low latency systems in some pretty stressful situations. In addition to the sage advice about monitoring / metrics, mentoring and not scaling until necessary I think the following are useful: * Design Services not Software. In particular read "On Designing and Deploying Internet-Scale Services" (http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf) and at least the first chapter of "The Art of Unix Programming" (http://catb.org/~esr/writings/taoup/html/) * Get to grips with debugging and profiling so you can figure out what's really happening. Tools like sar, sysstat, [d|k|s]trace, tcpdump, gdb etc and the equivalents for your datastore & application frameworks are invaluable and unfortunately for whatever reason you inevitably won't have all the metrics and monitors you need. * Do try to understand every layer of your service. I have helped debug scale out related issues from Layer 2 to Layer 8. I have also had to debug many Layer 1 issues while bringing up a new Site or similar. I may not be a DBA, Network Engineer or Software Engineer but in the past I have had to wear those hats while scaling. * Despite comments elsewhere about learning through Mentoring and baptism-by-fire there is a lot of real engineering & science theory you can lean on. Looking back on courses I took in school while I didn't do any courses on Scalable Web Programming over the years I have used content from courses on Computer Architecture, Math including Queuing…

Eventual consistency often gives up the ability to develop applications against the system, because more reasoning about the state of the system has to be embedded in the application logic. For a better description of this problem, see this post from an AWS employee: http://perspectives.mvdirona.com/2010/02/24/ILoveEventualConsistencyBu… Note that his "right answer" for performance (not availability, so it's a bit of an aside from your comment) is to dynamically repartition, which is what Clustrix does. Note that we can actually do this type of repartitioning on just the "hot" data, using MVCC to avoid any downtime (unlike Mongo, which blocks all writes). By the way, the Google Megastore project I believe you're referring to is not actually eventually consistent in the same way that, say, Cassandra is. Megastore is fully ACID within an entity group, meaning that lowering eventually consistency down to the per-node level was not the solution that Google went with. BigTable is also not eventually consistent. *Clustrix employee

The linked slides are a lot more informative than the summary on that page: http://mvdirona.com/jrh/TalksAndPapers/KushagraVaidMicrosoftDataCenter… It's interesting that in contrast to the likes of Google and HP who have been stating that the cost of power will soon be greater than the cost of servers — e.g. http://bits.blogs.nytimes.com/2008/11/03/kill-an-energy-hog-save-your-… — Microsoft seem to be having a very different experience with servers making up 61% of the TCO and power consumption and distribution/cooling only taking up 16% and 14% respectively. It's also interesting that they haven't (seemingly) adopted the Google strategy of shifting workloads from data centers based on the external weather conditions, e.g. http://www.datacenterknowledge.com/archives/2009/07/15/googles-chiller… — This sounded like an ingenious idea when I first heard about it and am surprised that everyone didn't copy it straight away.

AWS publicly stated I think in 2021 that the larger availability zones in US East 1 consisted of 17-18 datacenters each. It’s likely grown a lot since, and they recently announced AZ7 will be online in Maryland soon, so they must be running out of ability to grow the ones in NoVA. I can’t find a link now but it was one of the re:Invent talks like Peter DeSantis briefly explaining AZs before he dug into how Amazon optimizes their concrete mixtures to be more environmentally friendly or something… All things point to that being the biggest region any hyperscaler has in the world, and several gigawatts of power consumption. James Hamilton also gave a talk in 2021 about AWS having crossed 20 million Nitro cards deployed and 12GW power consumed — https://mvdirona.com/jrh/talksandpapers/JamesHamilton20221010.pdf

Until just a few years ago James Hamilton (who the article is about) was a fairly remote worker—as he was circumnavigating on his boat, which they decided to stop: * https://mvdirona.com/2021/09/changing-plans-again/ * https://mvdirona.com/2015/08/communications-at-sea/ * https://mvdirona.com/2009/06/remote-data-communication-costs/ * https://www.youtube.com/user/mvdirona

You may be thinking about Yahoo’s approach from 2010? > The Yahoo! approach is to avoid the capital cost and power consumption of chillers entirely by allowing the cold aisle temperatures to rise to 85F to 90F when they are unable to hold the temperature lower. They calculate they will only do this 34 hours a year which is less than 0.4% of the year. https://perspectives.mvdirona.com/2011/03/yahoo-compute-coop-design/

According to James Hamilton who is a distinguished engineer at AWS no: https://perspectives.mvdirona.com/2010/09/overall-data-center-costs/ and https://perspectives.mvdirona.com/2008/12/annual-fully-burdened-cost-o… And you can watch him say it too: https://youtu.be/kHW-ayt_Urk?si=DKyw0-Pk-dhU5zFG&t=323