AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems
Why Hacker News recommends it
HN points to it as insight into how AWS thinks about distributed systems and the difficulty of consensus.
What it is
An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.
Recommended by 1 HN commenters across 1 different months, 2021 to 2025.
What Hacker News said
Recommends rubiquity · 2022-05 · on “Databases = Frameworks for Distributed Systems”
> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures." I'm guessing the talk it came from is Under The Covers: Core Distributed Systems Primitives that Power Our Platform[0]. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial P
Cites ignoramous · 2021-02 · on “I followed my dreams to get demoted to software developer”
Reminds me of a "self-demotion" that worked out well: That of a certain Allan Vermeulen [0] who, by some accounts, was the chief architect of Amazon S3 [1], and by all accounts, an engineer extraordinaire. Vermeulen managed the Platform Services group when he joined [2], and later moved to a CTO role with focus on the then very nascent Web Services initiative [3], but eventually "demoted" himself to an "AWS Engineer" role to actually build key technological pieces that would underpin a lot of distributed systems at AWS [4]. Not bad at all. [0] [LINK] [1] [LINK] [2] [LINK] [3] [LINK] [4] [this video]
Cites ignoramous · 2021-02 · on “Consensus Is Harder Than It Looks (2020)”
You are not alone: [LINK] and [this video] Like TFA points out, distributed consensus is punishingly hard. AWS relied on TLA+ to prove consensus in DynamoDB and other systems [LINK] Interestingly, Kinesis [0] and SQS (?) avoid consensus for those same reasons. [0] [LINK]
Cites ignoramous · 2021-03 · on “A Second Conversation with Werner Vogels”
> A few things around S3 were unique. We launched with a set of ten distributed systems tenets in the press release. The ten tenets were separate from S3, stating that this is how you would want to build distributed systems at scale. We just demonstrated that S3 is a really good example of applying those skills. In my own naive way I believe that AWS, today, probably embodies James Hamilton's vision [0] far more than most of those original 10 tenets. - > One of the keys to the success of S3 was that, at launch, it was as simple as it possibly could be, offering little more than GetObject and PutObject. At the time that was quite controversial, as the offering seemed almost too bare bones...
Cites ignoramous · 2021-07 · on “Paxos vs. Raft: Have we reached consensus on distributed consensus?”
> Basic Paxos, is well, too basic and people mostly run modifications of this to get higher throughput and better latencies. After those modifications, it does not look very different from Raft. Alan Vermeulen, one of the founding AWS engineers, calls inventing newer solutions to distributed consensus an exercise in re-discovering Paxos. [this video]
Cites dtjohnnymonkey · 2022-08 · on “Why use Paxos instead of Raft?”
I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: [this video]
Cites throwdbaaway · 2025-10 · on “Summary of the Amazon DynamoDB Service Disruption in US-East-1 Region”
> But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. I guess they don't have a recovery procedure for the "congestive collapse" edge case. I have seen something similar, so I wouldn't be frowning at this. A couple of red flags though: 1. Apparent lack of load-shedding support by this DWFM, such that a server reboot had to be performed. Need to learn from [LINK] 2. Having DynamoDB as a dependency of this DWFM service, instead of something more primitive like Chubby. Need to learn more about distributed systems primitives from [this video]