AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)

What Hacker News said

Recommends ignoramous · 2019-03 · on “An introduction to distributed systems (2017)”

This is a great outline. Martin Kleppmann's Designing Data Intensive Applications [0] is very much in the same domain as what's in these notes. Speaking about scale and distributed systems: AWS specific resources I find interesting from their EdgeEngineering/NetEng teams (I haven't seen many other service teams in AWS openly share as much about design as them): - [LINK] series of articles on Route53's 100% data-plane availability architecture [1]. - [LINK] colmmacc [2] (seems to be the eng behind AWS HyperPlane [3][4]?) on 10 design patterns for building resilient systems. - [this video] Peter Vosshall (co-creator of Dynamo [5]) presenting "cell-based" design in-use at AWS. ---- [0] [LINK] [1] [L

Recommends ignoramous · 2019-03 · on “Post-mortem of March 12 Google outage”

From what you describe the 'AWS cells' are very different than Borg's. My view is that at a high level aws cells are isolated, fuzzy-partitioned, migratable, heterogeneous instances of a complete service stack that are managed such that they always fail independently of each other; contact points among these cell instances, if required, are established via highly available intermediate services (like ReplicatedStateMachinesOrStorage); the clients are routed to assigned cells by a highly available routing layer (like NetworkTrafficShapers/DNS), which can also help facilitate migration of clients from one/more cell to another. I encourage you to view the re:invent presentation on it [0] if res

Recommends talawahtech · 2020-11 · on “Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region”

Multi-AZ doesn't protect against a software/OS issue like this, Multi-AZ would be relevant if it was an infrastructure failure (e.g. underlying EC2 instances or networking). The relevant resiliency pattern in this case would be what they refer to as cell-based architecture, where within an AZ services are broken down into smaller independent cells to minimize the blast radius. They specifically mention in the write-up that this was a gap they plan to address, the "backend" portion of Kinesis was already cellularized but that step had not yet been completed on the "frontend". Celluarization in combination with workload partitioning would have helped, e.g. don't run Cloudwatch, Cognito and Cus

Cites ignoramous · 2019-03 · on “Post-mortem of March 12 Google outage”

> [LINK] > Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called "canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic. > Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back. Any Googlers around to explain why Ca

Cites ignoramous · 2020-02 · on “Ask HN: What were the things you did that made the biggest impact at your work?”

In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2]. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3]. [0] [LINK] [1] [LINK] [2] [this video] [3] [LINK]

Cites ignoramous · 2020-11 · on “Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region”

root-cause tldr: ...[adding] new capacity [to the front-end fleet] had caused all of the servers in the [front-end] fleet to exceed the maximum number of threads allowed by an operating system configuration [number of threads spawned is directly proportional to number of servers in the fleet]. As this limit was being exceeded, cache construction was failing to complete and front-end servers were ending up with useless shard-maps that left them unable to route requests to back-end clusters. fixes: ...moving to larger CPU and memory servers [and thus fewer front-end servers]. Having fewer servers means that each server maintains fewer threads. ...making a number of changes to radically improve

Cites 708145_ · 2022-07 · on “Cell-Based Architecture”

The concept have been mentioned by Amazon for years. For reference see: "AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures" [this video]

Cites faizshah · 2023-03 · on “Fly.io outage, recently deployed apps down, deployments disabled”

AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: [this video] AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.