Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
Why Hacker News recommends it
A perennial HN recommendation for outage and postmortem threads, praised as funny, educational and one of the best postmortems anyone has seen.
What it is
Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.
Recommended by 8 HN commenters across 11 different months, 2018 to 2025.
What Hacker News said
Strong rec OmarAssadi · 2022-01 · on “Famous outages along with deep postmortems?”
Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: [this video]
Strong rec int0x2e · 2023-01 · on “WAN router IP address change blamed for global Microsoft 365 outage”
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) [this video]
Strong rec KronisLV · 2023-03 · on “Ask HN: What is the best postmortem you've seen?”
The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: [this video] Here's the slides: [LINK] It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the various risks, architectures, how things fail and about software development in general.
Recommends merlincorey · 2019-07 · on “Root cause analysis: significantly elevated error rates on 2019‑07‑10”
> In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. Bryan Cantrill has a great talk[0] about dealing with fires where he says something to the effect of: > Now you will find out if you are more operations or development - developers will want to leave things be to gather data and understand, while operations will want to rollback and fix things as quickly as possible [0] Debugging Under Fire: Keep your Head when Systems have Lost their Mind - Bryan Cantrill: [this video]
Recommends pdkl95 · 2020-09 · on “They're deleting my channel, but they don't know why? [video]”
> I run a YouTube channel that is over 10 years old , has over 770,000 subscribers , almost 600 public videos , 120,000,000 views When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star). For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk[1] about the time an operator accidenta
Recommends KronisLV · 2021-02 · on “Google admits Kubernetes container tech is too complex”
> Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind [this video]
Recommends e12e · 2021-08 · on “Harbormaster: Anti-Kubernetes for your personal server”
Not (I think) the exacttalk/blog post gp was thinking of - but worth watching IMNHO: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017" [this video] Ed: oh, here we go I think? > Running Aground: Debugging Docker in Production Bryan Cantrill19,102 views16 Jan 2018 Talk originally given at DockerCon '15, which (despite being a popular presentation and still broadly current) Docker Inc. has elected to delist. [LINK]
Recommends KronisLV · 2021-10 · on “Is it time to rewrite the Operating System in Rust? (2018) [video]”
I'll give it a watch and will summarize, because while i'm not always on board with the Rust hype (and therefore want to know more, to help eliminate my biases, one way or the other). That said, i rather enjoyed Bryan Cantrill's talk from 2017, "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017": [this video] So i wouldn't necessarily turn away from any of his videos just because of the sometimes humorous or not awfully serious tone.