Zebras All the Way Down - Bryan Cantrill, Uptime 2017

What Hacker News said

Recommends gt2 · 2018-10 · on “Ask HN: What are some of the best technical talks you've heard?”

Zebras All the Way Down - Bryan Cantrill, Uptime 2017 [this video] Lots of great tangents that add to the subjects. I love talks like this.

Recommends ar0b · 2018-10 · on “iPhones are allergic to helium”

One of my favorite conference talks is semi related to problems like this. [this video] Brian Cantrill - Zebras all the way down

Recommends kbenson · 2019-07 · on “Replacing a silently failing disk in a ZFS pool”

On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: [this video]

Recommends kbenson · 2020-01 · on “Five Years of Btrfs”

There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to be quite entertaining as well as informative and historical (if from his POV). 1: [this video]

Recommends ignoramous · 2020-02 · on “Ask HN: What were the things you did that made the biggest impact at your work?”

In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2]. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3]. [0] [LINK] [1] [LINK] [2] [LINK] [3] [this video]

Recommends Bayart · 2021-01 · on “Installing Debian on Modern Hardware”

>Why does anyone really care about non-free firmware blobs? Aside from the philosophical considerations, stuff breaking in a black-box firmware that you're lead into believing "just works" is particularly aggravating. I'll link Bryan Cantrill's talk on the subject, he's more entertaining than I am. [1] [1] [this video]

Recommends thomasjudge · 2025-04 · on “Synology Lost the Plot with Hard Drive Locking Move”

For an entertaining/terrifying perspective on firmware, obligatory Bryan Cantrill talk "Zebras All the Way Down" [this video]

Cites floatboth · 2018-01 · on “Ask HN: What’s your favorite talk from 2017?”

On The Turing Completeness of PowerPoint [LINK] Compiling C to printable x86, to make an executable research paper [LINK] HDR Photography in Microsoft Excel?! [LINK] Zebras All the Way Down [this video] Solving Layout Problems with CSS Grid and Friends [LINK] Console Security - Switch [LINK] Let's move SMM out of firmware and into the kernel [LINK] Bringing Linux back to server boot ROMs with NERF and Heads [LINK] Sharing the Chisel [LINK] (looks like the actual conference video wasn't released)