Dev Deletes Entire Production Database, Chaos Ensues
Why Hacker News recommends it
Shared as a funny, accurate retelling of a classic outage, especially on a bad ops day.
What it is
Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.
Recommended by 1 HN commenters across 1 different months, 2023 to 2024.
What Hacker News said
Recommends xyst · 2024-08 · on “Gitlab is reportedly up for sale”
Every time GL is mentioned, I am reminded of the 2017 incident where a person 'rm -rf' the primary pg data logs. The reenactment by Kevin Fang is always a good source of comedy on a bad day. [this video] It’s a shame they are selling out, but I suppose it was bound to happen sooner or later. GH is vacuuming up all of the market share. I kind of liked the ability to self host my own GL. But for small/personal use cases. It was a bit overkill.
Cites sovietmudkipz · 2023-07 · on “Gitlab.com is down”
Here is a retelling of the time when gitlab went down because someone deleted a production database: "Dev Deletes Entire Production Database, Chaos Ensues" [this video]
Cites laserDinosaur · 2023-09 · on “Nginx Unit – Universal web app server”
"When a program has nothing surprising to say, it should say nothing." Which sounds good in theory, until you run into issues like this: [this video] TLDW: Most of the Gitlabs outage where they deleted the primary prod and backup prod database could have been solved if they just waited for the command line op to complete - but because of a lack of feedback they thought it had frozen, leading to several other plans which made the situation worse.