NYC Tech Talk Series: How Google Backs Up the Internet
Why Hacker News recommends it
Cited in backup and data-loss threads as a rare inside look at Google's backup practices, including its use of tape.
What it is
A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.
Recommended by 1 HN commenters across 1 different months, 2014 to 2021.
What Hacker News said
Recommends carapace · 2019-09 · on “Game developer loses multiplayer service code”
FWIW... "Backups are a tax you pay for the luxury of restore" "How Google Backs Up the Internet" [this video] Detailed notes: [LINK]
Cites timothya · 2014-10 · on “Dropbox confirms that a bug within Selective Sync may have caused data loss”
There may occasional bugs, but I don't know of any cases when Gmail actually lost emails. According to this talk [0], the bug referred to by that article caused some users' emails to be temporarily inaccessible, but all the emails were eventually recovered. [0]: [this video]
Cites drieddust · 2016-10 · on “Code hosting provider ceases trading after “well-orchestrated” DDoS attack”
This should serve as warning to anyone relying blindly on cheap cloud to keep themselves up and running. This also remind me of importance of offline tape backup. Google still uses tapes[1], yet some of the companies have started thinking they can eliminate tapes and move their backups to cloud. [1] [this video]
Cites stegro32 · 2016-11 · on “Don’t Mess with The Google”
I also remember that talk. I haven't verified by watching again, but if anyone's interested I suspect this is the one: [this video]
Cites twotwotwo · 2021-06 · on “Unreliability at Scale”
The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: [LINK] . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if you're trying to be more cautious than you usually want for commercial/academic online storage or even backup, what do you do? And would it work?" A lot (but not all) of the author's Queue piece talks about stats from online storage, which doesn't have some wins you can get if you're entirely about l