Reinventing Entropy | Compression is Intelligence Part 1
Why Hacker News recommends it
HN commenters recommend it as an intuitive introduction to information theory and entropy, and as background for discussions of LLMs as next-token predictors and lossy compressors of language.
Recommended by 6 people across 4 different months, 2026 to 2026.
What it is
Grant Sanderson of 3Blue1Brown opens a series on the idea that compression is intelligence. He builds up from a warmup prediction game to Shannon's definition of information and entropy, with a focus on how compressible language is. Animations are made with manim, and the video relates the ideas to language models such as nanoGPT.
What people said
Strong rec jamwise · 2026-06 · Hacker News · on “How information theory saved my word game”
Information theory has been a really fascinating topic to get more acquainted with. Not really related to the crossword, but I highly recommend 3Blue1Brown's video "Compression is Intelligence", [this video]
Strong rec sheeeeesh · 2026-08 · Hacker News · on “Compression is prediction”
Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - [this video]
Recommends ithkuil · 2026-06 · Hacker News · on “πFS”
You'll find this an interesting watch: Reinventing Entropy Compression is Intelligence Part 1 3blue1brown [this video]
Recommends lukifer · 2026-08 · Hacker News · on “Position: LLMs Can't Jump”
The lossy compression of language is why we should find it unsurprising that LLMs tend to perform better at code, than at human language tasks or reasoning. While there can be subtle semantic differences in real codebases (using "null" to mean "unknown" in one context, versus "intentionally blank" in another), there is a much tighter coupling of semantics to meaning (low ambiguity) compared to "love" in English (let alone any inexpressible je ne sais quoi). With apologies if this is common knowledge at this point, 3Blue1Brown has been doing an excellent series on compression, and its relationship to intelligence (or more controversially, that they are one and the same): [this video] But that also
Recommends mpodeley · 2026-09 · Hacker News · on “Can AI design circuit boards yet?”
“Next-token prediction” describes the output format, not the computation required to choose each token. During training, models develop internal representations of concepts, constraints, possible futures, and algorithms. The PCB agent also writes circuit code, runs simulations, reads failures, and revises the design. It isn’t one-shot autocomplete. Astra and Fable are already hard to square with “mere autocomplete.” We may be (really) close to AGI, and token-by-token generation certainly doesn’t rule out subjective experience (I think we should at least treat that as an open question). Great videos: [LINK] [LINK] [this video] [LINK]
Recommends yhjc2692 · 2026-10 · Hacker News · on “You could have invented transformers from memory”
For anyone interested, here is a recent 3Blue1Brown series on "Compression is Intelligence": [this video] (For anyone not familiar with 3B1B, it's a YouTube channel that visualizes intriguing math concepts to explain them intuitively!)
Cites john01dav · 2026-08 · Hacker News · on “I'm becoming AI-blind”
> There's some psychological mechanism by which my brain immediately recognizes AI generated text and just short-circuits to "there is no information here". The roots of llm math in part lie in compressing natural language such that there's only information there, and then running the reverse to create way more text without new information in a somewhat precise theoretical sense. Some more information: [this video]