Transformers, the tech behind LLMs | Deep Learning Chapter 5

What Hacker News said

Strong rec BenoitP · 2024-04 · on “Visualizing Attention, a Transformer's Heart [video]”

3Blue1Brown is really a godsend. I had been trying and failed to get an intuition for the self-attention mechanisms, and also why you want to stack you transformer layers. I went through a lot of material (Key-Query-Value interpretations, SVD of these, etc.) but his comment in chapter 5 video [1] made it click for me: "But you should think of the primary goal of this network that it (the word embedding) flows through as being to enable each one of those vectors to soak up a meaning that's much more rich and specific than what mere individual words could represent". Here is my intuition: Transformers are merely a Convnet for token sequences. In the sense that information is first local and ea

Strong rec Wilduck · 2024-05 · on “Ask HN: Why can't ChatGPT solve Wordle-type questions?”

3Blue1Brown has a fantastic video on GPT that I'd highly recommend called "But what is a GPT? Visual intro to transformers" [1]. You'll need to have some background in mathematical reasoning, but not too much. He does a great job of drawing pictures to show the mathematical transformations that happen under the hood. [1] [this video]

Strong rec tomashm · 2024-07 · on “The Illustrated Transformer (2018)”

This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [this video] 1. [LINK]

Strong rec manmal · 2025-05 · on “Embeddings are underrated (2024)”

This video explains the direction-encodes-trait topic very well IMO: [this video] It’s the first in a series of three that I can very highly recommend. > there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I don’t know about that! Once you go very high dimensional, there is a lot of direction vectors that are almost perfectly perpendicular to each other (meaning they can cleanly encode a trait). Maybe they don’t even need to be perfectly perpendicular, the dot product just needs to be very close to zero.

Strong rec gf000 · 2026-01 · on “King – man and woman is queen; but why?”

I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other. [this video] This even opens up a more data-based approach to linguistics, where it is also heavily used.

Strong rec robrenaud · 2026-06 · on “"They're made out of weights"”

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs. It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop. [this video]

Recommends CharlesW · 2024-04 · on “Ask HN: If AI can be creative, does this mean humans can be taught creativity?”

To quote GPT-4, "AI's 'creativity' is more about remixing and reinterpreting the information it has been fed". I think the degree to which a person perceives human-like creativity from AI/LLMs is inversely proportional to their understanding of how they work. Here's a great video that can help demystify GPTs: [this video]

Recommends MAXPOOL · 2024-07 · on “The Engineer’s Guide to Deep Learning: Understanding the Transformer Model”

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [LINK] 2/ Transformers from Scratch [LINK] 3/ Andrej Karpathy has really good series of intros: [LINK] Let's build GPT: from scratch, in code, spelled out. [LINK] GPT with Andrej Karpathy: Part 1 [LINK] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [this video] Attention in transformers, visually explained | Chapter 6, Deep Learning [LINK] Full 3Blue1Brown Neural Networks playlist [LINK]