LSTM is dead. Long Live Transformers!
Why Hacker News recommends it
Recommended as a clear explanation of why RNNs fall short and how transformers solve it.
What it is
Leo Dirac explains how transformer models have replaced LSTMs in NLP. He gives background from bag of words through RNNs and LSTMs, then a technical deep dive into multi-headed self-attention and positional encoding, with sample code.
Recommended by 4 HN commenters across 4 different months, 2020 to 2023.
What Hacker News said
Strong rec leopd · 2023-05 · on “Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper”
I don't think you really want a 5yo's version, because that's "computers can be smart". But for an SDE1-level explanation I made this video which has gotten tons of compliments and keeps people watching year after year: [this video] It's in the context of NLP, which is where transformers started of course.
Recommends FL33TW00D · 2020-10 · on “Rethinking Attention with Performers”
I've just recently been on a journey to understand transformers in and out, here are the resources I've found managed to drill it into my head: 1. Chapters 7, 9, 10 [LINK] This was really useful to really build up to the concepts of attention (although the actual attention section is still brief). 2. [LINK] This was great for visualization and understanding that attention wasn't exclusively for transformers and actually was for RNNs first. 3. [LINK] Getting to understand how transformers actually work visually. 4. [this video] This lecture by Leo Dirac was extremely helpful to finish off with, not only because it actually includes some pseudocode but it also revisits some key topics and covers wh
Recommends peterfirefly · 2023-01 · on “The Transformer Family”
Yannic Kilcher is great but this video worked better for me: "LSTM is dead. Long live transformers!" (Leo Dirac): [this video]
Recommends qigtyofhp · 2023-02 · on “Understanding and Coding the Self-Attention Mechanism of Large Language Models”
The attention mechanism started as a simple trick to not use recurrent neural networks. Read the intro in the original paper "Attention is all you need paper" ( [LINK] ) This video explains the drawbacks to RNNs and how transformers solve that: [this video] Andrej Karpathy explains attention here: [LINK] He explains how attention is seen as a communication network: [LINK]
Cites abetusk · 2023-04 · on “We aren't close to creating a rapidly self-improving AI”
I'm not sure I understand it well enough to say but watching a video on it [0] I think there were a few key points: * "Attention is all you need" introduced positional encoding which allows you to keep context of the word, allowing for more complex translation (and thus generative/chatgpt like tasks?) because words now have context relative to each other. Contrast this with "bag of words" models that only tells you whether the word is present or not. * I don't quite understand why but transformers (which "AiaYN" introduced) can be made fully parallel, compared with the RNN/LSTM networks which has to be serial per token. Fully parallel allows for GPU optimization, which means you can take adv