Let's build GPT: from scratch, in code, spelled out.
Why Hacker News recommends it
It is the go-to recommendation for learning how transformers and LLMs actually work, praised for its clarity and Karpathy's teaching.
What it is
Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.
Recommended by 14 HN commenters across 8 different months, 2023 to 2025.
What Hacker News said
Strong rec yonz · 2023-01 · on “The Transformer Family”
Great compilation, would be great to see Vision Transformer (ViT) included. Andrej Karpathy's GPT video is a must have companion for this [this video] I was going nuts trying to grok Key, Query,Position and Value until Andrej broke it down for me.
Strong rec durdn · 2023-02 · on “GPT in 60 Lines of NumPy”
Very impressive. Recently I watched this really amazing lecture on building GPT from scratch from Karpathy, I was blown away: [this video]
Strong rec wesammikhail · 2023-05 · on “Google DeepMind CEO Says Some Form of AGI Possible in a Few Years”
Andrej Karpathy, the director of AI at tesla, published an AMAZING video about how the current models are built which I think everyone in the tech space should watch. It is intuitive, easy to follow and quite frankly the best video I have seen on the topic. Here is the link: [this video] If you conclude, after watching this, that this is all intelligence is - namely cascading optimization and data structure problems chained in a row, then you and a fair number of people here wont ever find common ground. That is not to say that you're wrong, it just seems as if there is a missing component here that we haven't yet discovered.
Strong rec villedespommes · 2024-01 · on “Implementing a ChatGPT-like LLM from scratch, step by step”
+1, Andrey is an amazing educator! I'd also recommend his [this video] and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.
Recommends amrb · 2023-01 · on “Deep Learning Tuning Playbook”
I recommend this video on the subject. "Let's build GPT: from scratch, in code, spelled out." [this video]
Recommends qazpot · 2023-03 · on “Flow-Based Programming, a way for AI and humans to develop together”
I think you are taking a very major assumption that LLMs are deterministic infact they are exactly opposite. They are probabilistic systems. - They do not transpile Rough English to Deterministic English. Infact they do not do any transpiling at all. - LLMs learn the probabilities of words in the dataset for all contexts from that dataset (context is a ordered set of words). This is called training. - Once training is done, LLMs can generate text given a prompt and the probabilities that it has learned. The analogy of LLMs being auto-complete on steroids is very apt. - Whether the text generated by LLMs is factual or not is purely coincidental. I would highly recommend watching and working t
Recommends bfg_damien · 2023-04 · on “Ask HN: How to Learn LLM Fundamentals”
Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: [LINK] 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: [LINK] 3. Another thread specifically on large language models: [LINK] Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy on youtube. First video here: [this video]
Recommends pankajdoharey · 2023-04 · on “What Are Transformer Models and How Do They Work?”
I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": [this video] Additionally, for more comprehensive resources on Transformers, you may find these resources useful: * The Illustrated Transformer by Jay Alammar: [LINK] * MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: [LINK] * Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): [LINK] ... These resources cover different aspects