The spelled-out intro to language modeling: building makemore
Why Hacker News recommends it
HN recommends it as a long but worthwhile way to demystify how language models work technically.
What it is
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
Recommended by 2 HN commenters across 2 different months, 2024 to 2024.
What Hacker News said
Strong rec islewis · 2024-02 · on “Markov Chains are the Original Language Models”
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [this video]
Strong rec georgehill · 2024-04 · on “Show HN: Next-token prediction in JavaScript”
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [this video] - [LINK] - [LINK] 1: ex: [LINK]
Cites chrsig · 2024-02 · on “GPT in 500 Lines of SQL”
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [this video]
Cites caprock · 2024-07 · on “The n-gram Language Model”
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [this video] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.