State of GPT | BRK216HFS

What Hacker News said

Strong rec sagarpatil · 2023-06 · on “GPT Best Practices”

I would highly recommend watching - State of GPT video by Karpathy at Microsoft Build to get an idea of how LLM’s work. There are some really good prompting ideas. It changed the way I use GPT-4. Video: [this video]

Recommends mcint · 2023-06 · on “It’s infuriatingly hard to understand how closed models train on their input”

I share many of your concerns and frustrations, although I suspect what you're asking for their consider a moat along the lines of a trade secret, rivaled only by the collection of performance improvement techniques they've amassed in 1000s-10,000s of training runs, 100s of engineers, and (hundreds of?) millions spent on compute. People are hired and praised in the community for their skill in cleaning data. A non-answer for you, but for curious others, [State of GPT] 10 days ago provides a through introduction to the process used to train, Karpathy speaking at a Microsoft event providing a deep summary review of concepts, training phases, and techniques proving useful in the world of Genera

Recommends valgaze · 2023-06 · on “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT”

I recognize there's plenty of catnip here when it comes to calling this "engineering" or not, however, whatever you want to call it (prompt fiddling?), the techniques are crucial if you want to achieve reasonably consistent output from current-state LLMs. As models improve concerns about context window limitations will be reduced and it will be easier to discern user intent. These are good straight-to-the-point guides: - Prompt Engineering by BrexHQ: [LINK] - OpenAI guidance: [LINK] - [LINK] - (great examples): [LINK] - (~hr) Karpathy talk: [this video] tl;dr: - Begin with the best and most forgiving model available, optimizing it later if necessary. - Write your prompts in a clear, specific, and

Recommends jsight · 2023-06 · on “The future of AI according to thousands of forecasters”

Karpathy explains it far better than I did in his "State of GPT" talk about a month ago: [this video] Yes, language can do that, but books and other texts, on average, leave out a lot of steps that humans have learned to do naturally. They can be prompted to do them, and their answers improve, indicates that there is definitely some level of "reasoning" there. But it has weird limitations in its current forms. I expect that to improve pretty quickly, tbh.

Recommends tikkun · 2024-11 · on “Ask HN: Recommendation for a SWE looking to get up to speed with latest on AI”

It sounds like you want more broad stuff, not necessarily learning how to train models. More like learning to use them and how they work. [LINK] and Hacker's Guide to LLMs by Jeremy from Fast.ai - [LINK] State of GPT by Karpathy - [this video] LLMs by 3b1b - [LINK] Visualizing transformers by 3b1b - [LINK] How ChatGPT trained - [LINK] AI in a nutshell - [LINK] How Carlini uses LLMs - [LINK] For staying updated: X/Twitter & Bluesky. Go and follow people that work at OpenAI, Anthropic, Google DeepMind, and xAI. Podcasts: No Priors, Generally Intelligent, Dwarkesh Patel, Sequoia's "Training Data"

Recommends tintor · 2025-02 · on “RLHF Book”

Thanks. Here is presentation by Karpathy explaining different stages of LLM training. Explains many details in a form suitable for beginners. [this video]

Cites jiggawatts · 2023-05 · on “Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?”

Chat GPT 4 has ongoing training, such as using Reinforcement Learning from Human Feedback (RLHF) to tune it to provide "better" responses, "safer" answers, and to generally obey the system prompts. There's a release every few weeks. Yes, I've noticed too that recently it has become very "cagey", qualifying everything to death with "As an AI model...". A paper[1] that took snapshots monthly mentioned that as the initial bulk self-supervised learning went on, the model became smarter, as expected. However, once the "clicker training" was imposed on top to make it behave, its peak capabilities were reduced. I'm not sure if it's in the paper or the associated video, but the author mentioned that

Cites ra · 2023-06 · on “AI: First New UI Paradigm in 60 Years, Jakob Nielsen”

Kaparthy explained how LLMs can retrospectively assess their own output and judge if they were wrong. Source: [this video]