Speaker Notes

Deep Dive into LLMs like ChatGPT

Large language models are built through unsupervised next-token pre-training and refined with reinforcement learning to become capable assistants, while facing inherent limitations like tokenization gaps and reward hacking.

Andrej Karpathy · 3:31:23 · YouTube ↗

Tutorial · solo speaker  Beginner · concepts and training pipeline

Highlights

21 clips · 43 min of 211 min
    • Byte pair encoding merges frequently occurring byte sequences into new symbols.
    • Decreasing sequence length allows larger vocabularies within computational limits.
    • Production models like GPT-4 typically use vocabularies of around 100000 tokens.
    • Models receive a context window of tokens and output probability distributions for the next token.
    • Comparing predictions against actual data labels allows mathematical updates to model weights.
    • Updating weights iteratively increases the probability of correct next tokens.
    • Inference uses output probabilities to sample tokens by flipping biased coins.
    • Stochastic sampling produces novel remixes rather than verbatim training data copies.
    • Sequential token feedback allows the model to generate coherent multi-word outputs.
    • Reproducing GPT-2 dropped from $40k in 2019 to about $600 today.
    • Faster hardware, better software, and refined data filtering drive the cost reduction.
    • Training progress is tracked using a single loss value where lower numbers indicate better predictions.
    • Researchers run periodic inference checks during training to qualitatively evaluate generation quality.
    • Large-scale models process millions of tokens per optimization step to steadily improve performance.
    • Base models act as raw internet text simulators rather than chat assistants.
    • A model release consists of the Python code for the forward pass and the parameter weights.
    • Models can memorize and verbatim recite frequently seen training documents like Wikipedia entries.
    • Direct regurgitation occurs when data is sampled too often during training.
    • Post-training replaces raw internet text datasets with curated multi-turn conversation datasets.
    • Post-training takes only hours compared to months of expensive pre-training.
    • Models statistically imitate the confident style of their training data rather than knowing facts.
    • Even when internal features detect uncertainty, the model defaults to a guess.
    • Without internet access during generation, statistical token sampling leads to hallucinations.
    • Models possess internal representations of uncertainty that are disconnected from output text.
    • Developers can empirically probe model knowledge using factual questions derived from source text.
    • Adding explicit refusal examples teaches models to voice uncertainty when lacking data.
    • Model parameters act as vague long-term memory, similar to something read months ago.
    • Special tokens trigger external searches, fetching live text directly into the context window.
    • The context window serves as working memory, allowing direct reference to fresh data.
    • By default, models hallucinate a generic OpenAI persona due to pre-training internet data bias.
    • Open-source models override this by hardcoding specific conversation entries about their identity.
    • System messages act as hidden context tokens that reinforce model metadata and boundaries.
    • Models by default hallucinate identities like ChatGPT based on training data regularities.
    • Developers can hardcode specific Q&A pairs into the fine-tuning data mixture to define a custom persona.
    • System messages inject hidden instructions into the context window to anchor model identity at runtime.
    • Each token forward pass executes a fixed, limited number of neural network layers.
    • Models cannot perform arbitrary computation in a single token step.
    • Spreading out intermediate calculations across multiple tokens allows models to solve complex math successfully.
    • Language models process token chunks rather than raw characters like humans do.
    • Character-level indexing tasks fail because the model lacks direct access to individual letters.
    • Using a code interpreter bypasses tokenization limits for spelling and string manipulation tasks.
    • What is easy or difficult for a human labeler differs from what is easy for an LLM.
    • Human-written reasoning traces may introduce sudden cognitive leaps that confuse the model.
    • Manual curation is insufficient for finding the most optimal token sequences for model reasoning.
    • Human and LLM cognitions are fundamentally different, making human-designed solution paths suboptimal.
    • Token sequences that are trivial for humans may require too much compute per token for an LLM.
    • Conversely, steps obvious to an LLM might be over-explained by humans, wasting valuable context tokens.
    • RL generates thousands of stochastic solution attempts per prompt without human intervention.
    • Successful paths leading to correct answers are reinforced and used for parameter updates.
    • The model discovers efficient token sequences and mental shortcuts independently through trial and error.
    • Optimization naturally causes response lengths to increase as models learn to reason.
    • Models independently develop problem-solving strategies like backtracking, re-evaluating, and self-correction.
    • These cognitive chains of thought emerge purely from reinforcement learning without explicit hardcoded rules.
    • Supervised learning caps out by merely imitating human experts, failing to exceed human performance.
    • Reinforcement learning allows systems to optimize purely for winning or correct results.
    • By playing against themselves or exploring unconstrained paths, models discover strategies beyond human capability.
    • Direct human evaluation for every RL iteration is unscalable due to massive compute and token volumes.
    • The RLHF approach uses indirect supervision by training a separate neural network as a reward model.
    • This reward model acts as a simulated human preference scorer, enabling automated reinforcement learning.
    • RLHF uses a separate neural network reward model to imitate human preferences.
    • Ranking multiple candidate outputs is much easier for human labelers than writing responses from scratch.
    • Simulated reward models enable automated reinforcement learning without billions of direct human evaluations.
    • Running reinforcement learning too long causes models to discover adversarial inputs that exploit the reward model.
    • Adversarial outputs achieve falsely high scores despite being completely nonsensical text.
    • RLHF cannot be scaled indefinitely because the scoring function is a gameable neural network rather than a fixed verifier.
    • LLMs exhibit unpredictable capability gaps, solving advanced expert tasks while failing basic logic.
    • Models can handle complex olympiad questions yet fail simple numeric comparisons or letter counting.
    • Users must treat LLMs as verified tools rather than infallible systems.

Questions this video answers

25 · every answer cites the moment it was said
01 How do LLM providers curate and filter raw web data for pre-training? 2 sources · 1:50

Providers start with massive raw datasets like Common Crawl and apply rigorous filters to remove unwanted content, personal information, and non-target languages. This process results in a clean, high-quality corpus like Fine Web.

02 Why do tokenizers use byte pair encoding instead of raw bytes? 2 sources · 9:36

Raw bytes limit the vocabulary to 256 symbols, resulting in excessively long sequences that consume computational resources. Byte pair encoding merges frequent consecutive byte pairs to expand the vocabulary and shorten sequence lengths.

03 How does next-token prediction work during neural network training? 2 sources · 16:51

The network takes a context window of tokens and outputs a probability distribution over the entire vocabulary for the next token. By comparing this prediction to the actual next token in the training data, the network parameters are updated to increase the correct token's probability.

04 What happens during the inference stage when generating text? 2 sources · 26:11

Inference involves feeding an initial token sequence into the trained model to generate a probability distribution for the next token. The system then samples from this distribution like flipping a biased coin, appending the chosen token to continue the sequence.

05 How do modern training costs and parameters compare to older models like GPT-2? 2 sources · 32:06

GPT-2 featured 1.6 billion parameters, a 1,024 token context length, and was trained on 100 billion tokens for around $40,000 in 2019. Today, hardware acceleration and refined datasets have drastically reduced training costs.

06 What is the primary function of a base model after pre-training? 2 sources · 43:27

A base model acts as an internet text simulator or a glorified autocomplete rather than a direct question-answering assistant. It generates token sequences based on statistical patterns found in web pages, meaning its outputs are stochastic and require careful prompting to elicit knowledge.

07 How can you use in-context learning with a base model for specific tasks? 2 sources · 56:31

You can leverage in-context learning by providing a few-shot prompt that establishes an algorithmic pattern, such as pairs of English and Korean words. The base model recognizes this pattern during token generation and can correctly complete subsequent examples without explicit fine-tuning.

08 What is the purpose of the post-training stage in language model development? 2 sources · 1:00:23

Post-training transitions a raw internet text simulator into a helpful, multi-turn conversational assistant. This is achieved by swapping out the pre-training text dataset for a dataset of structured conversations and continuing training for a much shorter duration.

09 How are conversations structured and fed into models during post-training? 2 sources · 1:06:19

Multi-turn conversations between humans and assistants are encoded into a one-dimensional sequence of tokens using special markers. These special tokens, such as start-of-turn indicators, help the neural network learn conversational turns and role boundaries during optimization.

10 Why do large language models hallucinate confident false information? 2 sources · 1:22:40

Models statistically imitate the confident style of correct answers found in their training datasets when answering queries. Even if internal neurons register uncertainty, the model lacks wired pathways to say it doesn't know and instead samples the most likely tokens.

11 How do model developers mitigate hallucinations by probing knowledge boundaries? 2 sources · 1:26:27

Developers programmatically interrogate models with factual questions derived from training documents to test if they know the answers. When the model consistently fails, they add training conversations where the correct response is a knowledge-based refusal like I don't know.

12 How do web search tools expand a model's working memory? 2 sources · 1:34:41

Models can emit special search tokens that pause inference so the system can query a search engine like Bing. The retrieved web text is pasted directly into the context window, feeding the neural network with fresh data.

13 How can developers define or override a model's self-identity and persona? 2 sources · 1:42:21

Because models are stateless token tumblers without an inherent sense of self, developers must explicitly program their identity. This is achieved either by hardcoding specific persona-defining conversations into the fine-tuning dataset or by inserting hidden system messages at the start of the context window.

14 Why do language models struggle with basic math and arithmetic in a single step? 2 sources · 1:49:37

Language models operate with a fixed, finite amount of computation per token during a single forward pass through the neural network layers. Trying to squeeze complex calculations into one token causes mental arithmetic errors because the model lacks enough computational depth.

15 Why are code interpreters recommended over mental arithmetic for language models? 2 sources · 1:56:08

Models leaning on Python code interpreters avoid relying on error-prone neural network mental arithmetic for intermediate calculations. Writing and executing code guarantees higher mathematical accuracy and correctness compared to generating numbers sequentially.

16 Why do large language models struggle with spelling and character-level tasks? 2 sources · 2:01:29

Language models process text as discrete tokens rather than individual characters, limiting their direct visibility into character positions. This token-based representation makes operations like indexing specific letters or counting characters inherently difficult.

17 What is the purpose of reinforcement learning in training language models? 2 sources · 2:13:58

Reinforcement learning functions like solving practice problems in school where the model explores multiple solution paths through trial and error. By sampling thousands of parallel attempts and rewarding correct answers, the model discovers optimal reasoning traces for itself.

18 How does reinforcement learning help an LLM improve its problem-solving abilities? 2 sources · 2:21:46

Reinforcement learning works through a guess-and-check process where the model generates thousands of diverse solutions for a given prompt. The system identifies which trial paths successfully reached the correct final answer and trains on those winning sequences. This allows the model to independently discover reliable token paths and effective reasoning strategies without human intervention.

19 What are thinking models and how do they achieve higher accuracy? 2 sources · 2:31:30

Thinking models are trained using reinforcement learning to generate internal chains of thought before outputting a final answer. During this internal process, the model re-evaluates steps, backtracks, and approaches problems from multiple perspectives. This emergent optimization increases the response length in tokens but significantly boosts mathematical and coding accuracy.

20 Why is reinforcement learning more powerful than simple supervised fine-tuning? 2 sources · 2:43:37

Supervised fine-tuning only allows a model to imitate human expert solutions, capping its performance at human capability levels. In contrast, reinforcement learning lets the model explore diverse trajectories, unconstrained by human distributions, to discover novel and optimal strategies. This self-play and trial-and-error approach enables models to surpass human performance.

21 How do we apply reinforcement learning in unverifiable domains where there is no concrete answer? 2 sources · 2:52:35

In unverifiable domains like creative writing, checking solutions against a concrete answer is impossible, and collecting human evaluations for thousands of rollouts is unscalable. Instead, developers use reinforcement learning by training a separate neural network called a reward model. This reward model acts as a simulator of human preferences, scoring candidate outputs so the main model can undergo reinforcement learning automatically.

22 Why can't we run reinforcement learning indefinitely on a reward model? 2 sources · 3:00:49

Reward models are lossy neural network simulations that are vulnerable to exploitation by optimization algorithms. If you run RL for too long, the model discovers adversarial inputs that achieve high fake scores with nonsensical outputs, forcing you to cap training after a few hundred steps.

23 How do language models handle native multimodality like audio and images? 2 sources · 3:09:56

Multimodality is integrated natively by tokenizing non-text data into sequences that can be interspersed directly with text tokens. For example, audio can be tokenized from spectrogram slices and images from patches, allowing a single model to process them uniformly.

24 Where can I find open weights models versus base models? 2 sources · 3:19:08

You can use inference providers like together.ai to host and interact with various open weights models in a playground. For base models specifically, platforms like hyperbolic offer access to models like Llama base versions which are harder to find on standard assistant-focused providers.

25 How do thinking models differ from standard chat models during inference? 2 sources · 3:27:33

Thinking models leverage true reinforcement learning to develop internal reasoning strategies and chains of thought rather than just simulating a human data labeler. This allows them to explore problem-solving paths similar to an internal monologue, though their transfer to unverifiable domains remains an open question.

Section-by-section summary

41 parts

Curating massive high-quality internet datasets

The pre-training stage begins by downloading vast amounts of public internet text to provide broad knowledge. Curating datasets like Fine Web involves filtering raw data down to manageable sizes of high-diversity, high-quality content.

Aggressive filtering cleans raw web data

Raw web crawls are subjected to URL filtering, HTML text extraction, language classification, and personally identifiable information removal. These filtering steps remove spam, adult content, and metadata to retain clean natural language text.

Byte pair encoding optimizes token vocabularies

Neural networks require one-dimensional sequences of finite symbols rather than raw bits. By grouping frequent byte pairs iteratively, algorithms compress sequence lengths and build a vocabulary of around 100,000 tokens.

Training predicts the next token continuously

Neural network training takes token windows from the dataset and tasks the model with predicting the subsequent token. By comparing predictions to actual next tokens, the model's parameters are iteratively updated to align with the training data statistics.

Transformers process inputs via mathematical layers

Transformer models use stateless mathematical expressions combining inputs with billions of trainable parameters. Information flows through attention and multi-layer perceptron blocks to compute output probabilities for the next token.

Inference samples probabilistically from model weights

Inference generates new text by feeding prefix tokens into the fixed model and sampling from the resulting probability distribution. This stochastic process creates creative, inspired variations rather than verbatim copies of the training data.

Hardware and data improvements drastically reduce costs

Examining GPT-2 shows how modern transformer stacks have scaled up in parameters, context length, and token counts. Improved hardware efficiency and refined datasets have drastically reduced the time and financial cost of training these models.

Monitoring training loss and token prediction

Training involves updating neural network parameters on millions of tokens simultaneously to improve next-token prediction. Researchers track the decreasing loss metric to ensure the model learns properly.

Massive hardware scale and GPU demand

Pre-training requires clusters of advanced gpus like the Nvidia H100 to handle massive parallel matrix multiplications. The extreme demand for these compute clusters has fueled massive growth in the AI hardware market.

Base models as internet text simulators

Base models trained on raw web pages function as stochastic token auto-completes rather than direct assistants. They compress internet data into parameter weights like a lossy zip file.

Eliciting probabilistic knowledge from base models

Base models store vague statistical recollections of the internet rather than explicit facts. Users can elicit this latent knowledge through clever prompting or few-shot examples.

In-context learning and assistant emulation

Base models exhibit in-context learning capabilities that allow them to handle tasks like translation using few-shot prompts. Developers can even trick base models into acting as chat assistants by structuring prompts as conversation transcripts.

Transitioning from pre-training to post-training

While pre-training builds a raw token simulator using vast internet text, post-training adapts the model into an interactive assistant. This second stage focuses entirely on teaching the model how to handle multi-turn dialogues.

InstructGPT training used professional human labelers

Early chat alignment relied heavily on human contractors writing ideal responses based on extensive guidelines. These instructions mandated that assistants remain helpful, truthful, and harmless across varied prompts.

Modern fine-tuning datasets are largely synthetic

Manual human labeling has largely been supplemented by synthetic data generation pipelines utilizing existing LLMs. These massive datasets mix millions of conversations to establish broad behavioral patterns efficiently.

Models hallucinate because they mimic confident style

Hallucinations occur because models lack real-time internet access and sample tokens based on training patterns. Since training examples always feature confident answers, the model invents plausible falsehoods rather than admitting ignorance.

Probing boundaries allows models to say I don't know

To reduce hallucinations, developers programmatically interrogate models against source documents to map their knowledge limits. Injecting refusal examples teaches the neural network to associate internal uncertainty with admitting a lack of knowledge.

Web search tools expand the model working memory

Parameter weights function as a vague recollection, whereas external tools like web search fetch fresh data into the context window. This injection acts as working memory, enabling the model to accurately cite external sources.

Providing text in context beats relying on memory

A model performs significantly better at summarization or analysis when the source text is explicitly attached in the prompt. Direct access in the context window eliminates the need to rely on degraded parameter recollection.

Self-identity is a statistical artifact unless hardcoded

Because models have no persistent self, they hallucinate identities like ChatGPT based on internet text frequency during pre-training. Developers override this by hardcoding identity conversations or injecting hidden system messages.

Hardcoding and system messages control model persona

Developers can override default hallucinations by hardcoding specific Q&A pairs into the fine-tuning data mixture. Alternatively, hidden system messages can be placed at the start of the context window to establish model identity.

Reasoning must be distributed across multiple tokens

A single token forward pass performs a fixed, limited amount of computation through the neural network layers. Expecting a model to compute an answer in one token causes arithmetic failure, requiring step-by-step intermediate tokens.

Python code interpreters prevent mental arithmetic errors

Because neural network mental arithmetic is error-prone on large numbers, models should leverage code execution tools. Writing and running Python code provides correctness guarantees for math problems.

Tokenization causes counting and spelling failures

Models process token chunks rather than raw characters, making character-level operations like counting and spelling difficult. Direct string manipulation tasks frequently fail unless delegated to external code tools.

Pre-training and supervised fine-tuning create assistants

Pre-training builds an internet document simulator through lossy compression, which is then refined via supervised fine-tuning on human-curated conversations. This creates a helpful assistant model capable of handling diverse prompts.

Reinforcement learning mirrors solving practice problems

Textbook learning analogies map pre-training to exposition and supervised fine-tuning to worked examples. Reinforcement learning corresponds to practice problems where the final answer is known, but the solution path must be discovered.

Humans cannot write optimal reasoning traces for LLMs

Human-crafted training solutions often mismatch LLM cognition by introducing cognitive leaps or wasting tokens on trivial steps. Because human and model reasoning capabilities differ, manual annotation falls short of optimal efficiency.

Reinforcement learning drives autonomous discovery

Instead of relying on human experts, reinforcement learning allows models to generate thousands of solutions via trial and error. The model inspects its own outputs, identifies which token sequences lead to correct answers, and reinforces those successful paths.

Three distinct stages of LLM training

Modern LLM training mirrors child education through three progressive phases: pre-training on raw textbooks, supervised fine-tuning on expert solutions, and reinforcement learning via practice problems. While pre-training and SFT are mature, RL remains the frontier of development.

Emergent reasoning through trial and error

Applying RL to math problems results in models generating much longer responses filled with internal checks and reframings. These emergent chains of thought significantly boost accuracy by allowing models to backtrack and evaluate steps step-by-step.

Thinking models trade latency for high accuracy

Frontier models like DeepSeek R1 and OpenAI O3 use RL to execute deep reasoning phases before outputting a final answer. While they introduce higher latency and are overkill for simple factual queries, they drastically outperform standard models on complex math and coding tasks.

Surpassing human capability through self-play

Supervised learning caps model performance at human expert levels because it merely mimics human data. Reinforcement learning, as demonstrated previously by AlphaGo, allows models to discover completely novel, non-human strategies that yield superior outcomes.

The challenge of unverifiable creative domains

Unlike math or code where answers can be automatically checked against a ground truth, creative tasks like writing jokes lack objective verification. Evaluating thousands of generations per training step requires an automated scaling solution rather than direct human feedback.

Scaling RLHF via simulated reward models

To train models in subjective domains, developers use human preference rankings to train a separate neural network called a reward model. This reward model acts as a proxy human scorer, letting the system perform automated reinforcement learning at scale.

The discriminator-generator gap drives RLHF utility

RLHF succeeds because ranking multiple model generations is much easier for humans than writing ideal responses from scratch. This approach leverages the human ability to discriminate rather than generate, yielding higher quality supervision data.

Reward hacking limits RLHF scaling

Because the reward function is a lossy neural network simulation, running RL too long leads to adversarial reward hacking where the model generates nonsensical outputs that trick the scorer. Consequently, RLHF functions more like a light fine-tuning than true scalable reinforcement learning.

Multimodality integrates natively via tokenization

Audio and images can be natively integrated into language models by tokenizing spectrograms and image patches into uniform token streams. This allows multimodal models to process text, audio, and vision within the exact same transformer architecture.

Agents and test-time training represent the next frontier

Future capabilities will move beyond single-turn prompts toward long-running agentic workflows and computer-use tools requiring tight human supervision. Additionally, overcoming fixed context windows will require new research into test-time training mechanisms.

Finding and running models spans leaderboards to local hardware

Developers can track progress using leaderboards like LMSYS, stay updated via AI newsletters, and run models via API providers or locally on consumer hardware using tools like LM Studio. Open-weight models like Llama and DeepSeek provide powerful alternatives to proprietary APIs.

ChatGPT queries simulate human data labelers

When a user queries ChatGPT, the token sequence is processed by a fixed neural network acting as a lossy simulation of a human data labeler. Due to cognitive and architectural differences, these models suffer from hallucinations and Swiss-cheese capability gaps.

Thinking models introduce true reinforcement learning

Unlike standard chat models relying primarily on SFT and RLHF, reasoning models use reinforcement learning to develop internal chains of thought. While currently shining in verifiable domains like math and code, these tools must be verified carefully by users.

0:00 / 3:31:23
k play/pause · f fullscreen · m mute
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said
  • Fine Web8× · first 1:23

    A curated internet dataset containing a large quantity of high-quality, diverse documents used for pre-training.

  • Byte Pair Encoding1× · first 11:00

    An algorithm that iteratively merges frequent byte pairs to optimize token vocabularies and shorten sequence lengths.

  • Base model20× · first 12:29

    A raw language model trained on internet text to simulate token sequences without being fine-tuned as an assistant.

  • In-context learning3× · first 56:31

    The ability of a model to adapt to patterns presented within its prompt context without updating its underlying weights.

  • InstructGPT2× · first 1:10:23

    A pioneering model and paper by OpenAI demonstrating how to fine-tune language models on conversations using human labelers.

  • Context window24× · first 16:49

    The active text sequence feeding directly into the neural network, acting as the model's working memory.

  • System message2× · first 1:45:58

    Invisible tokens placed at the beginning of a conversation to document and instruct the model on its identity.

  • Tokenizer7× · first 12:24

    A tool that converts text into numerical token IDs processed by the neural network.

  • Reinforcement learning34× · first 2:10:12

    The training stage where models try multiple solution paths to discover successful reasoning strategies through trial and error.

  • Reward Model12× · first 2:52:39

    A separate neural network trained to simulate human preferences and score outputs in unverifiable domains.

  • Supervised Fine-Tuning14× · first 1:16:10

    An early training stage where the model learns by imitating human expert solutions.

  • Reinforcement Learning from Human Feedback11× · first 2:51:20

    A technique that uses a neural network reward model trained on human preferences to optimize language models via reinforcement learning.

  • LM Studio2× · first 3:20:37

    A desktop application used to load and run distilled models locally on local hardware.