Speaker Notes

Let's reproduce GPT-2 (124M)

Andrej Karpathy demonstrates how to build and train GPT-2 (124M) from scratch in PyTorch, covering model architecture, optimization tricks, performance engineering like FlashAttention, distributed training, and evaluation.

Andrej Karpathy · 4:01:26 · YouTube ↗

Tutorial · solo speaker  Advanced · PyTorch implementation and model architecture

Highlights

25 clips · 54 min of 241 min
    • GPT-2 treats positional embeddings as learned parameters rather than fixed sinusoids.
    • Optimization causes raw parameter tables to spontaneously develop sinusoidal-like structures.
    • Smoothness in embedding curves correlates with training progress and model maturity.
    • Layer normalizations are placed before attention and MLP blocks inside the residual stream.
    • A clean residual stream ensures uninhibited gradient flow straight back to the input tokens.
    • Attention acts as a communication reduce operation while MLPs act as individual token maps.
    • GeLU avoids the dead neuron problem by providing local gradients everywhere instead of flat zeros.
    • The approximate GeLU using tanh was originally created due to slow error function evaluations in TensorFlow.
    • Modern networks evolve past this variant, but exact replication requires the approximate GeLU.
    • Loading OpenAI weights into a custom architecture requires mapping state dict keys and filtering buffers.
    • TensorFlow-trained weights often require manual transposition to match PyTorch expectations.
    • Custom implementations validated against open-source checkpoints confirm architectural parity.
    • Wrap generation loops in torch.no_grad() to save memory and avoid tracking gradients.
    • Top-K sampling clamps all probabilities below the 50th rank to zero and renormalizes.
    • Top-K filtering prevents language models from generating rare tokens and going off the rails.
    • Reshape flat token lists into 2D batch tensors of shape (batch, time) using view().
    • Fetch a buffer of size (batch * time + 1) to ensure every input token has a corresponding target label.
    • Offset the target tensor by one index relative to the input tensor for next-token prediction.
    • PyTorch's cross_entropy function requires flattening multi-dimensional logits to 2D (B * T, vocab_size).
    • Targets must also be flattened to a 1D tensor of size (B * T) to match logits during loss computation.
    • A lightweight data loader can iterate through text files in steps of batch_size * time.
    • Loop back to the beginning of the dataset automatically when reaching the end of the text.
    • The input token embedding layer and the output language modeling head share the exact same weight matrix.
    • Weight tying saves approximately 30% of parameters in a 124M model.
    • Sharing weights enforces the inductive bias that semantically similar tokens should have similar embeddings and probabilities.
    • Adding residual connections repeatedly causes the variance of activations to grow with network depth.
    • Scaling down the weights of residual layers by 1 over the square root of the number of residual layers stabilizes the forward pass.
    • Residual streams accumulate variance across layers, growing standard deviation during forward passes.
    • GPT-2 scales residual layer initialization weights down by 1 over the square root of the number of residual layers.
    • This normalization controls activation growth and stabilizes training depth.
    • Tensor cores spend much of their time idle waiting for data to be loaded.
    • Memory bandwidth limits throughput because fetching data from global memory is slow.
    • Well-tuned applications often achieve only moderate hardware utilization due to these data movement bottlenecks.
    • TF32 uses the same sign and exponent bits as fp32 but truncates the mantissa down to 19 bits.
    • The truncation happens internally in hardware instructions without changing PyTorch code.
    • It provides an 8x speedup with empirical results showing negligible loss in training accuracy.
    • Standard PyTorch dispatches separate kernels for elementwise operations, causing round trips to global memory.
    • Torch compile detects full code graphs and fuses adjacent elementwise operations into a single pass.
    • Keeping working chunks on-chip avoids expensive memory bandwidth costs.
    • Performs an algorithmic rewrite using the online softmax trick
    • Eliminates high-bandwidth memory reads and writes of large attention matrices
    • Achieves faster runtimes despite executing more total flops
    • Ugly vocabulary numbers force inefficient Cuda boundary kernels
    • Padding vocabulary from 50257 to 50304 fits Cuda block tiles
    • Yields a measurable performance boost with harmless unused tokens
    • Exclude 1D tensors like biases and layer norms from weight decay
    • Weight decay acts as regularization by preventing individual weights from growing too large
    • Fused AdamW combines update kernels to reduce overhead and improve speed
    • Large model batch sizes often exceed physical GPU memory limits.
    • Gradient accumulation allows simulating any arbitrary batch size serially by summing gradients across multiple forward-backward passes before updating.
    • Large target batch sizes cannot fit in GPU memory directly
    • Gradient accumulation runs multiple micro-batches and sums their gradients
    • Simulates an arbitrary batch size sequentially before performing an optimizer step
    • Standard loss functions like MSE and cross-entropy average (mean reduction) over batch elements.
    • Accumulating gradients via loss.backward() is mathematically equivalent to a sum, which loses the division normalizer.
    • You must explicitly divide the loss by the gradient accumulation steps to match the gradient of a single large batch.
    • Distributed Data Parallel (DDP) runs identical code across multiple processes assigned to separate GPUs.
    • During the backward pass, DDP automatically triggers an all-reduce operation to average gradients across all ranks before depositing them.
    • When using gradient accumulation with DDP, you want to avoid syncing gradients on every micro-step to prevent massive communication overhead.
    • Manually toggling the model's required backward gradient sync flag allows accumulation without context managers or code duplication.
    • Set sequence length and batch sizes to match specific scaling laws like GPT-3.
    • Use 375 million tokens for learning rate warm-up steps to mirror established baselines.
    • Use a separate torch Generator for sampling to avoid polluting the global training RNG state.
    • Disable torch.compile temporarily if it conflicts with custom evaluation or sampling generation code.
    • Smaller models cannot handle traditional multiple-choice tasks, so evaluate via token completion probabilities instead.
    • Construct batch matrices for the shared context and possible completions, picking the option with the lowest average loss.
    • Repeating the exact same document ordering across epochs creates spurious correlations and periodicity in loss curves.
    • Implement random permutations of documents within shards and shuffle shard order on every new epoch.
    • The PyTorch nanogpt implementation serves as a clean reference for a pure C/CUDA implementation (llm.c).
    • A dedicated C/CUDA implementation for gpt2/3 training can reduce overhead and speed up training steps while matching exact loss values.

Questions this video answers

26 · every answer cites the moment it was said
01 How do I inspect the tensor shapes of the original GPT-2 checkpoint? 2 sources · 5:38

You can load the model using Hugging Face Transformers and extract its state dictionary to inspect key-value tensor shapes. This reveals the dimensions of components like the 50257-by-768 token embedding lookup table.

02 How are positional embeddings structured in GPT-2 compared to the original Transformer? 2 sources · 7:28

Unlike the original Transformer which used fixed sinusoidal functions, GPT-2 treats positional embeddings as trainable parameters initialized randomly from scratch. During optimization, these embeddings organically develop smooth sinusoidal and cosine structures.

03 What architectural changes did GPT-2 make to the original Transformer layer normalization? 2 sources · 14:29

GPT-2 moves the layer normalizations before the attention and multi-layer perceptron blocks rather than after them. It also adds an extra layer normalization right before the final classification head to maintain a clean residual stream.

04 Which activation function does GPT-2 use in its MLP blocks? 2 sources · 21:02

GPT-2 uses an approximate version of the Gaussian Error Linear Unit activation function backed by a tanh approximation. This avoids the flat zero-gradient regions seen in standard ReLUs and provides smooth, continuous gradients during optimization.

05 How do I prepare text data into batches for Transformer training? 2 sources · 48:06

Tokenize the text into a sequence of integers and view it as a two-dimensional batch tensor. You should fetch an extra token beyond the sequence length so you can create parallel input and target tensors offset by one position.

06 What is the expected initial cross-entropy loss for a random model? 2 sources · 56:01

The expected initial loss corresponds to the negative natural logarithm of a uniform probability distribution over the vocabulary. For a vocabulary size of 50,257, this evaluates to roughly 10.82, which sanity checks our initialization.

07 How do I handle multi-dimensional tensors when calculating cross-entropy loss? 2 sources · 54:06

PyTorch's cross entropy function cannot accept multi-dimensional inputs directly. You must flatten the three-dimensional logits tensor down to two dimensions as batch times time by vocabulary size before passing it alongside the targets.

08 Why is weight tying used in Transformer models like GPT-2? 2 sources · 1:08:02

Weight tying reuses the same weight matrix between the bottom token embeddings and the top language modeling head. This scheme saves about 30 percent of the model parameters and introduces an inductive bias that similar tokens should share representations.

09 How are the weights initialized according to the GPT-2 specifications? 2 sources · 1:14:09

Linear layer weights are initialized from a normal distribution with a standard deviation of 0.02, while biases are set to zero. Additionally, residual layer weights are scaled down by the square root of the number of residual layers to prevent activation growth.

10 How does tensor float 32 speed up matrix multiplications? 2 sources · 1:31:14

Tensor float 32 maintains the standard sign and eight exponent bits of a 32-bit float but truncates the mantissa down to 19 bits. This reduction happens internally within the GPU tensor core instruction, allowing matrix multiplications to execute significantly faster. Because the accumulation and outputs remain in float32, this approximation is empirically unnoticeable during deep learning training.

11 Why is GPU synchronization necessary when timing training iterations? 2 sources · 1:33:32

Because the CPU asynchronously schedules work and queues up kernels to run on the GPU, execution does not happen instantly. Without explicit synchronization, code timing measurements can capture just the CPU scheduling duration rather than the actual GPU workload execution time. Calling CUDA synchronize forces the CPU to wait until the GPU finishes all scheduled work before taking timestamps.

12 How does torch compile eliminate Python overhead and fuse kernels? 2 sources · 1:51:00

Torch compile analyzes the entire neural network structure ahead of time instead of executing operations sequentially through the python interpreter. It identifies element-wise operations and performs kernel fusion to keep data chunks on the GPU chip for multiple calculations. This avoids expensive round trips between high bandwidth memory and processing units, significantly speeding up training throughput.

13 How does FlashAttention speed up Transformer models? 2 sources · 2:01:31

FlashAttention fuses attention operations and avoids materializing the massive attention matrix in high-bandwidth memory. Even though it performs more total floating-point operations, it drastically reduces expensive HBM memory round trips and speeds up execution significantly.

14 Why should I pad the vocabulary size to a power of two? 2 sources · 2:09:13

Padding awkward vocabulary sizes like 50257 to powers of two like 50304 helps Cuda kernels execute more efficiently using standard block tiles. Although it introduces slight compute overhead, it prevents inefficient boundary kernels and improves performance by several percent.

15 What does gradient norm clipping prevent during training? 2 sources · 2:17:22

Gradient norm clipping caps the global norm of parameter gradients at a specified threshold like 1.0. This prevents occasional bad data batches from causing massive gradient spikes that could destabilize or shock the model optimization.

16 How do I implement cosine decay with warmup for learning rates? 2 sources · 2:21:14

A cosine decay schedule starts with a linear warmup phase from near zero up to a maximum learning rate, followed by a cosine curve down to a minimum rate. While PyTorch has built-in schedulers, writing a custom function over iteration steps offers clear transparency and full control.

17 Why should I separate weight decay for 1D and 2D parameters? 2 sources · 2:30:15

Weight decay regularizes models by preventing weights from growing too large, but it should not be applied to 1D tensors like biases and layer norm scales. Instead, weight decay is restricted to 2D matrix multiplication weights and embeddings to properly distribute work across channels.

18 How can I simulate large training batch sizes with limited GPU memory? 2 sources · 2:36:39

Gradient accumulation lets you process multiple smaller micro-batches sequentially while adding up their gradients via backward passes. Once the target token count is reached across these steps, a single optimizer update is performed to simulate a massive batch size.

19 How do I run training across multiple GPUs using DDP in PyTorch? 2 sources · 2:47:16

Launch your training script using the torchrun command, which automatically sets up environment variables like rank and world size for parallel processes. Each process assigns itself to a specific CUDA device using its local rank and runs an identical copy of the model.

20 How do I prevent data loaders from feeding the exact same data to every GPU? 2 sources · 2:56:59

Modify your data loader to accept the process rank and total number of processes to stride through dataset chunks. Each rank initializes at a different offset and advances its position by the total number of processes multiplied by the batch size and sequence length.

21 How should I handle gradient synchronization when using gradient accumulation with DDP? 2 sources · 3:02:30

By default, DDP synchronizes gradients across all GPUs after every backward pass, which is wasteful during intermediate gradient accumulation steps. You can avoid this by toggling require_backward_grad_sync to false during intermediate micro-steps and enabling it only on the final step before the update.

22 How do I prepare and structure large web datasets for pre-training? 2 sources · 3:14:18

You can write a script to download the dataset from Hugging Face, tokenize the documents with special start tokens, and save them into fixed-size numpy file shards on disk. Using uint16 arrays saves space, and sharding into chunks of 100 million tokens makes large datasets much easier to manage during training.

23 How do I implement HellaSwag evaluation without multiple-choice capability? 2 sources · 3:32:46

Since small models cannot handle multiple-choice labels directly, you construct a batch of token completions pairing the shared context with each of the four possible options. You then evaluate the cross-entropy loss or token probabilities for each option and pick the one with the highest average probability.

24 How do I structure a multi-epoch training run and save checkpoints? 2 sources · 3:48:46

You can scale up training by multiplying your epoch count such as running four epochs for an overnight run while periodically saving model checkpoints alongside optimizer states. Saving the optimizer state dictionary is essential for preserving Adam buffers like m and v if you intend to resume optimization later.

25 How does supervised fine-tuning transition from pre-training? 2 sources · 3:55:46

Transitioning to supervised fine-tuning involves swapping out the pre-training text dataset for a conversational one with a user-assistant structure. You continue training by filling in user tokens and sampling assistant tokens, which adapts the document-dreaming model into a conversational chat format.

26 How does a pure C and CUDA implementation compare to PyTorch? 2 sources · 3:57:32

You can implement a parallel training loop in pure C and CUDA to achieve faster startup and higher tokens-per-second performance than general frameworks like PyTorch. While PyTorch handles general neural networks, a dedicated implementation like llm.c directly targets GPT architectures to squeeze out maximum hardware speed while producing identical loss curves.

Section-by-section summary

44 parts

Reproducing the 124M GPT-2 model from scratch

The project aims to reproduce the 124 million parameter GPT-2 model using modern hardware, achieving comparable results in about an hour for roughly ten dollars. OpenAI's original release provides scaling laws and model weights, while the GPT-3 paper supplies clearer hyperparameter details.

Inspecting OpenAI checkpoint tensors and shapes

Loading the original TensorFlow-based GPT-2 weights via Hugging Face Transformers provides a PyTorch-compatible state dict. Inspecting the tensors reveals exact dimensions for token embeddings, position embeddings, and transformer layers.

Positional embeddings learn sinusoidal structures organically

Unlike the fixed sinusoidal encodings in the original Transformer paper, GPT-2 treats position embeddings as learned parameters. During optimization, these weights naturally develop smooth, sinusoidal-like patterns across channels and positions.

Building a custom GPT-2 architecture from scratch

While Hugging Face code provides a working baseline, writing a custom GPT-2 implementation from scratch ensures full comprehension. The architecture removes the encoder and cross-attention of the original Transformer, shifting layer normalizations inside the residual blocks.

Structuring the GPT-2 container and transformer blocks

The model skeleton uses PyTorch module dicts and module lists to mirror Hugging Face's naming conventions for easy weight loading. Each transformer block preserves a clean residual stream while placing layer normalizations before attention and MLP operations.

Utilizing approximate GeLU in the MLP blocks

The multi-layer perceptron uses two linear projections sandwiched around an approximate GeLU nonlinearity implemented with tanh. This choice mirrors historical TensorFlow limitations but prevents dead neurons by providing a continuous local gradient.

Implementing efficient multi-head self-attention

Self-attention handles token communication through parallel query, key, and value projections combined with an autoregressive mask. Reshaping and transposing tensors allows efficient batching of multiple attention heads in PyTorch.

Loading weights and running GPU-accelerated text generation

Porting OpenAI weights requires handling tensor transpositions and ignoring non-parameter buffers. Moving the model to CUDA and implementing token sampling with top-k filtering enables text generation that matches the original model's output.

Initializing a model from scratch with random weights

Creating a model from scratch uses PyTorch's default random constructors, resulting in gibberish output prior to training. An auto-detection utility selects the optimal hardware device between CUDA, Apple Silicon MPS, and CPU.

Preparing the Tiny Shakespeare dataset for training

The tiny Shakespeare text file is read and tokenized into a continuous integer sequence. A batch size and sequence length are chosen to reshape the 1D sequence into 2D tensors for transformer ingestion.

Calculating cross-entropy loss and verifying initial entropy

The model's forward pass is updated to compute cross-entropy loss against target labels by flattening tensors. The initial loss matches the theoretical uniform probability expectation over the vocabulary size.

Optimizing a single batch using AdamW

An AdamW optimizer is instantiated to update model parameters, requiring manual gradient zeroing via zero_grad() before each backward pass. Running optimization steps on a single batch successfully overfits the data, driving the loss down.

Building a lightweight data loader for iterative training

A simple data loader steps through the tokenized file in chunks of batch size multiplied by time length, fetching offset target labels. This allows the training loop to feed sequential batches instead of overfitting a single slice.

Implementing weight tying between embeddings and the LM head

Inspecting OpenAI's GPT-2 weights reveals that the token embedding layer and the top classifier layer share identical data pointers. Replicating this weight tying scheme reduces parameter count significantly and improves training efficiency.

Customizing weight initialization to match GPT-2 specifications

OpenAI's source code initializes linear layer weights from a normal distribution with a standard deviation of 0.02 and biases to zero. Applying these initialization rules via PyTorch's apply method mirrors the official GPT-2 setup.

Compensating for activation growth in residual streams

Repeated addition along residual paths causes activation variance to accumulate with depth. Scaling down the initialization weights of residual layers by one over the square root of the number of residual layers controls this growth.

Deep learning training tolerates lower precision formats

While PyTorch defaults to float 32, deep learning workloads do not require such high precision compared to scientific computing. Lower precisions like tf32 and bf16 unlock significantly higher teraflops from hardware tensor cores. Using reduced precision also decreases memory footprint, allowing larger batch sizes.

Tensor cores accelerate computations using 4x4 matrix multiplies

Tensor cores execute specialized 4x4 matrix multiplication instructions directly in hardware. Most Transformer workloads are dominated by linear layer matrix multiplications that map cleanly onto these hardware instructions. Leveraging tensor cores is essential for achieving maximum hardware throughput.

Measuring throughput with tokens per second and synchronization

Timing training iterations requires explicit GPU synchronization using cuda synchronize because CPUs merely queue up asynchronous work. Tracking tokens per second provides an objective and reliable metric for optimization progress. Batch sizes should use powers of two and maximize available GPU memory.

Bfloat16 preserves numeric range without gradient scalers

Bfloat16 truncates the mantissa while keeping the identical sign and exponent range as float 32. This avoids the reduced range issues of float 16 and eliminates the need for complex gradient scalers. PyTorch autocast handles mixed precision automatically by running specific matrix operations in reduced precision.

Torch compile eliminates Python overhead and fuses kernels

Torch compile analyzes the entire neural network graph to remove Python interpreter overhead during the forward pass. It optimizes execution by combining multiple elementwise operations into single fused kernels. This single line of code yields substantial speedups across typical transformer training loops.

GPU memory hierarchy limits off-chip bandwidth

GPUs feature limited lightning-fast on-chip memory alongside tens of gigabytes of slower high bandwidth memory off-chip. Moving data back and forth between global high bandwidth memory and processor chips creates a major performance bottleneck. Understanding this memory hierarchy explains the dramatic speedups achieved through kernel fusion.

Flash attention avoids materializing attention matrices

Flash attention fuses operations and uses an online softmax trick to compute attention incrementally. This eliminates the need to write the massive attention matrix to high-bandwidth memory, speeding up execution.

Padding vocabulary sizes accelerates Cuda execution

Cuda kernels operate efficiently on powers of two, while prime or odd numbers trigger inefficient boundary logic. Padding the vocabulary size to a nice number avoids this overhead and accelerates training.

Adopting GPT-3 hyperparameters and gradient clipping

Switching to GPT-3 hyperparameters includes setting specific Adam betas and adding global gradient norm clipping. Gradient clipping prevents outlier batches from destabilizing the model with massive gradient updates.

Implementing cosine decay learning rate with warmup

Training benefits from a cosine decay learning rate schedule featuring a linear warmup phase. This gradually ramps up the learning rate initially and then decays it smoothly toward a minimum value.

Skipping batch size warmup and sampling without replacement

Gradual batch size increases can be skipped to keep arithmetic simple during early training. Data loader sequences should be sampled without replacement until an epoch boundary is reached.

Applying weight decay selectively with fused AdamW

Weight decay should be applied only to 2D weight matrices and embeddings while excluding 1D biases and layer norms. Using a fused AdamW implementation combines update kernels to reduce dispatch overhead.

Gradient accumulation simulates large training batch sizes

Hardware constraints prevent loading massive target batch sizes directly into GPU memory. Gradient accumulation solves this by summing gradients across multiple micro-batch steps before updating weights.

Compensating for loss reduction in gradient accumulation

Standard loss functions compute the mean over batch items, but accumulating unscaled gradients acts as a sum and distorts the optimization trajectory. Dividing the loss by the number of accumulation steps restores the correct mathematical average so gradients match a monolithic batch.

Distributing workloads across multiple GPUs with DDP

Training is accelerated across multiple GPUs by launching parallel python processes via PyTorch's torchrun command. Each process runs the identical script on different data shards while a master process handles logging and checkpointing.

Sharding data loaders for multi-process training

Multi-gpu setups require the data loader to stride across the dataset so that each process consumes unique sequences in parallel. The position tracker advances by the total token count consumed across all active processes simultaneously.

Wrapping models in the DDP container

The model must be wrapped in a DistributedDataParallel container passing the local GPU rank as the device ID. This container coordinates gradient averaging across all ranks during the backward pass.

Optimizing DDP communication during gradient accumulation

Synchronizing gradients across all GPUs after every micro-step introduces severe performance overhead. Disabling gradient synchronization during intermediate micro-steps ensures all-reduce communication only occurs on the final accumulation step.

Curating high-quality web datasets for pre-training

Small datasets like tiny Shakespeare are quickly exhausted, necessitating large-scale text corpora like the FineWeb-Edu dataset. Subsampling filtered educational web content provides a robust data source that closely mirrors the training diets of modern LLMs.

Pre-tokenizing and sharding large datasets to disk

Raw text datasets are pre-processed, tokenized using the GPT-2 vocabulary with document boundary markers, and cast to 16-bit integers to save space. Saving the data into discrete numpy shards of 100 million tokens each simplifies disk I/O and iteration during training.

Configuring token schedules for serious pre-training

The training loop is updated to iterate through the FineWeb-Edu shards using a batch size scaled to match GPT-3 optimization schedules. With all infrastructure in place, the multi-GPU setup achieves high throughput suitable for a full-scale reproduction run.

Incorporating validation splits and periodic text sampling

A validation data loader is added to measure generalization and check for overfitting during long training runs. Generation code is updated with a separate random number generator to sample text periodically without disrupting training randomness.

Implementing HellaSwag evaluation for early signal

HellaSwag is introduced as a smooth benchmark offering early signals of model capability through sentence completion tasks. Because small models cannot process traditional multiple choice formats, the benchmark is evaluated by comparing average token loss across choices.

Logging training metrics and evaluating accuracy periodically

A logging script records training loss, validation loss, and HellaSwag accuracy into a simple text file every few hundred iterations. Multi-GPU workers collaborate on the benchmark evaluation by synchronizing their stats across distributed ranks.

Surpassing GPT-2 validation benchmarks on fewer tokens

After training on 10 billion tokens of fine web edu data, the model surpasses the validation loss and HellaSwag accuracy of the original GPT-2 124M checkpoint. Higher data quality and cleaner filtering likely contribute to this increased sample efficiency compared to older web text.

Extending training to multiple epochs and saving checkpoints

Running the training loop for multiple epochs further improves coherence and HellaSwag scores, nearing GPT-3 levels of performance. Periodic checkpointing saves both model weights and optimizer states to allow resuming training or performing thorough downstream evaluations.

Transitioning from pre-training to supervised fine-tuning

Pre-training only teaches the model to dream documents and predict next tokens rather than follow conversational instructions. Supervised fine-tuning requires swapping out the pre-training dataset for a conversational format with user and assistant turns.

Accelerating training with a pure C and CUDA implementation

A parallel project called llm.c provides a pure C and CUDA implementation of GPT-2 training that acts as a faster alternative to PyTorch. Side-by-side execution confirms that the low-level implementation yields identical loss curves while executing faster per step.

0:00 / 4:01:26
k play/pause · f fullscreen · m mute
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said
  • Positional embeddings8× · first 7:47

    Learned absolute position vectors added to token embeddings to give the model awareness of sequence order.

  • GeLU6× · first 21:05

    A smooth neural network activation function used instead of ReLU in transformers.

  • Residual stream8× · first 18:24

    A clean pathway allowing gradients and information to flow unchanged across transformer blocks.

  • Top-k sampling1× · first 38:36

    A sampling strategy that restricts token selection to the top most probable candidates.

  • Weight tying4× · first 1:08:02

    Sharing a single weight matrix between the input token embeddings and the output language modeling head.

  • AdamW6× · first 57:04

    An optimization algorithm that improves upon standard Adam by properly decoupling weight decay.

  • TensorFloat-3211× · first 1:25:27

    A lower precision format that accelerates matrix multiplications by internally truncating mantissa bits.

  • Bfloat166× · first 1:25:36

    A 16-bit numeric format preserving the full exponent range of float32 without requiring gradient scalers.

  • Torch.compile18× · first 45:21

    A PyTorch compiler that optimizes model execution by eliminating Python overhead and fusing kernels.

  • Kernel fusion5× · first 1:55:09

    An optimization technique that combines multiple operations to keep data on the GPU chip and reduce memory round trips.

  • FlashAttention9× · first 2:00:26

    An algorithm that fuses attention operations and avoids materializing the large attention matrix in memory.

  • Gradient accumulation13× · first 2:36:41

    A technique that accumulates gradients across multiple micro-batches before performing an optimizer step to simulate large batch sizes.

  • Distributed Data Parallel21× · first 2:47:18

    A PyTorch module that implements data parallelism by spawning multiple processes across GPUs and synchronizing their gradients.

  • FineWeb edu12× · first 3:12:38

    A filtered educational subset of common crawl web data used for pre-training language models.

  • HellaSwag16× · first 3:28:35

    A sentence completion benchmark used to evaluate language model reasoning and world knowledge.

  • FineWeb12× · first 3:12:38

    An educational web text dataset used for pre-training language models.

  • PyTorch64× · first 4:12

    An open-source machine learning framework used for building and training neural networks.