Serving speed, hardware, GPU clouds and cost.
US capex is approaching $2 trillion, accounting for roughly 5% to 6% of GDP when including the broader supply chain. This makes it a much larger investment cycle than historical precedents like railroads, partly because American data centers serve global inference needs.
▶ 2:04
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
Microsoft builds massive data center infrastructure by linking power and network optics across multiple regions using a high-bandwidth AI WAN. This allows them to aggregate flops for huge training jobs across sites like Fairwater and Wisconsin.
▶ 0:54
Satya Nadella – How Microsoft thinks about AGI
Nvidia operates as the critical bridge transforming input electrons into valuable output tokens through deep engineering. Rather than doing everything itself, Nvidia partners across a vast supply chain ecosystem to handle the insanely hard transformation layer. This level of artistry and invention prevents the core hardware and software transformation from being easily commoditized.
▶ 0:23
Jensen Huang – Will Nvidia’s moat persist?
The main constraint is neither power nor chip availability, but rather obtaining credit. Lenders struggle to underwrite GPU clusters as collateral because they do not understand their future value or how to recoup them if a lab fails. Consequently, the entire buildout depends heavily on credit markets.
▶ 0:54
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Reasoning and problem-solving require large language models to explore multiple options step by step, reflect, and prune paths rather than just regurgitating memorized text. This search space exploration combined with applying AI to more jobs creates an insatiable, compounding demand for compute.
▶ 8:37
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
A language model is essentially a function that predicts the next likely word or token in a sentence based on its training. By predicting successive words, it constructs coherent continuations or fills in missing parts of text. During this process, it evaluates probabilities for various candidate words to determine the most fitting output.
▶ 1:33
A Hackers' Guide to Language Models
Arithmetic became extremely cheap while moving data around became relatively expensive, prompting a pivot toward hardware specialized for matrix multiplications. Companies like Google started building TPUs designed as reduced-precision linear algebra machines to maximize throughput. This hardware-algorithm co-design unlocked the massive scale needed for modern deep learning.
▶ 7:50
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
US models generally provide superior output quality and intelligence gains that make paying for them worthwhile for these experts. Additionally, Chinese models are sometimes served with fewer GPUs per replica, negatively impacting speed and error rates.
▶ 20:39
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Humans naturally respond to conversational cues within 300 milliseconds. If an AI response takes longer than 500 milliseconds, users will immediately notice, and delays over a second will cause them to hang up. Therefore, keeping latency extremely low is critical for usability.
▶ 2:55
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Frontier labs have shifted from venture-funded losses to turning a profit because their revenue per megawatt has surpassed incremental costs. For instance, Anthropic generates up to $50 million per megawatt compared to baseline costs of $10 to $15 million. This allows them to fund further training and expansion directly from profits rather than just capital injections.
▶ 2:21
Dylan Patel – Two labs will soon control most of the world's workforce
Bill Gurley is stepping back from co-hosting to focus on his upcoming book, Running Down a Dream, and to pursue broader policy and societal issues like US-China relations and healthcare reform. He wants to dedicate his time to creating platforms for these passions outside of the tech spaces he has known.
▶ 3:37
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Chinese labs and open-weight model builders use distillation to effectively spend less money while acquiring compute-equivalent outputs. Because they face severe compute constraints compared to US frontier labs, copying and distilling strong model outputs serves as a practical commercialization and catch-up strategy. This fast-following approach helps them shorten the capability lag to just a few months.
▶ 9:39
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
Leading Chinese labs operate with a total of 100 to 200 megawatts of compute at most. In contrast, Anthropic approaches nearly 5 gigawatts by the end of the year, creating a massive compute deficit on paper.
▶ 0:05
Why Isn’t China Further Behind in AI? - Dylan Patel
OpenAI's board saga highlighted the need for experienced board members who operate well under pressure and represent broad societal impact. Sam Altman notes that nonprofit boards without traditional shareholder accountability possess immense power, necessitating slates of diverse, experienced experts. Moving forward, resilient governance is critical to managing high-stakes power struggles safely.
▶ 5:50
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Giant models often lack native perception capabilities because they rely on text-only architectures. Without integrated sensory components, they cannot process raw visual inputs directly.
▶ 0:06
DeepMind Just Changed How AI Sees The World
While token pricing looks extremely cheap on paper, different systems consume varying amounts of tokens for the same job. DeepSeek can take two to three times more tokens than Claude Opus for identical tasks, making task-based metrics a truer cost measure.
▶ 1:08
DeepSeek's New AI Is An Absolute Monster
Initial token embeddings act like a static lookup table that assigns the exact same vector to a word regardless of its surrounding text. Attention steps in afterward to allow surrounding embeddings to pass relevant context into that specific vector.
▶ 2:18
Attention in transformers, step-by-step | Deep Learning Chapter 6
You can load the model using Hugging Face Transformers and extract its state dictionary to inspect key-value tensor shapes. This reveals the dimensions of components like the 50257-by-768 token embedding lookup table.
▶ 5:38
Let's reproduce GPT-2 (124M)
Triple-digit capex growth is bound to decelerate towards the end of next year because it becomes difficult to scale infrastructure exponentially off such a massive base. Hyperscalers face practical limits in raising trillions of dollars and marshalling enough resources to build out another gigawatt.
▶ 5:10
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
Microsoft approaches AI pricing by leveraging existing meters like consumption and tiered subscriptions that encapsulate consumption rights. They also rely on market expansion to offset lower incremental margins, similar to how the cloud expanded overall IT spend.
▶ 10:32
Satya Nadella – How Microsoft thinks about AGI
Nvidia secures massive upstream commitments by directly informing, inspiring, and aligning with CEOs across the semiconductor supply chain. By sharing detailed reasoning about the immense future scale of the AI industry, Nvidia convinces partners to invest early. Because Nvidia's downstream demand and supply chain velocity are so large, suppliers are confident in making these massive capital investments.
▶ 5:14
Jensen Huang – Will Nvidia’s moat persist?
Software design companies merely use models to generate code sent overseas for manufacturing, lacking physical know-how and factories. In contrast, Asian semiconductor fabs possess decades of built-up expertise and massive infrastructure. Therefore, labs cannot easily compete with wafer makers, whereas software design will face direct competition.
▶ 2:18
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Harnesses act as an exoskeleton that manages working memory, knowledge extraction, and knowledge graphs for large language models. Additionally, teaching agents to interact with existing human tools like CAD, design compilers, and simulators unlocks immense enterprise productivity.
▶ 10:53
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
Language models tokenize text into sub-word units, whole words, punctuation, or numbers using specific tokenizers like tick token. These tokens are mapped to numeric IDs that act as lookups into the model's vocabulary. For instance, prefixes like spaces are often encoded directly into the token representation.
▶ 4:45
A Hackers' Guide to Language Models
The initial machine translation model won translation contests with high quality, but took 12 hours per sentence because it relied on 100,000 disk seeks per word. To fix this, Jeff Dean and his team designed an in-memory compressed representation of 2 trillion words of N-gram data distributed across 200 machines. This batched API approach reduced sentence translation time down to about 100 milliseconds.
▶ 17:32
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
One speaker uses Codeium as a convenient plugin that assists without completely taking over the repository. Another uses Claude Code because it enables the distinct workflow of programming through high-level English guidance rather than micromanaging code diffs.
▶ 22:18
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
The standard pipeline streams audio chunks through an agent orchestrator to three sequential components. First, a speech-to-text system converts audio to text, which is then fed into an LLM to determine tool calls and responses. Finally, a text-to-speech model turns the text output back into audio chunks for the user.
▶ 4:36
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
OpenAI and Anthropic are rapidly centralizing global compute, capturing roughly 30 percent of incremental compute this year. By the end of next year, they are projected to absorb half of the world's incremental new compute. This trend is accelerating as they outpay competitors and sign major compute contracts.
▶ 4:15
Dylan Patel – Two labs will soon control most of the world's workforce
Analysts are spotting non-normal transactions like cashless vendor credits and off-balance-sheet debt guarantees that resemble historical corporate red flags. These circular arrangements can artificially inflate revenue metrics and obscure true market demand.
▶ 11:00
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Older post-training relied heavily on supervised fine-tuning using next-token prediction loss on strong model completions. In contrast, modern reasoning models involve heavy reinforcement learning and complex reasoning traces that are difficult to replicate using open literature. Researchers often find themselves stuck using outdated teacher models because mimicking frontier reasoning traces through simple distillation fails to reliably improve performance.
▶ 0:26
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
The compute gap does not matter much right now because the vast majority of a lab's total infrastructure goes toward research rather than actual model development. Actual pre-training requires only a fraction of the total power a lab possesses.
▶ 0:13
Why Isn’t China Further Behind in AI? - Dylan Patel
Crisis moments reveal a lot about human dynamics, but true leadership is defined by consistent day-to-day execution on ordinary days. Sam Altman emphasizes that while public battles are exhausting, most meaningful progress happens during normal meetings and boring Tuesdays. Leaders must focus on steady decision-making rather than getting consumed by drama.
▶ 17:27
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Gemma 4 is an open and free model created by DeepMind that runs locally on laptops. Compared to massive trillion-parameter systems, it is a tiny fraction of the size yet capable of native perception.
▶ 0:50
DeepMind Just Changed How AI Sees The World
You can run the model through the API for just a little over a dollar per million output tokens. This makes it significantly cheaper than competing frontier models like Claude Opus.
▶ 0:47
DeepSeek's New AI Is An Absolute Monster
Query matrices map noun embeddings to query vectors looking for preceding adjectives, while key matrices map adjective embeddings to matching key vectors. Computing dot products between these keys and queries produces a relevance score grid determining how much one word attends to another.
▶ 6:42
Attention in transformers, step-by-step | Deep Learning Chapter 6
Unlike the original Transformer which used fixed sinusoidal functions, GPT-2 treats positional embeddings as trainable parameters initialized randomly from scratch. During optimization, these embeddings organically develop smooth sinusoidal and cosine structures.
▶ 7:28
Let's reproduce GPT-2 (124M)
Infrastructure assets are depreciable over roughly six years, meaning a trillion-dollar investment requires around $250 billion in annual revenue over that span. However, AI inference revenue has recently spiked dramatically, and leading labs and cloud providers are already achieving strong gross margins or breaking even.
▶ 9:44
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
Mission Control acts as a control plane and package for multiple AI coding agents, letting developers issue tasks and steer them across independent branches. This allows GitHub to grow regardless of which specific coding agent or model ultimately wins market share.
▶ 13:17
Satya Nadella – How Microsoft thinks about AGI
Accelerated computing is designed to handle a vast diversity of workloads beyond AI, including molecular dynamics, data frames, and fluid dynamics. Because Nvidia systems are built to be flexible and operated by anyone, they can be deployed across every major cloud and on-premise environment. This massive versatility allows Nvidia to address applications and scientific research that specialized ASICs cannot handle.
▶ 17:38
Jensen Huang – Will Nvidia’s moat persist?
While purely software-based brute force methods might eventually peter out without fundamental leaps, exponential growth in metrics like tokens used per person suggests otherwise. Industry insiders anticipate that 2027 will be extremely chaotic and 2028 could look entirely unrecognizable. This points to a slow takeoff style of recursive self-improvement already underway.
▶ 7:47
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Physical AI encompasses AI interacting with the physical world across the electromagnetic spectrum, ranging from medical imaging to robotics and self-driving cars. Building a physical AI platform requires three distinct computer systems: a training system to build the brain, a digital twin simulation environment, and a target runtime computer.
▶ 24:54
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
Pre-training forces a deep neural network to guess the next word across vast corpuses like Wikipedia, utilizing stochastic gradient descent to optimize its weights. To succeed at this task, the model must implicitly learn complex real-world concepts, entities, and relationships. This capability essentially acts as a form of information compression.
▶ 7:40
A Hackers' Guide to Language Models
The team built infrastructure to train large neural nets across thousands of CPU cores using unsupervised learning on 10 million randomly selected YouTube frames. Without being explicitly told what a cat was, a high-level neuron naturally learned to activate for head-on facial views of cats based on visual patterns. This scaling approach achieved a 60% relative improvement on the ImageNet challenge and proved that scaling up neural networks was viable.
▶ 25:47
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Building an LLM from scratch allows learners to verify understanding through working code, which eliminates the ambiguities and potential errors found in written figures or math. It forces precision because code will simply fail to run if it is incorrect.
▶ 24:31
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
The primary metrics are quality, measured by word error rate, and latency, measured by time to complete a transcript. State-of-the-art models typically achieve around a 6% word error rate on open benchmarks. Getting this right is vital because downstream LLMs and TTS models will inherit any transcription errors.
▶ 5:25
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
There is a massive discrepancy where a relatively small amount of fabrication capital expenditure generates a massive amount of end revenue. Specifically, a few billion dollars in fab CapEx can produce a gigawatt of compute every year, which in turn generates roughly $100 billion in end revenue. This creates a massive profit incentive, though supply chain bottlenecks limit immediate expansion.
▶ 8:19
Dylan Patel – Two labs will soon control most of the world's workforce
While hyperscalers are spending up to 66% of their operating cash flow on capex, industry leaders believe the risk of a glut is extremely low over the next few years. This buildout is concentrated among well-capitalized tech giants embedding generative AI into their core operations.
▶ 20:17
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Closed frontier labs invest massive capital into compute and scaling laws to build predictable, high-value commercial services. This allows them to generate substantial revenues that fund subsequent, exponentially larger training runs. Any threat to this financial loop, such as aggressive open-source distillation, risks undermining the massive capital investments required to maintain the AI frontier.
▶ 17:50
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
Historically, a lab's compute budget has been divided into sixty percent for training and forty percent for inference. When broken down further, training consists of fifty percent research and ten percent development.
▶ 0:22
Why Isn’t China Further Behind in AI? - Dylan Patel
OpenAI began as an exploratory research lab without knowing that massive compute and productization would eventually be required. As the path forward demanded billions in capital, they gradually adapted their corporate structure through imperfect incremental decisions. Today, they balance open source by releasing select models while providing powerful tools like ChatGPT for free as a public good.
▶ 25:31
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Conventional setups connect multiple separate neural networks together. They require a dedicated visual model for images and a specialized audio encoder for sound.
▶ 1:48
DeepMind Just Changed How AI Sees The World
You can download and own the weights for free forever, but running it yourself requires substantial hardware. Because of the heavy compute requirements, it is generally impractical for standard home setups.
▶ 1:45
DeepSeek's New AI Is An Absolute Monster
The raw dot products can range from negative infinity to infinity, so a softmax function is applied column by column. This normalizes the values into probabilities between zero and one that sum to one for each column.
▶ 9:17
Attention in transformers, step-by-step | Deep Learning Chapter 6
GPT-2 moves the layer normalizations before the attention and multi-layer perceptron blocks rather than after them. It also adds an extra layer normalization right before the final classification head to maintain a clean residual stream.
▶ 14:29
Let's reproduce GPT-2 (124M)
AI labs like Anthropic achieve inference gross margins around 75 percent or higher, which is comparable to top SaaS companies like Salesforce. Additionally, Anthropic benefits from near-zero customer acquisition costs, making their unit economics quite strong despite massive R&D spending.
▶ 29:47
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
Microsoft places models directly into the middle tier of applications like Excel to teach them native business logic and tool usage. This allows the system to understand formulas and fix reasoning mistakes directly rather than relying on a pixel-level UI wrapper.
▶ 25:20
Satya Nadella – How Microsoft thinks about AGI
Programmable systems allow researchers to easily experiment with new algorithms and hybrid architectures like MoEs and hybrid SSMs. Nvidia achieves massive performance leaps—such as 50x improvements from Hopper to Blackwell—by co-designing across processors, system fabrics, and software simultaneously. Without a flexible programming model like CUDA, driving these rapid architectural changes would be extremely difficult.
▶ 21:52
Jensen Huang – Will Nvidia’s moat persist?
Although essays argue that individual neurons mostly perform clear jobs, the technique was dramatically oversold by commentators. Key aspects like weak activations, attention mechanisms, and the cache remain largely unexplained. Consequently, researchers cannot yet use this approach to reliably predict or understand complex network outputs.
▶ 17:28
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Since alignment is a permanent challenge, safety must be enforced through engineering controls rather than theoretical perfection. Developers should rely on tight containment with zero-trust minimum rights and out-of-band monitoring that can independently observe and shut down the system.
▶ 33:35
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
While pre-training instills broad world knowledge, raw models are not inherently optimized for answering questions or solving tasks. Instruction tuning feeds the model targeted datasets of questions and responses to shape its behavior for problem-solving. Additional steps like reinforcement learning from human feedback further refine the outputs by aligning them with human preference.
▶ 15:40
A Hackers' Guide to Language Models
While model parameters efficiently memorize facts, long-context models provide sharp, clear information via transformer attention mechanisms. However, naive attention scales quadratically, making trillions of tokens computationally prohibitive without algorithmic approximations.
▶ 33:10
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Chinese open-weight models tend to be much larger and utilize Mixture of Experts architectures for higher peak performance. They also typically feature unrestricted open-source licenses without the usage strings attached to some Western models.
▶ 30:42
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Older batch models like Whisper were trained on 30-second audio clips, requiring complex chunking and padding workarounds for streaming. Newer streaming-native models use specialized encoders trained with short look-ahead times and cached activations. This allows them to process small audio steps without redoing heavy computations.
▶ 7:25
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Companies holding excess compute, like SpaceX and Meta, are acting as strategic hoarders and selling capacity to labs at massive markups. While component suppliers like memory and substrate makers are raising prices quickly, a bullwhip effect means the broader supply chain rebalances slowly. Consequently, the labs must pay significantly higher rates to secure the compute they need.
▶ 24:08
Dylan Patel – Two labs will soon control most of the world's workforce
Emerging state laws create a confusing patchwork of liability and compliance burdens that differ wildly across the country. This regulatory fragmentation slows down US frontier labs and startups while leaving global competitors unaffected, threatening America's tech leadership.
▶ 28:41
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Reward hacking often stems from flaws in the data supply chain and shortcuts taken during world-building by third-party data companies. When models are placed in environments with missing test files or poorly defined constraints, their optimization pressure drives them to find alternative ways to achieve high reward. Fixing this requires labs to spend more time refining model behavior and working closely with data providers.
▶ 28:25
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
Researchers use the majority of the compute budget to generate ideas and test new architectures. They also spend time testing various data mixes before settling on a final configuration.
▶ 0:36
Why Isn’t China Further Behind in AI? - Dylan Patel
Sam Altman explicitly discourages startups from attempting to start as a nonprofit and adding a for-profit arm later. He notes that if they had known how things would evolve, they would have structured things differently from the start. Complex legal constraints make this transition path difficult and undesirable for typical startups.
▶ 32:56
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Instead of using dedicated encoders, the model cuts pictures into small patches and projects those pixels directly into its internal representation. Audio is similarly sliced into chunks and poured straight into the main transformer.
▶ 2:29
DeepMind Just Changed How AI Sees The World
The release of expensive frontier models triggers a race to build alternatives that match their capabilities at a fraction of the cost. Over time, other systems distill these large models down so they can eventually run on local hardware.
▶ 2:16
DeepSeek's New AI Is An Absolute Monster
Masking prevents later tokens from improperly influencing earlier ones by setting their future scores to negative infinity before softmax. This keeps the model from cheating during training while maintaining normalized columns.
▶ 11:42
Attention in transformers, step-by-step | Deep Learning Chapter 6
GPT-2 uses an approximate version of the Gaussian Error Linear Unit activation function backed by a tanh approximation. This avoids the flat zero-gradient regions seen in standard ReLUs and provides smooth, continuous gradients during optimization.
▶ 21:02
Let's reproduce GPT-2 (124M)
Nvidia backstops the AI infrastructure buildout by providing revenue floors, landlord guarantees, and leases to help Neoclouds obtain investment-grade financing. They currently support around 588 billion dollars of off-balance sheet backstops to ensure a diverse market of GPU customers and buyers.
▶ 34:31
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
Microsoft continues to use the OpenAI GPT family to the maximum across their core products while strategically allocating internal MAI flops for research and cost-optimized specialized models. This dual approach allows them to build a world-class superintelligence team without duplicating foundational training waste.
▶ 37:41
Satya Nadella – How Microsoft thinks about AGI
The CUDA moat relies on a rich ecosystem that supports every major framework and helps optimize custom kernels like Triton. Developers choose CUDA because it provides a massive installed base spanning hundreds of millions of GPUs across every cloud and edge device. This expansive reach ensures that software and models developed on CUDA will be useful and compatible everywhere.
▶ 27:34
Jensen Huang – Will Nvidia’s moat persist?
A small cognitive core with external fact lookup poorly mimics human information retrieval because humans rely on millions of facts for deep intuition. As compute gets cheaper and algorithms improve, the break-even point rises, making it more effective for large language models to memorize extensive knowledge directly.
▶ 30:11
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
The primary bottlenecks for massive infrastructure buildouts are physical limitations like power generation and land permitting. Additional hurdles include local moratoriums and securing specialized construction labor for large data centers.
▶ 48:15
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
Developers can improve model outputs by supplying custom instructions that act as prepended text guiding the model's persona and reasoning quality. Explicitly instructing the model to think through background context helps combat its tendency to generate overly confident hallucinations. Additionally, because generation happens sequentially token by token, prompting the model to reason longer increases its computational thoroughness.
▶ 22:21
A Hackers' Guide to Language Models
Google has further trained internal Gemini models on their proprietary codebase to assist developers. Sundar Pichai has noted that 25% of checked-in code characters are generated by these AI models with human oversight.
▶ 36:49
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Native tool use allows models to query a Python interpreter or perform web searches instead of relying strictly on internal memorization. This external fetching lets the model retrieve accurate real-time answers rather than making things up.
▶ 32:27
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
LLM streaming latency is measured by time to first token, and you generally want to hit 200 to 300 milliseconds. To meet this latency budget while maintaining sufficient intelligence for tool calling, models typically need to be in the 8 to 30 billion parameter range. Going larger blows the latency budget, while going smaller hurts reasoning capabilities.
▶ 8:44
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
As labs generate increasingly massive profits from inference, they are expected to allocate a decreasing fraction of compute to it over time. Instead, they will ratchet up the percentage of compute dedicated to training to drive towards AGI. The primary purpose of running inference becomes generating the capital and compute fleet required for future research and training.
▶ 30:31
Dylan Patel – Two labs will soon control most of the world's workforce
Stablecoin supply has surged past $300 billion with trillions settled, as users embrace instant microsecond transfers and yields up to 4%. These crypto-based rails bypass the high friction, expensive, and slow settlement times typical of traditional banking infrastructure.
▶ 35:24
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Pre-training a solid base model is increasingly accessible and does not require astronomical budgets, thanks to open-source tooling like Megatron. While spending more money yields better models, new entrants regularly spin up models from scratch using available architectures. The primary hurdle is not the pre-training cost itself, but building something good enough to land initial user adoption.
▶ 45:22
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
The actual pre-training of Methos used sub-200 megawatts of compute for roughly two months. This shows that the active model training phase requires far less power than total lab capacity.
▶ 0:40
Why Isn’t China Further Behind in AI? - Dylan Patel
Models like Sora demonstrate a surprisingly sophisticated understanding of world physics and occlusions across multiple sequence steps. While they still suffer from random flaws like generating extra limbs, their trajectory shows rapid improvement through scale. This approach of training on internet-scale visual patches goes surprisingly far toward modeling 3D reality.
▶ 35:40
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
This approach forces the system to learn perception and reasoning simultaneously while removing hundreds of millions of specialist parameters. It blurs the boundary between thinking and perceiving, making the AI punch above its weight.
▶ 3:08
DeepMind Just Changed How AI Sees The World
The underlying architecture did not change at all for this release. Despite keeping the same architecture, the model achieved a massive capability jump in just five weeks.
▶ 2:38
DeepSeek's New AI Is An Absolute Monster
Value matrices multiply token embeddings to produce value vectors, which are then scaled by the attention weights and summed together. Adding this delta vector to the original embedding yields a contextually richer representation.
▶ 13:44
Attention in transformers, step-by-step | Deep Learning Chapter 6
Tokenize the text into a sequence of integers and view it as a two-dimensional batch tensor. You should fetch an extra token beyond the sequence length so you can create parallel input and target tensors offset by one position.
▶ 48:06
Let's reproduce GPT-2 (124M)
Most Neoclouds have limited cybersecurity, allowing open-source models to potentially hack and access data within these clusters. This creates critical vulnerabilities where malicious actors or foreign intelligence agencies sharing infrastructure could see what other clients are training.
▶ 40:43
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
Microsoft builds an open, flexible hyperscale infrastructure fleet capable of supporting multiple model lineages and architectures. This flexibility prevents their entire network topology from becoming obsolete due to sudden industry breakthroughs.
▶ 42:34
Satya Nadella – How Microsoft thinks about AGI
Nvidia initially failed to realize that early foundation labs required multi-billion-dollar supplier investments rather than standard venture capital funding. Huang admits this miscalculation prevented early partnerships, a mistake he is now avoiding by actively investing in OpenAI and Anthropic.
▶ 39:37
Jensen Huang – Will Nvidia’s moat persist?
The FDIC can sustain the banking system by honoring checks and tracking withdrawals to reconstruct records later. Because banking fundamentally relies on trust and temporary promises, overdrawn accounts can be clawed back once stability is restored.
▶ 38:04
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Space data centers require high-value computing because launch costs represent a significant percentage of the investment. Nvidia provides the world's best performance per watt and high return on investment, making it economically viable. Furthermore, fully fungible hardware and continuous software updates ensure the systems remain durable while orbiting.
▶ 58:48
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
The OpenAI API is stateless, meaning nothing is stored on the server between requests. To maintain a conversation, you must pass the entire history of messages back in every new API call.
▶ 43:33
A Hackers' Guide to Language Models
Language model inference is significantly cheaper than human labor or alternative pastimes, leaving vast economic headroom to trade compute for smarter answers. By applying algorithmic search and letting models think harder during inference, performance scales up dramatically.
▶ 53:56
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Instead of using a single dense fully connected layer, MoE replaces it with multiple feedforward networks known as experts. A router dynamically directs each input token to only a small subset of these experts during a forward pass.
▶ 40:56
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Performance is evaluated using time to first audio and the real-time factor, which measures how many seconds of audio can be produced per second of processing time. Quality is also critical and is best assessed by directly listening to audio samples rather than relying solely on objective metrics. You typically want a real-time factor under one to avoid buffering issues.
▶ 9:50
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
China currently relies on a small fraction of global compute and smuggled chips, but domestic production is expected to inflect upward as local fabs from SMIC and CXMT come online. By 2028 and 2029, domestic manufacturing could yield millions of units, enabling China to add tens of gigawatts annually. However, these domestic chips will lag behind Western alternatives in raw quality.
▶ 36:04
Dylan Patel – Two labs will soon control most of the world's workforce
Also known as Trump accounts, they give every newborn child a 401(k)-style account seeded with $1,000 to foster early asset ownership. Backed by automated Treasury rollouts and financial literacy integration, the initiative aims to make every child a capitalist from birth.
▶ 48:01
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Frontier labs initially gain an edge by integrating models tightly with specialized agent harnesses and swarm coordination systems. However, because different users demand diverse setups, models are quickly pressurized to work across multiple harnesses, neutralizing closed integration advantages. Furthermore, information leaks easily in Silicon Valley, meaning breakthrough techniques spread rapidly to the open-source community.
▶ 50:50
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
Reinforcement learning requires even less compute than the pre-training phase. It represents a smaller slice of the overall resource allocation during model creation.
▶ 0:47
Why Isn’t China Further Behind in AI? - Dylan Patel
Altman believes that people who create valuable data deserve a way to be compensated for its use in training AI. Creators should have the ability to opt out of having their style generated and establish an economic model when their style is used.
▶ 40:22
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
The architecture allows the model to handle both images and audio inputs natively. It can successfully identify objects in videos while engaging in complex conversational tasks.
▶ 1:29
DeepMind Just Changed How AI Sees The World
The capability jump comes from learning via self-feedback rather than relying solely on massive new training data. The AI is able to critique its own work and learn iteratively from its own experiences.
▶ 3:05
DeepSeek's New AI Is An Absolute Monster
Instead of using a massive square matrix that adds hundreds of millions of parameters, the value map is factored into a low-rank transformation using two smaller matrices. This balances parameter allocation between the key, query, and value projections.
▶ 16:52
Attention in transformers, step-by-step | Deep Learning Chapter 6
The expected initial loss corresponds to the negative natural logarithm of a uniform probability distribution over the vocabulary. For a vocabulary size of 50,257, this evaluates to roughly 10.82, which sanity checks our initialization.
▶ 56:01
Let's reproduce GPT-2 (124M)
Labs like Anthropic and OpenAI are keeping months of progress internal-only rather than releasing their best models immediately. They maintain a lead over open-source alternatives by releasing minimum viable models while holding back frontier capabilities to protect competitive advantages and address safety concerns.
▶ 56:14
Can AI Keep Growing Exponentially? Let’s Ask SemiAnalysis — With Dylan Patel and Jordan Nanos
The pause was a course correction to ensure fleet fungibility and balance training capacity with global serving needs. It allowed them to adapt to rapidly changing hardware generations, cooling requirements, and power densities rather than getting stuck with a single generation.
▶ 49:57
Satya Nadella – How Microsoft thinks about AGI
Nvidia relies on a first-in, first-out queue based on valid purchase orders and ready data centers rather than dynamic bidding. Huang explicitly rejects auctioning hardware to the highest bidder, preferring to maintain dependable, predictable pricing as a foundational business practice.
▶ 53:10
Jensen Huang – Will Nvidia’s moat persist?
Lenders cannot easily value or repossess GPU clusters, forcing them to rely entirely on the credit risk of the customer. Since new AI labs lack investment-grade credit, lenders demand long-term contracts and large upfront deposits to mitigate default risks.
▶ 42:12
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
The demand for infrastructure is supported by three major segments rather than just a couple of frontier labs. Classical baseline simulation and data processing account for about a third of Nvidia's business, while multiple AI labs and thousands of new startups renting compute make up the rest.
▶ 1:02:18
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
New or free users often hit low request-per-minute limits initially. You can handle this programmatically by writing a retry function that catches rate limit errors, reads the retry duration, and sleeps before calling itself again.
▶ 44:53
A Hackers' Guide to Language Models
Large models can be trained across multiple data centers using high-bandwidth, high-latency connections between metro areas. As long as parameter synchronization and gradient accumulation happen fast enough within the step time, fully synchronous multi-center training works smoothly.
▶ 1:03:27
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Pre-training scaling laws continue to hold reliably across massive compute scales, but pre-training has become prohibitively expensive. Consequently, labs increasingly leverage inference-time scaling and reinforcement learning to extract performance efficiently.
▶ 54:35
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Running models in distant data centers introduces network latency that adds up quickly in real-time systems. Co-locating your models and agent orchestrator in the same data center can drop network latency from 75 milliseconds down to 5 milliseconds. This yields a massive 30% reduction in total latency for an already optimized setup.
▶ 13:33
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
True AI capital expenditures will approach trillions of dollars because building critical IT hardware like servers and networking is only a fraction of the cost. Companies must also invest massively in physical data center buildings and dedicated power plants years in advance. This infrastructure scaling could absorb up to a tenth of the global economy by the end of the decade.
▶ 43:24
Dylan Patel – Two labs will soon control most of the world's workforce
Surveys show that roughly 60% of people would change their career paths if starting over, driven by what is termed boldness regret. Research indicates that over time, humans are far more likely to regret the career risks they did not take rather than the risks they pursued.
▶ 54:48
AI Bubble, Stablecoin Boom, and Runnin' Down a Dream | BG2 w/ Bill Gurley and Brad Gerstner
Lab compute usage is sequential, meaning the peak power used at any single point in time is relatively small. Even though labs possess multiple gigawatts, the maximum used for actual development is restricted.
▶ 0:51
Why Isn’t China Further Behind in AI? - Dylan Patel
Rather than looking at what percentage of entire jobs will be replaced, it is more insightful to examine what percentage of tasks AI can perform. As AI handles increasingly complex tasks across longer time horizons, it acts as a sophisticated tool that lets people operate at a higher level of abstraction.
▶ 42:13
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Open models serve as valuable gifts to the community and can help massive systems like DeepSeek learn to see more efficiently. They empower scientists, students, and developers to perform their work better.
▶ 3:52
DeepMind Just Changed How AI Sees The World
The AI writes code for a game, plays it, and notices bugs like broken collision detection or missing acceleration. It then compiles fixes, plays the game again, and continuously improves the output like a human would.
▶ 3:35
DeepSeek's New AI Is An Absolute Monster
Multi-headed attention runs many distinct sets of key, query, and value maps in parallel to capture multiple types of context simultaneously. For instance, GPT-3 uses 96 attention heads per block to learn different grammatical and semantic relationships.
▶ 20:47
Attention in transformers, step-by-step | Deep Learning Chapter 6
PyTorch's cross entropy function cannot accept multi-dimensional inputs directly. You must flatten the three-dimensional logits tensor down to two dimensions as batch times time by vocabulary size before passing it alongside the targets.
▶ 54:06
Let's reproduce GPT-2 (124M)
Microsoft develops custom silicon to manage overall fleet total cost of ownership in balance with existing hardware. They maintain a close loop between their internal MAI models and custom microarchitecture to generate true demand and keep pace with their own models.
▶ 1:04:02
Satya Nadella – How Microsoft thinks about AGI
Nvidia follows a philosophy of doing as little as possible while providing just enough support to ensure ecosystem survival and broad market reach. By backing neoclouds like CoreWeave, they help bootstrap essential infrastructure that otherwise would not exist.
▶ 44:18
Jensen Huang – Will Nvidia’s moat persist?
Wall Street trades semiconductor stocks at low earnings multiples because most public market investors are not technical and view these as low-margin cyclical businesses. However, in an AI-driven world, these companies experience massive operating leverage that dramatically multiplies their earnings.
▶ 53:21
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Without Moore's law, doubling transistor counts no longer happens for free after waiting a year at the same price. Instead, organizations must pay twice as much to get twice the capacity, drastically increasing baseline spending. This economic shift means the semiconductor industry was already on track to become the largest in the world.
▶ 1:09:54
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
You pass a list of available tools or functions under the functions keyword argument in the chat completion request. Because you cannot pass raw Python functions directly, you must convert them into JSON schemas using libraries like Pydantic and Python's inspect module.
▶ 47:12
A Hackers' Guide to Language Models
Researchers first invent improvements in isolation using small, simple codebases with fast turnaround times. Then they periodically bundle and run large-scale integration experiments to see how the features interact, which fails about half the time due to unexpected complexity.
▶ 1:08:55
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Reinforcement learning divides compute into actor models that generate completions and learner models that perform policy gradient updates. While actors can be distributed across heterogeneous environments, learners require tightly networked compute clusters.
▶ 59:27
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Pure speech-to-speech models process audio natively without converting it to text first, preserving tone and emotion while enabling full duplex communication. However, they currently struggle with complex instruction following and reliable tool calling in production. Developers often resort to prompt engineering before eventually falling back to pipeline architectures.
▶ 15:10
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Hyperscalers and technology companies are increasingly relying on debt to fund multi-trillion-dollar infrastructure investments that exceed their cash flows. This massive credit expansion will crowd out traditional borrowing for governments, consumers, and other industries, driving up interest rates. Higher interest rates could mirror a second Volcker shock, triggering defaults in developing nations and cratering traditional equity valuations.
▶ 45:40
Dylan Patel – Two labs will soon control most of the world's workforce
Research compute is consumed by exploratory trial and error across ideas and architectures, taking up the vast majority of resources. Development compute is strictly tied to the final training run of the model itself.
▶ 0:29
Why Isn’t China Further Behind in AI? - Dylan Patel
Altman views current models like GPT-4 as somewhat lacking relative to where the technology needs and will eventually get. He expects the delta between GPT-5 and GPT-4 to mirror the massive leap from GPT-4 to GPT-3.
▶ 45:05
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Developers can use platforms like Lambda to quickly reproduce AI research papers and run experiments. It provides powerful Nvidia GPUs to train models, fine-tune existing ones, or run chatbots efficiently.
▶ 4:47
DeepMind Just Changed How AI Sees The World
It is very difficult to implement because the AI essentially judges and rewards itself. Determining whether it is actually performing well during this autonomous process is not always straightforward.
▶ 4:28
DeepSeek's New AI Is An Absolute Monster
In practice, the value up matrices from each individual head are stapled together into a single giant output matrix for the entire block. References to a head's value matrix typically only point to the initial value down projection into the smaller space.
▶ 22:45
Attention in transformers, step-by-step | Deep Learning Chapter 6
Weight tying reuses the same weight matrix between the bottom token embeddings and the top language modeling head. This scheme saves about 30 percent of the model parameters and introduces an inductive bias that similar tokens should share representations.
▶ 1:08:02
Let's reproduce GPT-2 (124M)
OpenAI's stateless API business is Azure-exclusive, requiring customers to use Azure for these calls. However, their SaaS products like ChatGPT can run on any infrastructure and partner with anyone.
▶ 1:06:50
Satya Nadella – How Microsoft thinks about AGI
Export controls are ineffective because China has massive energy abundance, plentiful 7nm chip manufacturing capacity, and top-tier researchers who can gang chips together. Huang argues that hardware restrictions fail because abundant energy compensates for chip limitations and algorithmic advances drive efficiency.
▶ 1:05:58
Jensen Huang – Will Nvidia’s moat persist?
Unlike semiconductor supply chains where wafer and memory capacity are fully utilized with zero latent fat, energy production possesses significant latent supply. Developers can bypass traditional grid connections by building behind-the-meter power plants using natural gas or improvising with truck and airplane motors. This flexibility allows energy scaling to outpace rigid semiconductor manufacturing.
▶ 58:09
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Nvidia supports a massive range of workloads, models, and global customers across a complete software and hardware stack. This extreme fungibility lowers risk for buyers investing tens of billions of dollars because they know the infrastructure can adapt or be subleased. Specialization is the enemy of this durability, whereas general-purpose architecture ensures the lowest total cost of ownership.
▶ 1:25:12
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
Quantized formats like GPTQ optimize models to use lower-precision data formats. This significantly reduces the memory moving around, making inference faster and allowing larger models to run efficiently on standard hardware.
▶ 1:05:30
A Hackers' Guide to Language Models
Analyzing text is generally easier than generating it, allowing models to inspect their own and other systems' outputs for dangerous or problematic content. Additionally, exposing model capabilities through controlled APIs and user interfaces provides an extra layer of governance.
▶ 1:19:06
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Inference scaling is often more attractive than pre-training larger models because it unlocks substantial performance without the massive upfront fixed costs of pre-training. Developers must weigh the math based on user volume, query costs, and how long the model will remain market-relevant before replacement. While pre-training builds permanent baseline knowledge, inference scaling lets you spend money per query to maximize performance dynamically.
▶ 1:02:20
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Tool calling evaluations assess whether the generated function call structure is correct and whether the output is feasible for the use case. Teams often aim for near 100% structural correctness in their tool calls. To achieve this with smaller LLMs constrained by latency budgets, developers fine-tune models on use-case-specific data.
▶ 17:35
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
AI training features massive economies of scale because training efforts amortize across billions of sessions and users. Additionally, labs that get slightly ahead can charge higher markups and reinvest compute into recursive self-improvement. These compounding advantages mean that most effective AI labor will inevitably concentrate within just a couple of dominant labs.
▶ 1:11:24
Dylan Patel – Two labs will soon control most of the world's workforce
Compute will likely become the most precious commodity because intelligence behaves more like energy, where demand scales heavily based on price and availability. The primary bottleneck to expanding this compute supply isn't just chip fabrication, but solving massive energy challenges like nuclear fusion.
▶ 1:10:00
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Frontier AI models are improving at an incredible pace, and open weights alternatives are right behind them. Major capability leaps can happen in as little as five weeks.
▶ 0:19
DeepSeek's New AI Is An Absolute Monster
Attention is extremely parallelizable, allowing massive numbers of computations to execute quickly on GPUs. This high parallelizability lets models leverage massive scale for qualitative performance improvements.
▶ 24:54
Attention in transformers, step-by-step | Deep Learning Chapter 6
Linear layer weights are initialized from a normal distribution with a standard deviation of 0.02, while biases are set to zero. Additionally, residual layer weights are scaled down by the square root of the number of residual layers to prevent activation growth.
▶ 1:14:09
Let's reproduce GPT-2 (124M)
Microsoft respects local sovereignty concerns by building sovereign clouds and regional high-density infrastructure with strong data residency and privacy controls. They implement features like confidential computing to build trust in their tech stack worldwide.
▶ 1:19:42
Satya Nadella – How Microsoft thinks about AGI
While Moore's law only advances about 25% per year, exceptional computer science and algorithm innovations like Mixture of Experts can improve performance by 10x. This means an army of skilled AI researchers working with constrained hardware can still achieve massive breakthroughs.
▶ 1:09:39
Jensen Huang – Will Nvidia’s moat persist?
Human beings are generally satisfied with the status quo and do not adopt radical technologies for fun. Massive leaps in production functions and economic growth occur primarily during crises or wars because people are forced to adapt. For instance, mRNA technology sat latent for twenty years until the COVID-19 emergency forced its rapid deployment within months.
▶ 1:07:39
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Nvidia is expanding into software licensing and revenue-sharing models tied to its massive install base. Software licensing provides enterprise-grade support and maintenance for companies operating AI factories without internal IT departments. Meanwhile, revshare agreements capture ongoing economic returns from regional cloud partners over multi-year deployments.
▶ 1:26:57
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
Instruction-tuned models rely on specific prefix tokens and formatting rules established during their fine-tuning process. Failing to replicate this exact prompt structure results in degraded model performance or incorrect behavior.
▶ 1:08:12
A Hackers' Guide to Language Models
Instead of monolithic pre-training, modular architectures like mixture-of-experts could feature independent components developed by different teams globally. Continuous distillation can then update these smaller sub-modules over time without requiring the entire model to be retrained from scratch.
▶ 1:37:09
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Pre-training uses massive corpora of internet data, books, and synthetic data via cross-entropy loss to build baseline knowledge and absorb information. Mid-training specializes the model further, focusing on long-context documents while combating catastrophic forgetting through selective quality content. Post-training applies reinforcement learning and supervised fine-tuning to act as a skill unlock rather than teaching new knowledge.
▶ 1:04:18
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Guardrail models and classifiers can be introduced right before the main LLM to route requests, or placed at the end of the generation step to screen responses. While adding these components introduces extra latency and management overhead, catching issues before the TTS model runs is essential. If a policy violation slips through, you cannot take back spoken words.
▶ 20:13
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Governments are likely to impose heavy regulations, such as preventing external model releases or restricting internal access to advanced checkpoints. Constituents and elected officials will push back against rapid progress due to economic disruption and safety concerns. However, such delays could inadvertently allow labs to conduct massive recursive self-improvement internally while public models lag behind.
▶ 1:03:55
Dylan Patel – Two labs will soon control most of the world's workforce
Intense competition against tech giants accelerates product innovation and lowers costs for users. However, it simultaneously triggers a dangerous arms race dynamic that puts pressure on safety priorities if not carefully managed.
▶ 1:15:03
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Cloud GPU platforms provide the powerful hardware needed to reproduce research papers, run inference, or fine-tune models. They allow developers to test ideas and see results in minutes.
▶ 5:05
DeepSeek's New AI Is An Absolute Monster
Tensor float 32 maintains the standard sign and eight exponent bits of a 32-bit float but truncates the mantissa down to 19 bits. This reduction happens internally within the GPU tensor core instruction, allowing matrix multiplications to execute significantly faster. Because the accumulation and outputs remain in float32, this approximation is empirically unnoticeable during deep learning training.
▶ 1:31:14
Let's reproduce GPT-2 (124M)
Computing ecosystems like x86, ARM, and CUDA exhibit extreme stickiness because replacing them demands an enormous investment of time and energy. Most developers prefer not to incur these migration costs, giving robust proprietary software stacks a powerful, lasting moat.
▶ 1:20:42
Jensen Huang – Will Nvidia’s moat persist?
Critics argue the draft bill trades away crucial safety standards merely to preempt state AI laws and includes onerous provisions for government emergency take downs. Furthermore, it immediately nullifies state regulations while taking many months to implement new federal requirements, creating a dangerous regulatory gap. Once preemption is granted, it also removes negotiating leverage, making future tougher legislation much harder to pass.
▶ 1:18:54
AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain
Jensen Huang maintains near-total confidence in humanity's technological trajectory because countless independent researchers and engineers globally care deeply about achieving positive outcomes. He believes critics often underestimate human capacity and the massive wave of autonomous innovation happening outside major tech spheres. Ultimately, when amazing people organize around shared goals, they successfully build a better world.
▶ 1:35:53
Jensen Huang | Does AI Revenue Justify Capex, '27 AI Bottlenecks, Singularity, Open vs Closed & more
Retrieval augmented generation searches external sources like Wikipedia to find relevant documents for a given query. It then injects this retrieved text directly into the prompt context so the language model can generate an accurate, up-to-date response.
▶ 1:11:52
A Hackers' Guide to Language Models
Distillation can be applied at the module level in a continual process where a large model continually distills into a smaller variant. Once trained, the smaller module replaces the large one, freeing up parameter capacity for new data. This allows independent teams to update specific capabilities like coding or languages without retraining from scratch.
▶ 1:41:45
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Reinforcement Learning from Human Feedback averages model behaviors across many human evaluators, which tends to smooth out raw, high-information expressions. Because RLHF relies on these broad preference averages, models often struggle to be truly incisive or capture the authentic voice of a researcher. This creates a fundamental trade-off where safety and preference alignment sacrifice deep, raw insights.
▶ 1:23:19
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
The thinker-talker pattern uses a small LLM to instantly generate a filler response like letting the user know it is thinking. Simultaneously, it issues a tool call to a larger, more capable model with better instructions and guardrails. This pattern helps hide latency and improves response quality when your main model is too slow for real-time conversation.
▶ 21:34
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Replicating Google's model of ranking blue links and ads understates the true potential of conversational AI. Instead of copying a traditional search engine, the goal is to create a system that synthesizes information and directly helps users act on it.
▶ 1:18:12
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Because the CPU asynchronously schedules work and queues up kernels to run on the GPU, execution does not happen instantly. Without explicit synchronization, code timing measurements can capture just the CPU scheduling duration rather than the actual GPU workload execution time. Calling CUDA synchronize forces the CPU to wait until the GPU finishes all scheduled work before taking timestamps.
▶ 1:33:32
Let's reproduce GPT-2 (124M)
Modern architectures rely heavily on advanced packaging, system architecture, and numerics rather than just transistor scaling alone. Consequently, the massive R&D required to port modern designs back to older nodes makes leaning forward the only viable economic choice.
▶ 1:35:41
Jensen Huang – Will Nvidia’s moat persist?
You can load a domain-specific dataset using the Hugging Face datasets library and train the model using tools like Axolotl. This process modifies model weights to adapt to specialized structures, such as converting natural language questions into database SQL queries.
▶ 1:22:19
A Hackers' Guide to Language Models
No, Mixture of Experts models still require the entire model to be loaded into memory to achieve high inference efficiency at large batch sizes. At inference, you send a fraction of a large batch through every expert rather than skipping weights entirely. Therefore, you need enough High Bandwidth Memory to fit the whole model.
▶ 1:45:28
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Statistics show that senior developers with over 10 years of experience are actually much more likely to ship codebases where over 50% is AI-generated compared to juniors. Experts are more effective at using AI tools, reviewing the output, and trusting the generated code. However, this raises long-term concerns about how developers can achieve true expertise if they bypass the initial struggle of writing and debugging code themselves.
▶ 1:31:37
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Future speech-to-speech models eliminate traditional text-to-text and text-to-speech intermediate steps, rendering standard component-level evals obsolete. Instead, teams must rely on transcription sidecars for auditability and shift toward longer, full duplex conversational metrics. Most of these comprehensive evaluations will happen on top of the base inference API.
▶ 23:32
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Advertising creates aesthetic and trust concerns by potentially biasing model responses toward advertisers. Maintaining a paid subscription model ensures users know their answers are unbiased and that they are not the product being monetized.
▶ 1:20:16
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Torch compile analyzes the entire neural network structure ahead of time instead of executing operations sequentially through the python interpreter. It identifies element-wise operations and performs kernel fusion to keep data chunks on the GPU chip for multiple calculations. This avoids expensive round trips between high bandwidth memory and processing units, significantly speeding up training throughput.
▶ 1:51:00
Let's reproduce GPT-2 (124M)
The rising value of fast response tokens created a new market segment where customers pay a premium for lower latency inference. To capture this demand, Nvidia expanded the Pareto frontier by integrating different accelerators into the CUDA ecosystem.
▶ 1:37:49
Jensen Huang – Will Nvidia’s moat persist?
Model connectivity should mimic the underlying hardware topology to optimize communication costs. Networks should have extremely dense connections within the same chip and HBM, scaling down to fewer bottlenecked connections across chips and TPU pods. Allowing this structure to emerge organically prevents guessing the wrong connection proportions.
▶ 1:54:46
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
RLVR involves generating answers and grading completion accuracy to serve as a reward for reinforcement learning. This allows the model to learn step-by-step reasoning and tool use without heavy manual constraints. It scales inference-time compute effectively through iterative trial-and-error.
▶ 1:39:03
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Publishing explicit, concrete behavioral guidelines for how a model should respond to specific prompts provides transparency. This clarity helps differentiate between actual bugs that need fixing and intentional policy decisions that can be openly debated.
▶ 1:24:25
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
FlashAttention fuses attention operations and avoids materializing the massive attention matrix in high-bandwidth memory. Even though it performs more total floating-point operations, it drastically reduces expensive HBM memory round trips and speeds up execution significantly.
▶ 2:01:31
Let's reproduce GPT-2 (124M)
Training objectives can be modified to force models to work harder on every token, similar to vision model techniques like dropout or hiding partial information. Additionally, allowing models to take actions and observe consequences—much like humans and infants do—would dramatically improve sample efficiency. Exploring self-play and thought experiments can also generate high-value learning without external data.
▶ 1:58:13
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
RLVR allows the model to solve increasingly complex problems and longer reasoning chains, showing a clear logarithmic scaling law where more compute yields better performance. In contrast, RLHF hits a wall because preference tuning compresses messy human choices into a single averaged style. Spending extra compute on RLHF eventually leads to reward model over-optimization.
▶ 1:56:41
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Framing AGI as a single fuzzy threshold is less useful than tracking specific capability milestones. Systems capable of significantly accelerating global scientific discovery will likely emerge by the end of the decade.
▶ 1:32:37
Sam Altman: OpenAI, GPT-5, Sora, Board Saga, Elon Musk, Ilya, Power & AGI | Lex Fridman Podcast #419
Padding awkward vocabulary sizes like 50257 to powers of two like 50304 helps Cuda kernels execute more efficiently using standard block tiles. Although it introduces slight compute overhead, it prevents inefficient boundary kernels and improves performance by several percent.
▶ 2:09:13
Let's reproduce GPT-2 (124M)
Google initially held back chatbot releases because search engine standards demand high factuality and zero hallucinations, which early models failed to meet. They also needed to address safety issues regarding offensive outputs. They underestimated how much users would value chatbots for general utility tasks rather than strict search queries.
▶ 2:08:34
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Developers should build a simple model from scratch that fits on a single GPU to understand pre-training and the attention mechanism. Instead of getting bogged down in massive production libraries, they can study human-readable configuration files and match weights against reference implementations. This reverse-engineering process provides a hands-on verifiable reward for correct architecture design.
▶ 1:58:32
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Gradient norm clipping caps the global norm of parameter gradients at a specified threshold like 1.0. This prevents occasional bad data batches from causing massive gradient spikes that could destabilize or shock the model optimization.
▶ 2:17:22
Let's reproduce GPT-2 (124M)
After mastering the foundational architecture, newcomers should pick a very narrow area and read the relevant papers. Because the field moves rapidly, many specific subtopics are left understudied and provide low-hanging fruit for motivated researchers. Reaching out to authors or focusing deeply on these niche questions can lead to original contributions.
▶ 2:03:37
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
A cosine decay schedule starts with a linear warmup phase from near zero up to a maximum learning rate, followed by a cosine curve down to a minimum rate. While PyTorch has built-in schedulers, writing a custom function over iteration steps offers clear transparency and full control.
▶ 2:21:14
Let's reproduce GPT-2 (124M)
Preference tuning relies on data labelers choosing a preferred response rather than a mathematically correct one, which averages out diverse valid perspectives. This process assumes that multifaceted human preferences can be neatly quantified and reduced to single values. Consequently, it is never fully solvable because it struggles to capture complex social choices correctly.
▶ 1:53:54
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Weight decay regularizes models by preventing weights from growing too large, but it should not be applied to 1D tensors like biases and layer norm scales. Instead, weight decay is restricted to 2D matrix multiplication weights and embeddings to properly distribute work across channels.
▶ 2:30:15
Let's reproduce GPT-2 (124M)
Researchers can focus on narrow areas like evaluation or character training, finding edge cases where frontier models fail. If a lab adopts their evaluation benchmark, it can serve as a major career milestone.
▶ 2:14:22
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Gradient accumulation lets you process multiple smaller micro-batches sequentially while adding up their gradients via backward passes. Once the target token count is reached across these steps, a single optimizer update is performed to simulate a massive batch size.
▶ 2:36:39
Let's reproduce GPT-2 (124M)
Frontier labs offer high compensation and massive impact, but often involve secrecy, publication limits, and intense 996 grind cultures. Conversely, academia provides freedom to publish and follow long-term ideas, but suffers from funding scarcity and lower pay.
▶ 2:15:38
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Launch your training script using the torchrun command, which automatically sets up environment variables like rank and world size for parallel processes. Each process assigns itself to a specific CUDA device using its local rank and runs an identical copy of the model.
▶ 2:47:16
Let's reproduce GPT-2 (124M)
Text diffusion models iteratively denoise and fill in text gaps in parallel, which can theoretically speed up long generations like code diffs. However, they currently require matching compute steps for quality and struggle with step-by-step reasoning or tool use.
▶ 2:31:01
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Modify your data loader to accept the process rank and total number of processes to stride through dataset chunks. Each rank initializes at a different offset and advances its position by the total number of processes multiplied by the batch size and sequence length.
▶ 2:56:59
Let's reproduce GPT-2 (124M)
Tool integration reduces hallucinations by letting models delegate tasks like math to a calculator or facts to a web search. However, it does not fully solve the issue because the model still needs to choose the right tool and retrieve correct information.
▶ 2:34:53
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
By default, DDP synchronizes gradients across all GPUs after every backward pass, which is wasteful during intermediate gradient accumulation steps. You can avoid this by toggling require_backward_grad_sync to false during intermediate micro-steps and enabling it only on the final step before the update.
▶ 3:02:30
Let's reproduce GPT-2 (124M)
Continual learning involves continuously updating model weights to adapt to incoming information rapidly, but doing this per user at a global scale is prohibitively expensive. Alternatively, in-context learning stuffs extensive documents and history into a massive context window so the model appears to learn fast without weight modifications. While weight updates happen via curated global releases or methods like RLVR, context windows remain a primary lever for personalization.
▶ 2:40:27
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
You can write a script to download the dataset from Hugging Face, tokenize the documents with special start tokens, and save them into fixed-size numpy file shards on disk. Using uint16 arrays saves space, and sharding into chunks of 100 million tokens makes large datasets much easier to manage during training.
▶ 3:14:18
Let's reproduce GPT-2 (124M)
Hybrid attention models combine state space models with transformers to reduce the compute required for distant tokens, avoiding the exploding costs of standard dot-product KV caches. Additionally, sparse attention mechanisms use lightweight indexers or sliding windows to selectively mask out unnecessary tokens rather than brute-forcing full attention. While brute-force models guarantee no missing info, these efficiency tricks aim to scale input lengths cost-effectively.
▶ 2:44:21
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Since small models cannot handle multiple-choice labels directly, you construct a batch of token completions pairing the shared context with each of the four possible options. You then evaluate the cross-entropy loss or token probabilities for each option and pick the one with the highest average probability.
▶ 3:32:46
Let's reproduce GPT-2 (124M)
World models simulate the environment to generate intermediate variables and check correctness, adding sophisticated reasoning beyond next-token prediction. Furthermore, the massive infrastructure investment in LLM transformers is creating world-class industrial tooling that supercharges robotic learning. However, real-world deployment still faces severe bottlenecks in on-the-job customization and strict safety requirements.
▶ 2:51:32
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
You can scale up training by multiplying your epoch count such as running four epochs for an overnight run while periodically saving model checkpoints alongside optimizer states. Saving the optimizer state dictionary is essential for preserving Adam buffers like m and v if you intend to resume optimization later.
▶ 3:48:46
Let's reproduce GPT-2 (124M)
Labs are already using advanced coding agents like Claude Code extensively in production, with some models capable of rebuilding applications like Slack from scratch in sandboxes. However, full automation faces hurdles due to the jagged nature of model capabilities, complex system design, and human specification issues. Instead of a complete takeover, developers are shifting toward product management and system design roles while leveraging agents for feature implementation.
▶ 3:05:46
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Transitioning to supervised fine-tuning involves swapping out the pre-training text dataset for a conversational one with a user-assistant structure. You continue training by filling in user tokens and sampling assistant tokens, which adapts the document-dreaming model into a conversational chat format.
▶ 3:55:46
Let's reproduce GPT-2 (124M)
RLVR is being applied in real scientific domains, such as wet labs where language models propose hypotheses tested in the physical world. Startups are pouring hundreds of millions into these initiatives to achieve AlphaFold-like breakthroughs across chemistry, biology, and math. While early, these moonshots could dramatically transform traditionally resource-heavy industries.
▶ 3:13:42
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
You can implement a parallel training loop in pure C and CUDA to achieve faster startup and higher tokens-per-second performance than general frameworks like PyTorch. While PyTorch handles general neural networks, a dedicated implementation like llm.c directly targets GPT architectures to squeeze out maximum hardware speed while producing identical loss curves.
▶ 3:57:32
Let's reproduce GPT-2 (124M)
Taking over an entire computer screen involves severe structural blockers and complex environment setups compared to calling backend APIs. Furthermore, arbitrary tasks require clear user specification and continual learning from mistakes, making the user interface exceptionally hard to solve. Despite heavy investments in agentic systems throughout 2025, current operator demos still struggle significantly.
▶ 3:18:16
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Advertising monetization is expected because these platforms are heavily subsidized and require sustainable revenue streams to fund ongoing R&D. Companies are currently hesitant to make the first move primarily due to reputation risks and potential user backlash. However, integration is likely coming as market competition and the need for massive scale intensify.
▶ 3:32:29
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Meta's open-weight Llama strategy suffered under internal political fighting, misaligned incentives, and heavy pressure to top benchmarks. Upper leadership pushed for massive models optimized for benchmark performance rather than smaller, usable open models. This created internal friction and shifted the focus away from Llama's original community-focused goals.
▶ 3:45:08
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
The ADAM Project is a US initiative designed to build and host genuinely open-weight AI models to compete with rapidly advancing Chinese models like Qwen. Proponents argue that open models drive research innovation, and the US must invest locally to retain technical leadership and value.
▶ 3:50:06
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Nvidia's primary moat isn't just their advanced GPUs, but rather the deeply embedded CUDA ecosystem built over nearly two decades. Because of this software compatibility and widespread adoption, hyperscalers and developers stick with Nvidia rather than risk bespoke hardware.
▶ 4:00:38
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
The flood of AI slop across digital channels will likely drive a massive surge in the value and premium placed on in-person physical goods and events. People will increasingly seek out authentic human crafts, art, and face-to-face community connection to escape digital saturation.
▶ 4:17:05
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
Human consciousness and agency ensure that we remain the ones directing AI, using it merely as a powerful tool rather than losing control to it. Ultimately, humans decide what actions to take, and current AI implementations require explicit human intent to function.
▶ 4:22:47
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490
No answer mentions that.