Tests written by, for, and around AI-generated code.
Standard test suites only check components in isolation and rely on situations developers have actively anticipated. Production failures are usually caused by environmental factors, weird timing, or complex interactions that engineers never thought to test for.
▶ 0:22
From Unit Tests to Whole Universe Tests (with Will Wilson)
While AI coding tools can increase the volume of pull requests, unchecked AI code generation often amplifies codebase entropy and technical debt. Studies show that without clean codebases, teams spend more time reworking and refactoring code, leading to little net productivity gain.
▶ 2:49
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Fast automated tests act as essential guardrails to catch regressions and prevent the AI agent from breaking or deleting code. Running in milliseconds, these test suites provide immediate feedback to keep the unpredictable genie in check.
▶ 49:59
TDD, AI agents and coding with Kent Beck
You transition from writing every line to directing the agent and carefully reviewing its outputs. As models have improved in reliability, some teams have even moved to policies where developers neither write nor read the code directly. However, trusting an agent requires finding ways to have it prove its work functions correctly.
▶ 3:39
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
As tokens are continuously added to a single context window, the attention relationships scale quadratically and strain the system. This pushes the model past its smart zone into the dumb zone around 100K tokens. Developers should keep tasks small to avoid this performance degradation.
▶ 3:40
Full Walkthrough: Workflow for AI Coding — Matt Pocock
Subjective tasks lack clear deterministic pass-fail criteria like string matching or symbolic verification. When we replace a deterministic grader with a non-deterministic LLM judge, we introduce massive noise and fluctuations. This makes it very difficult to draw reliable conclusions from our evaluation results.
▶ 1:22
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
You start with a small number of basic properties like ensuring the system never crashes or times out. As you go deeper, you can turn formal API guarantees and written SLAs into automated test properties.
▶ 8:08
From Unit Tests to Whole Universe Tests (with Will Wilson)
Red-green TDD is a development workflow where you start by writing a failing test for a new feature request. In the green phase, you focus purely on speed to get the test to pass, followed by a dedicated refactoring phase to ensure code quality. When combined with AI, agents rapidly generate the failing tests and passing code, allowing developers to focus their time on refactoring.
▶ 5:18
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
AI tools grant your wishes for features or code changes, but often in unexpected or perverse ways that do not match your true intent. This dynamic creates an addictive intermittent reinforcement loop similar to playing a slot machine.
▶ 6:44
TDD, AI agents and coding with Kent Beck
Starting sessions with red-green TDD instructions prevents agents from writing more code than necessary. Because automated tests are effectively free with AI, writing tests first dramatically increases the chances of getting working code. This approach eliminates the historical friction of TDD while keeping the agent focused on a minimal implementation.
▶ 5:56
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
Matt prefers clearing the context entirely to return the AI to a base state, comparing it to the character from Memento who continually forgets. Compacting squeezes previous conversation into a written history record, but resetting ensures the system prompt and starting state remain consistently fresh.
▶ 8:42
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You start by partnering with subject matter experts to create clear quality guidelines using binary or Likert scores. Then, you extensively calibrate this process by taking multiple human evaluation samples for the same data and updating guidelines based on disagreements. This iterative process acts as your North Star for measuring and improving task quality over time.
▶ 4:32
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Traditional tests rely on rigid sequences of steps that break when completely incidental changes occur. Property testing checks general rules and invariants, meaning incidental updates like level layouts or timing don't automatically break your tests.
▶ 12:45
From Unit Tests to Whole Universe Tests (with Will Wilson)
Traditional unit tests frequently overindex on code coverage and implementation details rather than verifying overall system behavior. This causes tests to break during refactoring even when the underlying functionality remains correct, and AI-generated unit tests can sometimes be self-affirming without validating real behavior.
▶ 6:44
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Mundane implementation details and syntax memorization become largely irrelevant because AI handles boilerplate and package management. Instead, architectural vision, setting milestones, and managing system complexity become vastly more leveraged skills.
▶ 11:35
TDD, AI agents and coding with Kent Beck
Passing an automated test suite does not guarantee that a server will successfully boot or function in practice. To catch these issues, instruct the agent to run the server in the background and exercise the API using curl commands. Tools like Showboat can even generate markdown documentation tracking these manual test results.
▶ 7:24
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
Instead of relying solely on a specs-to-code approach, you should use an interactive technique like the grill me skill. This skill has the AI interview you relentlessly about every aspect of the plan to reach a shared design concept. It walks down decision tree branches one by one to resolve dependencies.
▶ 12:27
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You can use a statistical concept called power analysis to estimate the required volume of data. This estimation looks at factors like your desired effect size, the observed noise in resampled completions, and your target significance level. Using these factors ensures you have enough data to confirm model performance differences are legitimate rather than noise.
▶ 6:19
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
The hypervisor implements a deterministic machine model underneath standard Linux so operations always take predictable clock cycles. When the system attempts a non-deterministic call like getting a random number, the hypervisor traps it and returns a deterministic answer.
▶ 22:30
From Unit Tests to Whole Universe Tests (with Will Wilson)
Playwright is an open-source testing framework built by Microsoft that automates end-to-end testing directly in the browser by simulating user interactions. It supports multiple programming languages like Python, TypeScript, and C#, and allows running tests in either headed or headless modes.
▶ 8:18
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
It emerged from a series of workshops to offer an economic and human-centric alternative to rigid, phased waterfall development. The goal was to reach the early majority with a standard framework emphasizing rapid feedback loops and continuous iteration.
▶ 19:29
TDD, AI agents and coding with Kent Beck
Conformance suites are ideal when an existing standard has detailed language-agnostic specifications and tests, such as Web Assembly or custom file upload requirements. You can give the test suite to a capable agent and instruct it to write code until all tests pass. This allows you to rapidly reverse engineer or implement complex standards across different projects.
▶ 8:16
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
Workflows can be split into human-in-the-loop tasks like strategic planning and requirement alignment that require active participation. In contrast, tasks like implementation can be treated as away-from-the-keyboard once the destination is mapped out. You must stay involved for crucial design choices but can let the model handle execution steps.
▶ 26:39
Full Walkthrough: Workflow for AI Coding — Matt Pocock
Evaluation datasets should focus heavily on diversity and realism to accurately reflect production tasks. They also need to be difficult and discriminative so they can effectively differentiate between various model capabilities. Achieving this accuracy usually requires a combination of model and human-based filtering over time.
▶ 7:18
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Determinism allows the hypervisor to restart the simulation from the beginning and follow the exact same path without saving heavy state snapshots. Once a rare intermediate event is reached, the system locks it in and only explores further paths from that point onward, drastically reducing exponential search times.
▶ 33:57
From Unit Tests to Whole Universe Tests (with Will Wilson)
Developers can connect their coding agents to Playwright using the Playwright MCP server, command-line tools, or specialized Playwright agents. Using Playwright agents installs specific markdown files that provide a planner, generator, and healer to manage testing workflows.
▶ 10:17
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Kent Beck invented TDD as a conceptual blend of vintage tape application workflows and his Smalltalk testing framework. Reading about typing expected output tapes before writing transformation programs inspired him to write expected values before code. This process transformation dramatically reduced his programming anxiety.
▶ 36:02
TDD, AI agents and coding with Kent Beck
For quick, single-page throwaway tools, code quality does not matter as long as it works. For long-term projects, accepting poor agent code is a choice, and you can explicitly prompt the agent to refactor messy sections. By taking the time to guide refactoring, you can end up with code quality that surpasses what you would write manually.
▶ 10:20
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
LLMs excel at summarization, and since you already share a design concept from the grilling session, reading the PRD only tests the LLM's summarization skills. Therefore, skipping this review saves time without risking alignment.
▶ 35:36
Full Walkthrough: Workflow for AI Coding — Matt Pocock
LLM judges rely on high-level quality guidelines to evaluate solutions similarly to how humans do across various tasks. Rubrics use a task-specific checklist of individual items that are separately verified by an LLM judge to make the process more objective. Both serve as automated proxies to scale evaluation when human review becomes a bottleneck.
▶ 8:39
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
The system monitors which lines of code or new functions run by tracking call stacks or developer-added callbacks. It also scans log messages for specific errors and watches control flow branches, such as unexpected loop or conditional branch outcomes, to guide further exploration.
▶ 37:32
From Unit Tests to Whole Universe Tests (with Will Wilson)
Work IQ is a Microsoft development skill that allows developers to connect the M365 suite directly to their terminal environment. This enables developers to pull project requirements and information from tools like Outlook or PowerPoint straight into command-line interfaces like Copilot CLI.
▶ 12:37
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Critics argue TDD ignores upfront architecture, but practitioners constantly bounce between abstraction levels. Writing a test forces a moment of design to determine the API interface before implementation details are locked in. Once a test turns green, developers get a release of tension to think about broader design and refactoring.
▶ 42:34
TDD, AI agents and coding with Kent Beck
Prompt injection targets systems built on LLMs by exploiting their gullibility and tendency to follow instructions embedded in untrusted data. Unlike SQL injection, you cannot easily parameterize queries to separate data from instructions. If a malicious input contains hidden commands, an unconstrained agent might mistakenly execute them.
▶ 13:13
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
Larger context windows often just provide more dumb zone rather than improved reasoning for coding tasks. A smaller context window around 100k tokens is considered the optimal smart zone for code.
▶ 37:53
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You perform meta-evaluation by measuring the agreement between your automated judge and golden data from trusted human evaluations. You can evaluate this using correlation metrics, binary accuracy, or by testing if judge outputs correlate causally with real production metrics like user abandonment. You should aim for a judge-human agreement level that matches the agreement seen between different humans.
▶ 9:58
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
The platform uses a special shim that intercepts every request for a random number from a controlled random source. By periodically changing a single register or decision value at a specific instruction count, it can force the system to branch down entirely different execution paths.
▶ 41:02
From Unit Tests to Whole Universe Tests (with Will Wilson)
While traditional unit testing is typically triggered by adding a new method to a class, modern agentic workflows focus testing on overall behavior and feature requests. When a new feature request arrives from a product management team, it serves as the direct trigger for writing behavioral tests.
▶ 13:07
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Tests act as immutable guardrails that prevent AI agents from making disruptive changes across a codebase. Because AI tools lack architectural taste and often introduce regressions, maintaining a fast suite of tests allows you to catch accidental breakages instantly. You can feed expected values as immutable assertions to keep the agent aligned.
▶ 51:31
TDD, AI agents and coding with Kent Beck
The lethal trifecta occurs when a model has access to private data, is exposed to malicious instructions, and possesses an external exfiltration vector. An example includes an assistant with email access forwarding sensitive data after reading a malicious message. The guaranteed solution is to cut off at least one leg of the trifecta, such as blocking external communication.
▶ 15:08
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
Vertical slices represent thin functionality chunks crossing all system layers instead of coding layer by layer. This approach provides rapid feedback because you can immediately test that all integrated layers work properly together.
▶ 43:44
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You can apply a kitchen sink of statistical methods to establish tight confidence intervals and report error bars alongside your results. Techniques like power analysis help determine benchmark size, while resampling model and judge rollouts helps reduce variance. This ensures that reported performance improvements reflect actual capability gains rather than random noise.
▶ 12:02
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Initial adoption is strongest among data infrastructure, distributed computational frameworks, and databases like Confluent and MongoDB. It is also widely used by blockchain and fintech companies where the financial cost of a production bug is catastrophic.
▶ 1:02:29
From Unit Tests to Whole Universe Tests (with Will Wilson)
Developers should capture screenshots of Playwright functionality tests and add them to pull requests to visually track changes. Additionally, running tests in headless mode and committing code before letting agents make changes ensures history and context are preserved.
▶ 16:28
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Early Facebook engineers operated in an environment with rapid feedback loops, including local dev servers, incremental rollouts, and rich observability. Programmers took ultimate personal responsibility for their code because they felt the pain of their own production mistakes. With high ownership and immediate feedback, writing traditional unit tests was unnecessary.
▶ 56:23
TDD, AI agents and coding with Kent Beck
Sandboxing limits damage if an agent receives malicious instructions or executes unsafe commands. Solutions like cloud-based web containers run agents in isolated virtual machines managed by third parties, protecting your local environment. Alternatively, using proper mocking prevents the dangerous practice of cloning production databases directly onto developer laptops.
▶ 16:20
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
You convert the PRD into a Kanban board of independent issues with blocking relationships that allow agents to process tasks sequentially or in parallel. Once this backlog is queued up, the human steps back and lets the AI handle the implementation phase AFK.
▶ 49:45
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You can track explicit metrics like thumbs-up and thumbs-down feedback or preference ratings commonly seen in chatbots. Alternatively, you can use implicit metrics like session length and abandonment rates to gauge performance. These production quality signals help you evaluate whether your deployed agent is functioning as intended.
▶ 13:43
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
For complex applications with heavy state management, using specialized Playwright agents is recommended because they include built-in instructions tailored to handle state effectively. Alternatively, developers can choose to directly test APIs if an API is available instead of relying entirely on browser automation.
▶ 18:08
Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft
Feature flags enable developers to deploy a single software artifact containing multiple modes that can be gradually ramped up. This provides a safe subdeployment mechanism where you can test features on a subset of users and quickly turn them down if error rates increase. It removes deployment anxiety by letting you observe behavior live before full release.
▶ 1:01:43
TDD, AI agents and coding with Kent Beck
Running multiple coding agents simultaneously across different projects requires firing on all cylinders and leads to severe mental fatigue. Rotating between projects while waiting for agents helps maintain throughput, but human cognitive limits act as a natural bottleneck. Embracing this exhaustion means recognizing that you cannot indefinitely scale yourself to manage a thousand projects at once.
▶ 23:38
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
TDD forces the agent to write a failing test first using red-green-refactor before writing the implementation, which prevents it from cheating by generating tests that match faulty logic. Because feedback loops set the quality ceiling for AI outputs, having strong automated tests ensures the agent does not code blind.
▶ 1:06:48
Full Walkthrough: Workflow for AI Coding — Matt Pocock
A/B testing allows you to measure production metrics and gain a relative sense of how your new agent performs compared to existing solutions. Instead of pushing the model to all users at once, you can vet its quality safely in a live environment. Power analysis can help determine the necessary duration and sample size for running these tests.
▶ 14:11
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Coding agents can write custom components and stitch together libraries on demand, which threatens traditional business models for paid component libraries. However, agents still heavily rely on the open source ecosystem for recommendations and underlying patterns. At the same time, maintainers face challenges with open source repositories being flooded by low-quality AI-generated contributions.
▶ 27:08
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit
Deep modules expose a small, simple interface while encapsulating a large amount of functionality internally, making it much easier to wrap them in test boundaries. In contrast, shallow modules create a complex web of tiny files and dependencies that confuse AI agents and make testing difficult.
▶ 1:17:07
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You can continuously monitor production to identify concerning performance trends and uncover novel failure cases or use cases. By feeding these real-world failure examples back into offline evaluations and human reviews, you ground your eval pipeline in reality. This creates a continuous improvement cycle where every part of the framework is regularly refined.
▶ 15:02
How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
You should use a pull strategy for implementers, allowing them to pull coding standards via skills when they need guidance. Conversely, you should use a push strategy for automated reviewers by pushing the coding standards directly to them alongside the written code.
▶ 1:28:18
Full Walkthrough: Workflow for AI Coding — Matt Pocock
You can use tools like Sandcastle to run agents in isolated Docker work trees and Git branches in parallel. A planner chooses issues from the backlog, separate sandbox implementers write the code, and a merger agent resolves any test or type conflicts before merging.
▶ 1:30:07
Full Walkthrough: Workflow for AI Coding — Matt Pocock
No answer mentions that.