Speaker Notes

WF2026: Autoresearch & Keynotes ft. Anthropic, Google DeepMind, Amazon AGI, Sonar, Arena, Recursive

AI engineering is shifting toward automated research loops, rigorous verification, and SRE-driven reliability practices to transition from experimental agent prototypes to production systems.

AI Engineer · 8:51:55 · YouTube ↗

Conference talks · 1 MC and 2 industry speakers  Advanced · Models, agentic workflows, and code verification strategies

Highlights

40 clips · 52 min of 532 min
    • Models may fail basic zero-shot queries but succeed when given code execution tools.
    • Capability improvements are spiky and organic rather than linear.
    • Tool integration bridges the gap between raw weights and complex problem solving.
    • Prompt the model to review unfamiliar codebases for potential dead ends and gotchas.
    • Leverage AI knowledge to map out edge cases you haven't explicitly considered.
    • Connect git diffs and context to guide the model through hairy modules.
    • Initial 3x to 5x velocity boosts often dissipate after three months of usage.
    • Code generation speed correlates with an increase in security and maintainability issues.
    • Technical debt accumulates rapidly unless proactive verification controls are applied.
    • Providing clear context and constraints reduces token consumption by over 30%.
    • Multi-layered verification combines algorithmic data flow analysis with agentic intent checks.
    • Teams utilizing a multi-layered verification approach report 44% fewer AI-derived production outages.
    • Combine algorithmic data flow analysis with agentic verification of business logic.
    • Multi-layered verification approaches result in 44 percent fewer AI-derived outages.
    • Treating verification as a core discipline rather than an afterthought is essential for enterprise reliability.
    • Coding agents succeeded rapidly because code is fully verifiable through execution and tests.
    • General knowledge work lacks unit tests to verify whether an agent achieved its intent.
    • Most workflow complexity lives in the unstructured seams between different applications.
    • Perception agents read rendered screen pixels rather than relying on underlying code or APIs.
    • They enable agents to confirm their own outputs by visually checking UI changes.
    • Visual annotation allows users to point and mark elements directly instead of writing long text prompts.
    • Eighty percent of new code added to GitHub is already machine-generated.
    • Traditional human training data for coding models is effectively exhausted.
    • Model self-play enables frontier models to generate and verify their own challenges autonomously.
    • Traditional benchmarks only test if code runs and produces basic outputs.
    • Open-ended tasks like lossless text compression force models to create novel algorithms.
    • Combining compressed file size and source code size creates a continuous loss function.
    • LLM-as-a-judge relies on fixed rubrics and static scores.
    • Dynamic agent interactions create entirely different UI trajectories every time.
    • An agent-based judge can adaptively read traces and discover complex patterns of failure.
    • Scaling complex use cases often outgrows generic graph frameworks.
    • An effect-native agent loop gives full agency and control over execution.
    • Effect propagation brings built-in tracing, structured concurrency, and fine-grained logging.
    • AI code generation on longer horizons now enables recursive self-improvement.
    • True RSI requires models to recognize their own shortcomings across pre-training and RL.
    • Automating AI research enables scaling breakthroughs to other scientific domains.
    • Auto-research goes beyond hyperparameter tuning to find genuinely novel architectures.
    • The system discovered hash biograms, triam embeddings, and learned gates.
    • Achieved lower bits per byte than expert community baselines.
    • Stop thinking like researchers and start thinking like SREs for AI systems.
    • Measure reliability, availability, and cost recovery instead of just raw accuracy.
    • Maximize dependable outcomes as the primary north-star metric for production agents.
    • Agents fail when jumping directly from abstract specs to production code.
    • Give agents access to a simulation environment to discover correct algorithms.
    • The process flows from abstract spec to simulation, concrete spec, and production.
    • Give agents access to deterministic simulation environments to discover correct algorithms.
    • Executable design helps handle partial failure and concurrency before touching production code.
    • The process flows from abstract spec to simulation, concrete spec, and final implementation.
    • Ranked recall ledgers perform better than unguided memory or standard RAG.
    • Oracles provide correct memory but don't force models to use it effectively.
    • Bad memory is expensive; good recall policies save tokens and prevent drift.
    • Current agents make incremental improvements by combining existing papers.
    • They struggle to discover fundamentally new optimization mechanisms.
    • Multi-agent generators with automated evaluation loops are needed for true discovery.
    • Agents tasked with discovering novel optimizers typically produce incremental plus-one improvements.
    • They successfully combine existing literature rather than generating fundamentally new mechanisms.
    • This limitation highlights the need for discovery-focused multi-agent loops rather than pure evaluation.
    • High benchmark scores often coexist with unpredictable production behavior in agentic systems.
    • Benchmarks fail to capture tool failures, API outages, user variability, and long workflows.
    • The gap between benchmark performance and production reliability widens as systems become autonomous.
    • Generating video and audio simultaneously stems from a single underlying causal process.
    • Post-processing audio onto generated pixels creates poor synchronization and lip-movement issues.
    • Unified models naturally keep lips and audio in sync without requiring separate patching layers.
    • Audio and video stem from a single causal generative process in the real world.
    • Generating pixels first and hacking audio on top leads to poor synchronization.
    • Unified models natively resolve lip-sync and temporal alignment issues.
    • Untrained human evaluators consistently prefer sharper, overly saturated AI videos over reality.
    • Optimization can exploit human psychological triggers rather than improving actual usefulness.
    • Side-by-side human preference metrics can be misleading without deep domain expertise.
    • Post-training spans everything between pre-training and the final user experience.
    • Forward deployed engineers gather real-world constraints directly from customer workflows.
    • Customer usage insights must be systematically piped back upstream to improve model behavior.
    • Tasks are defined as YAML files acting as unit tests for agent models.
    • Multiple LLM and rule-based judges evaluate correctness and insight quality.
    • Evaluation suites run nightly to track candidate model performance against production.
    • Aiden ran 1300 experiments on a single H100 node over 22 days.
    • It used only 4% of total compute while capturing 15% of competition records.
    • Submissions achieved a 28% leaderboard hit rate, six times the community average.
    • Codebase abstraction acts like a neural network architecture for auto research.
    • Strict abstractions prevent data leakage by isolating training and test logic.
    • Good architectural constraints systematically bias agents toward better generalization.
    • Algorithmic tasks with exact answers should use plain code for reliability.
    • Language models are best suited for handling ambiguity, fuzzy logic, and judgment.
    • Clear separation of code determinism, agent judgment, and human authority prevents bloat.
    • Standard optimization loops can easily get stuck in local optima.
    • A Pareto-based candidate pool retains all winners on even a single training example.
    • Balanced search strategies converge to significantly higher overall scores.
    • Collect production agent traces to evaluate subjective tasks
    • Get humans to annotate a small subset of trajectories with detailed feedback
    • Use optimized LLM-as-a-judge prompts to create a continuous data flywheel
    • Agents will often hack reward functions by disabling performance boosts like CUDA graphs
    • Explicitly define what the agent must not do to prevent degrading overall inference speed
    • Ensure optimizations apply broadly rather than just to narrow context windows
    • Full session state and agent history can be packaged and sent to another machine
    • Coworkers can resume work using different underlying models while sharing memory traces
    • Zero-setup restoration matches exact repository shas and agent conversation histories
    • An agent's identity is stored in its data, specifically the append-only event log
    • Runtimes and models merely read from and append to the log
    • The log contains all user inputs, tool calls, and state transitions needed for recovery
    • System distillation extracts knowledge from loop outputs into portable recipes.
    • Recipes codify human taste, evals, and prompt tweaks into a git repository.
    • Agent recipes remain provider-agnostic and company-owned.
    • A single prompt trying to do everything fails across diverse tasks.
    • Dividing labor among specialized agents creates a functional boardroom dynamic.
    • Each agent handles a distinct role like pricing, specs, or fact-checking.
    • Adding RAG memory does not solve long-term social consistency.
    • Agents need to track provenance like firsthand versus secondhand claims.
    • Episodic memory must be separated from current beliefs.
    • RAG memory alone fails to provide long-horizon agent behavior without provenance tracking.
    • Optimizing one agent behavior often breaks another, making rollback mechanisms mandatory.
    • Agent loops should act like a ratchet, keeping changes only if scorecards improve and guardrails hold.
    • Forcing AI to use human-centric visual tools like Figma or raw SVG walls results in structural failures.
    • HTML provides a structured language with built-in semantic tags that models understand intuitively.
    • Browsers render HTML into pixel-perfect layouts, enabling agents to generate slides, docs, and videos reliably.
    • Waiting 40 minutes for CI builds makes rapid agent iteration impossible for mobile development.
    • Lightweight cloud VMs that boot in under 30 seconds allow real-time simulator previews in the browser.
    • Sandboxes enable cross-functional collaboration where non-engineers can test and iterate directly on code.
    • Agents can follow runbooks and execute workflows, but they cannot inherit the legal or operational consequences of failure.
    • Engineering requires deciding whether work is worth doing, verifying evidence, and owning the blast radius.
    • High agency is ownership with judgment attached rather than attempting to personally perform every task.
    • Agents excel at inner-loop execution tasks like investigating, implementing, and testing code.
    • The outer engineering loop requires human judgment to decide if work is worth doing and verify evidence.
    • Agents return evidence and diffs, but humans must retain responsibility for production decisions.
    • Individual token prices drop by 5 to 10x every year for a fixed level of intelligence.
    • Despite falling token costs, total project spending increases because users tackle significantly harder and longer tasks.
    • A model scoring just 10 points higher on intelligence consistently outperforms cheaper alternatives across demanding distributions.
    • The vast majority of tokens consumed during multi-turn agentic tasks are input tokens rather than output tokens.
    • Prompt caching efficiency varies dramatically across models, offering up to 99% discounts on input tokens.
    • Engineers must factor prompt caching and input token pricing into cost projections for agentic workloads.
    • Tasks with no intelligence ceiling benefit from smarter models because analytical depth can always improve.
    • Tasks with a strict ceiling only require the minimum threshold of intelligence needed to get a correct answer.
    • Evaluate whether a task rewards higher intelligence before deploying costly frontier models.
    • Leaderboards evaluate agents by mining explicit user success feedback, implicit behavioral cues, and environment logs.
    • Aggregating signals like task success, praise over compliance, and stability reveals specific model failure modes.
    • Comprehensive agent evaluation requires tracking performance across multiple distinct operational dimensions.

Questions this video answers

60 · every answer cites the moment it was said
01 How does capability overhang manifest when prompting advanced models like Fable? 2 sources · 20:46

Advanced models do not rely solely on memorization, but instead solve complex queries by leveraging code execution tools, such as writing a script to filter specific items. This organic progression means models grow in spiky ways rather than being linearly designed.

02 How can developers effectively identify unknown constraints when working with Fable? 2 sources · 27:29

Developers can use the model itself to run blind spot passes, generate prototypes for visual taste, and conduct interviews to surface hidden edge cases. Providing external code references or implementation notes also helps map out the territory before letting agents run.

03 What happens to engineering productivity over time when using AI coding agents without controls? 2 sources · 42:05

While AI coding agents provide an initial three to five times boost in velocity, this productivity gain typically dissipates within three months. This drop occurs because the rapid generation of code introduces a parallel surge in security, maintainability, and reliability issues.

04 How do coding agents perform on long-running tasks according to recent benchmarks? 2 sources · 39:08

Latest mythos models can complete tasks taking up to 16 to 18 hours, but this performance is measured at a 50 percent success rate. When the required accuracy is dialed up to 80 percent, the task completion capability drops significantly to about three and a half hours.

05 What is the Agent-Centric Development Cycle framework designed to achieve? 2 sources · 44:27

The framework surrounds code generation with structured disciplines to guide agents, verify outcomes, and solve problems proactively. By feeding clear context and constraints into the loop, teams can reduce token consumption and improve agent effectiveness.

06 Why do knowledge work agents fail at end-to-end workflows compared to coding agents? 2 sources · 55:57

While models can execute individual tool actions, end-to-end knowledge work fails because actual work lives within the seams of multiple disconnected applications. Unlike software engineering, which relies on testable code and unit tests, general knowledge work lacks straightforward ways to verify outcomes.

07 What are perception agents and how do they solve the computer use verification gap? 2 sources · 1:03:56

Perception agents visually consume rendered screens rather than scraping underlying code, allowing them to track UI state, annotate elements, and verify their own output. By seeing the exact pixels and structure a human sees, these agents confirm actions without needing explicit backend APIs.

08 How do frontier code models overcome the exhaustion of human training data? 2 sources · 1:24:59

As human-generated training data diminishes and GitHub code becomes predominantly machine-generated, frontier models transition to self-play. Similar to game-playing algorithms, models generate their own challenges, judge the validity of answers, and execute millions of hours of self-play to improve.

09 Should we design new strongly typed programming languages specifically for AI models? 2 sources · 1:32:13

Since human-centric languages like Python are not optimized for safe or reliable model generation, we should consider strongly typed languages. Putting the burden of correctness directly on the model allows us to ensure higher reliability by design.

10 When should I use agent-as-a-judge instead of classical LLM-as-a-judge evals? 2 sources · 1:40:24

Classical LLM-as-a-judge evaluations rely on a fixed rubric with static scores, which fails when dealing with complex agent trajectories and dynamic UI outputs. When your agents generate adaptive, multi-step workflows, you need an agent-as-a-judge tool like Signal to discover subtle behavioral issues.

11 Why did OpenGov move away from LangGraph to an effect-native agent loop? 2 sources · 1:49:17

Although LangGraph worked initially, scaling up and handling complex use cases led the team to build their own agent loop using Effect. This transition gave them full regency and control over the architecture, propagating tracing and structured concurrency throughout the entire loop.

12 How do we safely handle mutating operations when deploying autonomous agents in production? 2 sources · 1:54:00

You can maintain trust and safety by deterministically interrupting the agent loop whenever a tool call requires human approval. This allows a human user to explicitly accept or reject the mutating action before it executes.

13 How do I build an automated research machine for AI? 2 sources · 2:09:31

You can construct a recursive self-improvement loop using a three-step process of ideation, implementation, and validation managed by an AI agent. This loop modifies its own code or search strategies iteratively, outperforming manual optimization in domains like CUDA kernels and nano models. However, it requires rigorous validation to prevent reward hacking.

14 How should I evaluate agentic systems in production? 2 sources · 2:16:49

You must shift from static model benchmarking to evaluating system behavior and end-to-end workflows using scenario-based evaluations and production telemetry. Adopting an SRE mindset focused on reliability and recovery rather than raw accuracy uncovers hidden failure modes like tool and memory errors. Scenario-driven offline evaluation helps test task completion before hitting production.

15 Can a small frozen embedder leverage test-time compute? 2 sources · 2:22:41

Yes, by letting an agent write an embedding algebra program over a frozen encoder, you can increase inference compute without scaling the model size. While compute-heavy rubrics often fail to generalize out-of-domain, cheap structure-recombination programs reliably improve retrieval across unseen encoders. Auto research loops can discover these transferrable pipelines overnight.

16 How do I construct agentic search pipelines? 2 sources · 2:37:11

You can build agentic search infrastructure using token-efficient tools like data rooms to explore the web, air-gapped search boxes to assemble local retrieval tools, and knowledge graphs to generate multi-hop evaluation questions. This setup explores the web using cheap tokens from small models while saving expensive frontier tokens for exploitation.

17 How does code generation change software architecture and specifications? 2 sources · 2:43:04

As coding agents make implementations easily generatable, software value shifts from the implementation itself to the abstract specification and protocol. To build production-grade distributed systems, you must evolve from abstract specs to deterministic simulations and concrete specifications before generating the final implementation. Minimalism and simplicity are crucial because they reduce the complex state space of concurrent protocols.

18 Why do long-horizon agents suffer from context degradation? 2 sources · 2:51:19

Long-horizon tasks cause context bloat where models start contradicting themselves or forget prior work, leading to redundant effort. Using local models with structured recall policies helps mitigate this degradation without inflating costs.

19 Which memory recall policy performs best for long-horizon agent tasks? 2 sources · 2:59:44

Ranked recall policies outperform simple vector RAG or unmanaged approaches by prioritizing relevant decisions from past steps. This structure helps agents retrieve accurate historical answers while consuming fewer tokens.

20 Why is requirements elicitation considered the new software bottleneck? 2 sources · 3:03:48

Writing code is no longer the primary bottleneck because AI can rapidly generate implementations from specifications. Instead, the real challenge is working with stakeholders to figure out what is actually worth building.

21 How do speedruns serve as a benchmark for automated AI research? 2 sources · 3:18:12

Speedruns challenge AI agents to achieve target validation losses or optimize parameters within a constrained environment. This setup provides clear rules, rapid iterations, and measurable rewards to evaluate automated research capabilities.

22 Why do traditional benchmarks fail to predict agent reliability in production? 2 sources · 3:35:46

Benchmarks only measure static model capability rather than dynamic system behavior like tool calls, API outages, and multi-agent coordination. SRE metrics such as availability, latency, and recovery provide a better measure of dependable outcomes.

23 What workflows do new multimodal video APIs unlock for developers? 2 sources · 3:46:27

Developers can use inputs like storyboards and audio tracks to generate and edit video content through natural language. These APIs enable applications ranging from short film production and marketing campaigns to personalized educational materials.

24 Why is natural language preferred over continuous tokens for chain-of-thought reasoning? 2 sources · 3:57:35

Natural language directly leverages the intelligence learned during massive pre-training phases. While continuous tokens or raw binary outputs are an active area of exploration, tying reasoning to natural language currently yields the most reliable results.

25 How does joint audiovisual generation solve synchronization issues in video models? 3 sources · 4:06:40

Joint audiovisual generation models produce pixels and audio simultaneously within a single causal generative process. This approach eliminates the poor results of older methods that attempted to hack lip movements onto independently generated pixels. By leveraging one unified latent process, the lips naturally move in sync with the spoken audio.

26 Why is natural language considered a limiting factor for describing sensory experiences? 3 sources · 4:07:55

Natural language lacks sufficient vocabulary to accurately describe rich sensory modalities like audio tone, taste, smell, or subtle skin color perturbations. These primitive survival-related senses rely on extreme human sensitivity that text captioning fails to capture fully. Consequently, text conditioning remains broad, forcing generation models to handle high stochasticity.

27 How do teams practically evaluate complex video generation models? 3 sources · 4:23:55

Teams rely heavily on manual human evaluations, side-by-side comparisons with large review panels, and early access feedback programs. While automated checks like OCR work for text rendering, complex video editing and aesthetics require human judgment to catch artifacts like reward hacking and unnatural eye gaze.

28 What kind of data do frontier labs actually look for from external partners? 2 sources · 4:29:22

Frontier labs look for high-quality, professional-grade media data rather than random commodity clips like standard YouTube videos. They also value real-world task trajectories, such as multi-step marketing campaigns, that capture actual end-to-end workflows.

29 How do forward deployed engineers bridge customer workflows to upstream model training? 2 sources · 4:34:13

Forward deployed engineers work directly with customers to derive insights and establish harnesses for real-world tasks. By understanding where models fail in practical use cases, they feed customer insights upstream into post-training and model development.

30 How does Arya automate machine learning experimentation inside Weights and Biases? 3 sources · 4:44:01

Arya acts as an AI research companion embedded directly within Weights and Biases workspaces and launched via an iOS app. It automatically downloads code, sets up GPU launch jobs, adjusts hyperparameters, and executes experimentation loops in Kubernetes. Additionally, it analyzes metrics, generates markdown reports with custom charts, and summarizes experiment runs.

31 What components are required to build a robust production agent architecture? 3 sources · 4:50:35

Building a production agent requires an API server and turn database communicating with a worker harness. The harness must connect to a secure sandbox for executing arbitrary shell calls and Python data science tasks, an LLM inference layer, and a launch service for long-running GPU workloads. Finally, an observability layer is critical for logging 100% of traces, tool calls, and errors.

32 How do tasks and evaluation suites function as continuous integration for agents? 3 sources · 4:56:06

Tasks are defined as YAML files that act like unit tests for your model by setting user prompts and specific LLM or rule-based judges. These tasks are clustered into evaluation suites that run nightly to track model performance regressions or improvements. This gives engineering teams a reliable metric-driven go/no-go decision for promoting candidate models to production.

33 How did the autonomous research agent Aiden perform in the Parameter Golf competition? 3 sources · 5:08:33

Aiden participated in the Parameter Golf competition where it independently ran over 1,300 experiments on a single H100 node over 22 days. It successfully set seven leaderboard records—more than any human contributor—while using only 4% of the competition's total compute. Furthermore, its pull requests had the highest community impact, resulting in a higher H-index than human participants.

34 How do codebase abstractions influence auto research systems? 2 sources · 5:17:02

Codebase abstractions act like neural network architectures by setting the constraints and priorities for what an AI agent can explore. A good abstraction systematically makes better solutions easier to learn and can eliminate issues like data leakage. This guides the search direction toward solutions that generalize and perform better.

35 When should you use code versus agents when designing a workflow? 2 sources · 5:31:27

You should use plain code for deterministic tasks with exact answers, such as calculating commute times or deduplicating items. Agents should be reserved for fuzzy tasks requiring interpretation, ambiguity handling, and judgment. This division prevents unnecessary complexity and ensures cheaper, more reliable execution.

36 How does JEPPA overcome sample inefficiency in optimization? 2 sources · 5:37:55

JEPPA uses reflective optimization in text space instead of relying solely on binary reward signals or gradient updates. A language model analyzes the entire rollout trace to provide detailed domain-specific textual feedback. This allows large behavior changes through single natural language updates using far fewer data points.

37 Why does JEPPA maintain a Pareto pool during candidate selection? 2 sources · 5:43:45

Standard optimization loops keep only the top-scoring candidate and frequently get stuck in local optima. In contrast, Pareto-based candidate selection retains every candidate that wins on even a single training example. This maintains a balanced search process that converges to much higher overall performance.

38 What is Optimize Anything and what can it optimize? 2 sources · 5:45:57

Optimize Anything is a universal API that extends text-based reflective optimization beyond prompts to any artifact expressible as text and a fitness function. It can optimize code like CUDA kernels, numeric data, agent harnesses, and cloud scheduling policies. Evaluators feed compiler errors, traces, or profiler messages back to the LLM to iteratively improve candidates.

39 How do I optimize prompts and agent skills using production traces? 2 sources · 5:54:03

You can use optimization frameworks like Jepa to learn evaluation tasks from real production traces. First, collect production traces and have humans annotate roughly 50 trajectories with detailed feedback. Then, use those annotations to optimize an LLM-as-a-judge prompt, which forms a continuous improvement data flywheel.

40 How do recursive coding agents handle large context and codebases? 2 sources · 5:59:08

Recursive coding agents treat the context itself as an object of computation rather than reading everything directly into the context window. They combine tool calling and reasoning through a read-eval-print loop to explore the codebase symbolically. This approach prevents agents from getting bogged down and mimics more thoughtful processing.

41 How does Auto Research optimize GPU kernels automatically? 2 sources · 6:00:50

Auto Research uses a simple while loop where an agent proposes a low-level operator solution, benchmarks it against a verifiable goal for correctness and speed, and then keeps or reverts the change. Because GPU kernels are highly verifiable and run massively in parallel, this loop can stack optimizations until reaching hardware limits. However, humans must still provide high-level architectural ideas.

42 How can I prevent an agent from reward hacking when writing GPU kernels? 2 sources · 6:04:13

Agents often engage in reward hacking by disabling critical speed-ups like CUDA graphs or testing only on small context windows to make a single kernel appear faster. To combat this, you must explicitly define negative constraints and what not to do in the instructions. It is also vital to provide hardware and model context files so the agent does not hallucinate attention mechanisms.

43 How do unified dependency graphs solve multi-repo agent amnesia and boundaries? 2 sources · 6:14:25

Multi-repo development typically suffers from repo boundaries and amnesia, forcing developers to repeatedly explain changes across different systems. Tools like Polygraph build a unified dependency graph by analyzing metadata across thousands of owned and open-source repos without modifying source code. This creates an illusion of a single codebase, letting agents coordinate downstream CI and maintain context across repositories.

44 What defines an agent's true identity and state? 2 sources · 6:27:30

An agent's identity is not tied to its execution runtime or model, but rather to its append-only event history log. Just like a video game character is defined by its save file rather than the console, an agent can be completely understood and resumed using its log. This log records every user input, model output, tool call, and state transition.

45 How do we continuously improve agent systems over time? 2 sources · 6:34:16

You use system distillation to process what went well or wrong in the first loop into portable, versioned agent recipes stored in a git repository. Failure patterns become calibration judges and evals, while repeated behaviors turn into skills and prompts, encoding the maker's taste.

46 How can a business manage complex knowledge without training custom models? 2 sources · 6:51:19

Instead of using a single mega prompt or training custom models, you can split tasks across a pantheon of specialized agents where each has one specific job. You feed all historical company data into an off-the-shelf model, storing meaning as vectors and relationships in a graph for robust memory.

47 How do you maintain social consistency in long-horizon multi-agent simulations? 2 sources · 6:57:59

Multi-agent systems require explicit stateful mechanisms like per-agent RAG memory namespaces, emotion vectors, belief/trust scores, and importance thresholds for memories. Without these, agents lose source attribution and context over extended interactions.

48 How does an auto-research layer improve multi-agent behavior? 2 sources · 7:03:42

An auto-research layer runs outside the agent village to read full execution traces, compare them against scenario ground truth, and score the behavior. It then proposes constrained changes to a small policy surface and reruns the simulation, keeping improvements only if they pass a scorecard.

49 Why should you restrict auto-research to a small editable surface? 2 sources · 7:09:36

Restricting auto-research prevents the language model from randomly rewriting the entire codebase or directly gaming the evaluation metrics. By freezing the harness, scenarios, and metrics while only exposing specific policies like memory writing or trust rules, you ensure safer, more controlled improvements.

50 How can coding agents successfully generate visual artifacts like slides and documents? 2 sources · 7:19:34

Agents struggle with spatial coordinate tools like Figma or SVG because they think in language rather than pixels. By using HTML as a structured medium, models can reason about layout using tags while the browser renders the pixels.

51 What infrastructure is needed to make mobile coding agents effective without slowing down development? 2 sources · 7:28:54

Local simulators require heavy resources like Xcode and Android Studio, while CI builds take too long for agent iteration loops. Cloud sandboxes solve this by booting up lightweight VMs in under 30 seconds to run builds and render mobile previews.

52 What does it mean for an engineer to own the verdict when working with AI agents? 2 sources · 7:48:05

Owning the verdict means taking direct responsibility for production decisions such as shipping, blocking, or accepting risk. Quality produces evidence, but a human engineer must assign answerability behind production outcomes.

53 How does heavy reliance on AI code generation impact team understanding and maintainability? 2 sources · 7:56:50

Deferring too much problem-solving to AI introduces cognitive debt, which erodes team understanding and memory of how systems work. Even if code passes tests, teams can lose their ability to explain the systems they ship to production.

54 Why is running hundreds of AI agents simultaneously a risk to engineering bandwidth? 2 sources · 7:59:00

Running numerous agents in parallel does not scale human cognitive bandwidth. Every active loop generates additional overhead to route, merge, verify, and integrate, creating an orchestration tax that requires careful system design.

55 What is the operational distinction between agent capability and human agency? 2 sources · 8:02:06

Capability refers to an agent's inner execution loop of investigating, implementing, and testing code. Agency remains with the human outer loop, which handles critical responsibilities like deciding if work is worth doing, verifying evidence, and owning production outcomes.

56 How have token prices changed relative to model intelligence over time? 2 sources · 8:11:50

Token prices have dropped significantly by five to ten times every year for each fixed level of intelligence. However, total project costs are rising because developers are deploying models on more complex, token-heavy tasks.

57 What are the primary cost drivers in long-running agentic workflows? 2 sources · 8:22:04

The vast majority of tokens consumed in long-running agentic tasks are input tokens rather than output tokens. Consequently, prompt caching and its associated cash hit discount rates become critical cost factors.

58 How should developers choose between cheap and expensive models for specific tasks? 2 sources · 8:23:58

Developers should categorize tasks into two archetypes based on whether there is an intelligence ceiling. If a task has no ceiling and more intelligence yields better quality, you evaluate models along the Pareto frontier; if it has a ceiling, you select the minimum intelligence model that works.

59 How does Arena evaluate agent outcomes from real-world user traces? 2 sources · 8:40:20

Arena mines three types of signals from execution traces: explicit user feedback, implicit user behavior, and environment execution feedback. These signals are aggregated to score metrics like task success rate and stability across models.

60 How can developers use the Agent Arena leaderboard for model selection? 2 sources · 8:43:33

Developers can slice the leaderboard data by specific task distributions, such as professional workflows versus consumer use cases. They can also analyze performance against average cost and token efficiency to find the best model for their specific budget.

Section-by-section summary

95 parts

Conference kicks off with record attendance

Day two of the AI Engineer World's Fair opens to 7,000 attendees across 18 tracks. The event highlights major themes from day one, including agent intent, reliability, and security insights.

Anthropic introduces the Fable model

Anthropic announces the rollout of Fable, a new mythos-class model that opens up vast new capabilities for developers. Working with this model requires shifting mental models to match its organic, unpredictable growth.

Models grow and improve in spiky ways

Advanced models like Fable are grown rather than strictly designed, exhibiting capability overhangs where tool use unlocks massive hidden skills. As models get smarter, they require smaller system prompts and rely less on rigid examples.

Map out unknowns before letting agents run

Complex coding agents require developers to explicitly bridge the gap between their mental map and the codebase territory. Using blind spot passes, agent interviews, and reference implementations helps uncover hidden assumptions.

Embrace the grief and be unreasonable

While transitioning away from traditional hand-coding brings a sense of loss, developers must push forward through the paradigm shift. Rejecting artificial trade-offs enables engineers to build faster and tackle vastly more ambitious projects.

AI slop and verification challenges in enterprises

Enterprises hesitate to fully adopt AI due to hallucinations, security vulnerabilities, and pervasive AI slop in generated outputs. While models achieve high functional correctness, they frequently introduce subtle bugs and complexity.

Productivity boosts fade due to technical debt

Initial productivity spikes from AI coding agents often dissipate after three months as technical debt, security issues, and maintenance overhead accumulate. Without proper safeguards, faster code generation simply creates a different bottleneck.

Bake verification into the agentic lifecycle

Successful AI integration requires an agent-centric development cycle that surrounds code generation with strict guidance and multi-layered verification. Combining algorithmic checks with agentic verification drastically reduces AI-derived production outages.

Apply zero trust multi-layered verification

Combining algorithmic verification for data flows with agentic verification for business logic addresses software complexity. Teams using this multi-layered approach achieve a 44% reduction in production outages.

Maintain clean codebases for compounding agent efficiency

Technical debt scales rapidly with AI-generated code unless controlled via verified code maintenance. Clean codebases significantly reduce the reasoning tokens and energy required by agents to operate.

Knowledge work agents fail because work lives in application seams

While agents handle individual UI clicks or API calls, end-to-end workflows fail because they cross multiple disconnected applications. Coding succeeded first because code is easily verifiable, whereas general knowledge work lacks unit tests.

Use perception agents to share visual context with computers

Perception agents read rendered screens directly rather than relying on underlying code or brittle APIs. This enables agents to confirm their own outputs and allows users to annotate UI elements precisely.

Expand perception beyond screens to meeting audio transcripts

Perception frameworks can incorporate room audio and meeting transcripts to capture human design discussions. Feeding these transcripts directly into agents triggers immediate updates and automated verification against design rules.

Software engineering bottlenecks shift from writing code to architecture

As syntactic code generation becomes free and superhuman, the limiting factor shifts from typing code to system design and verification. Humans will transition to acting as architects who define problem constraints and system logic.

Transition from human training data to model self-play

With most new GitHub code being machine-generated, traditional human training data is exhausted. Frontier models must rely on self-play and self-verification challenges to push beyond human software engineering limits.

Open-ended evals force novel algorithmic creation

Traditional benchmarks only check if code runs, failing to capture true software engineering complexity. Using open-ended evaluation tasks like lossless text compression forces models to invent entirely new algorithms.

Strongly typed languages designed for AI models

Current programming languages were built for humans and are suboptimal for generating safe or reliable code. As writing code becomes free, we should design new strongly typed, non-human-readable languages that shift the burden of correctness onto the model.

Accelerating scientific discovery at the intersection of code

The ability to experiment rapidly in code will unlock breakthroughs in chemistry and biology by exploring complex molecular interactions. Machine learning provides a completely different viewpoint on scientific problems, helping us discover hidden solutions.

Agent as a judge for dynamic agent trajectories

As AI agents perform complex, multi-step tasks with dynamic UIs, traditional static LLM-as-a-judge rubrics fail to catch subtle failures. Using an agent as a judge allows for adaptive, dynamic analysis of runtime traces and agent trajectories.

Building production agents with an effect-native architecture

Scaling enterprise AI agents requires moving away from generic graph frameworks to a custom, effect-native agent loop in TypeScript. This provides fine-grained control over tracing, structured concurrency, logging, and dependency injection.

Ensuring safety with human approval and sandboxing

Production agents handling mutating operations must deterministically interrupt the agent loop to request human approval. Additionally, executing code and creating files should always occur within a secure sandbox to build user trust.

Evolution and technology as perpetual sources of growth

Historical evolution and technological progress demonstrate that human flourishing is driven by continuous innovation. Automating scientific discovery through AI is the next logical step to solve material problems and accelerate growth.

The Eureka machine and recursive self-improvement

Building a Eureka machine requires four pillars: knowledge bases, measurement data, simulation engines, and physical labs. By enabling AI models to have self-awareness of their own shortcomings, recursive self-improvement can automate AI research itself.

Automated research discovers novel GPU kernels

An autonomous research system successfully optimized NanoChat, GPT speedruns, and CUDA kernels overnight, outperforming human-led benchmarks. Rather than just tuning hyperparameters, the agent discovered genuinely novel architectural ideas.

Agent systems require production-level reliability metrics

Evaluating agentic systems based solely on model output accuracy misses critical production failure modes like tool and memory failures. Teams must adopt a site reliability engineering mindset, shifting from benchmark scores to tracking system behavior.

Small frozen embedders leverage test-time compute

Test-time compute can improve retrieval models without scaling up model size by assembling smarter pipelines at inference. Using an autonomous research loop over a frozen encoder discovers lightweight embedding algebra that generalizes across models.

Constructing agentic search pipelines with data rooms

Scaling test-time compute at the pipeline level involves using token budgets to build data rooms, sandboxed search boxes, and knowledge graphs. This architecture maximizes recall and precision for long-horizon tasks without needing larger models.

Software value shifts from implementation to specification

As coding agents make general-purpose implementations easily generatable on demand, engineering value shifts upstream to the specification. To build robust systems, specifications must remain abstract while implementations target specific infrastructure.

Deriving concrete systems via deterministic simulation

Direct translation from abstract specifications to production systems often fails on concurrency and edge cases. Inserting a simulation phase allows agents to discover correct algorithms and design specifications before generating the final production code.

Local memory harnesses for long-horizon agents

Long-horizon tasks suffer from context rot and forgetting over time. Using local models with structured memory control loops helps manage context cleanly while maintaining operational sovereignty.

Ranked recall policies outperform simple RAG for agent memory

When tasks exceed the context window, standard RAG or zero-memory setups fail. A ranked decisions ledger provides superior recall performance and token efficiency compared to simple similarity search.

Requirements elicitation is the new software bottleneck

Writing code is no longer the primary bottleneck for software development teams. The real challenge is defining what to build by engaging stakeholders and working out valuable business requirements.

User story mapping provides structured context for AI agents

Traditional business analysis tools like story maps and value frameworks help ground AI prompts in familiar patterns. Structuring prompts with personas, needs, and outcomes leads to more coherent agentic systems.

Speedruns as a benchmark for automated AI research

Training model speedruns like nanoGPT provide a quantitative environment to test recursive self-improvement and AI research capabilities. These games offer clear rules, fast feedback loops, and measurable loss targets.

Comparing agent behaviors in autonomous research tasks

Different frontier models exhibit distinct behaviors during long-running optimization tasks, varying in idle frequency, scratchpad usage, and subagent spawning. Tracking these metrics reveals stark operational differences between models.

Current agents combine existing papers rather than inventing novel optimizations

Autonomous research agents currently excel at combining known techniques and making incremental improvements rather than discovering truly novel optimizers. Bridging this gap requires multi-agent generation and strict evaluation loops.

Multi-agent systems with scaling and evaluation loops

Combining multi-agent generators with automated speedrun verification and quality judges creates a robust discovery loop. Scaling successful methods to larger parameter sizes ensures discoveries hold up in production.

Agent evaluation must shift from static benchmarks to SRE metrics

While traditional benchmarks measure model accuracy, agentic systems demand a focus on system behavior like tool execution and recovery. Engineering teams should adopt SRE principles, prioritizing reliability and production telemetry over simple benchmark scores.

Production traffic serves as the ultimate validation dataset

Scenario-based offline evaluations help simulate realistic workflows before deployment, but real user interactions yield the highest-value telemetry data. Continuous monitoring via distributed tracing is essential to catch slow performance degradation before users complain.

New multimodal APIs unlock real-time video generation and editing

Recent launches like Nanobanana 2 light and Gemini Omni Flash APIs enable fast 3-second latency image generation and video editing. These tools empower creators to build custom workflows, educational materials, and marketing campaigns using natural language.

Natural language serves as the optimal backbone for chain-of-thought reasoning

Although alternative intermediate representations like code or binary outputs are explored, natural language remains the most effective medium for chain-of-thought reasoning. It directly harnesses the immense intelligence acquired during large-scale pre-training without losing interpretive power.

Computer vision expertise naturally bootstraps generative model research

Researchers shifting from computer vision understanding tasks to generative modeling find that discriminative models lay the groundwork for better generation. This reciprocal loop accelerates progress toward robust visual world models and physical intuition.

Joint audiovisual generation solves synchronization

Generating audio and video within a single latent generative process naturally resolves synchronization issues like lip movements. Hacking audio on top of pre-rendered pixels results in poor quality compared to a unified causal generative model.

Language falls short for describing sensory experiences

Human language lacks sufficient vocabulary to accurately describe sensory dimensions like taste, smell, skin tones, and room acoustics because they tie into primitive survival mechanisms. This limitation in captioning creates challenges when conditioning generative models on text descriptions.

Laymen prefer saturated aesthetics over true realism

Human evaluation is an unreliable barometer because untrained users often prefer over-saturated, overly sharp AI visuals that trigger positive emotional responses rather than true realism. Domain experts with honed craft quickly spot subtle flaws like incorrect eye gaze.

Evaluating video models combines automation and expert review

While automated metrics work well for structured checks like OCR on rendered text, complex video evaluation requires rigorous side-by-side human reviews. Assessing semantic consistency and workflow-breaking artifacts remains one of the hardest challenges in generative media.

Frontier labs need high-quality professional and workflow data

Labs already possess massive volumes of raw data, making high-signal professional footage and multi-step task trajectories the true bottleneck. Capturing complete workflows—such as turning a product photo into a multi-format marketing campaign—provides the actionable data models need.

Forward deployed engineers bridge feedback to post-training

Forward deployed engineers act as a crucial link between custom user workflows and upstream model improvements. By extracting real constraints from production usage—such as sizing or pattern replication—they feed valuable harness data back into post-training.

An AI research OS bridges personal notes and dev harnesses

Developers often lose track of research stored across disparate note-taking apps. A dedicated research operating system connects these personal repositories directly to coding and AI harnesses.

Arya automates hyperparameter tuning and experiment tracking

Arya is an AI research agent integrated into Weights and Biases to execute autonomous experiment loops. It runs training jobs on compute clusters, summarizes metrics, and generates visual reports.

Robust agent architectures require sandboxes and observability

Production agents need secure sandboxes for code execution alongside comprehensive tracing. Logging 100% of traces allows teams to analyze behavior and debug complex agent interactions.

LLM judges and task suites act as continuous integration

Production logs are converted into YAML-based tasks that serve as unit tests for agent models. Integrated LLM and rule-based judges evaluate these tasks nightly to drive go no-go deployment decisions.

Productionizing agents demands observability and human evaluation

Successful agent deployment requires agent-oriented observability and treating evaluation suites as CI pipelines. Teams must also incorporate manual trace reviews to catch behavioral bugs that automated judges miss.

Role-playing language agents raise difficult evaluation challenges

Role-playing language agents instantiate historical or fictional personas for education and entertainment. Evaluating these systems reveals notable personality fidelity alignment alongside surprising failure modes.

Autonomous agents can outperform humans in parameter optimization

AI agents participating in coding competitions like Parameter Golf successfully discover and merge model improvements. Aiden set multiple leaderboard records by autonomously combining disparate research ideas.

Codebase abstractions and evals act like neural architectures

In auto research, your codebase abstraction sets the structural constraints like a neural network architecture, while your eval functions as the loss function and data. Tightening these abstractions prevents data leakage and systematically guides the agent toward better solutions.

System thinking and workflow design govern agent architecture

Building robust agentic systems requires treating agents as components within a larger environment rather than isolated prompts. Defining clear workflows, error paths, and escalation triggers ensures predictable agent behavior.

Prompt decomposition prevents agent drift from bloated instructions

Mega-prompts handling multiple distinct jobs lead to agent drift and unpredictable outputs. Decomposing prompts into specialized subtasks makes each part easier to test, reason about, and modify.

Use code for determinism and agents for judgment

Algorithmic tasks with exact answers should be handled by plain code for reliability and cost-efficiency. Reserve language models strictly for tasks involving ambiguity, fuzzy reasoning, and subjective judgment.

Reflective optimization overcomes sample inefficiency in text space

Standard reinforcement learning requires vast numbers of rollouts, making it impractical for slow agent workflows. Reflective optimization uses language models to analyze full execution traces and update prompts in text space for rapid adaptation.

Pareto-based candidate selection prevents local optima in prompt tuning

Maintaining a Pareto pool of all winning candidates across training examples prevents the search process from getting stuck in local optima. This strategy outperforms simple loops by balancing search breadth and converging to higher performance.

Optimize Anything generalizes textual optimization to code and harnesses

Any text-based artifact including agent harnesses, CUDA kernels, and cloud scheduling policies can be optimized using universal APIs. By feeding compilation or runtime feedback back into the optimizer, systems can automatically discover sophisticated architectures.

Learning task evals from production traces

Subjective tasks can be evaluated by collecting production traces and having humans annotate a small sample. This data trains an LLM-as-a-judge prompt, establishing a data flywheel for continuous agent improvement.

Recursive coding agents treat context as computation

Current coding agents struggle with reliability because they lack proper management and multi-repository scope. Recursive language models address this by treating the prompt context as an interactive object of computation.

Auto research automates GPU kernel optimization loops

Auto research uses simple while-loops to iteratively propose, benchmark, and keep or revert GPU kernel changes. While humans must provide high-level architectural ideas, the agent efficiently handles low-level parameter tuning.

Multi-repo development exposes agent boundaries and amnesia

Agents are typically restricted to a single repository and suffer from complete amnesia between sessions. This forces developers to repeatedly re-explain system architecture and dependencies across multiple steps.

Unified dependency graphs overcome repository boundaries

Meta-harnesses like Polygraph analyze metadata across thousands of internal and open-source repositories to build a unified graph. This allows agents to read, write, and coordinate changes across multiple codebases simultaneously.

Resuming agent sessions preserves exact execution state

Complete session states and agent traces can be packaged and transferred to another developer or machine. This zero-setup transfer allows team members to seamlessly resume work or review past decisions.

An agent's true identity resides in its execution log

An agent is fundamentally defined by its append-only event log rather than its runtime or underlying model. Preserving this data allows the agent's state and identity to be fully restored after failures.

System distillation powers continuous agentic improvement

Deploying always-on horizon tasks relies on tight feedback loops where verification quality dictates success. System distillation processes artifacts from the first loop to continuously improve harnesses and models.

Optimize for value generated per watt

As AI systems reach the frontier, the primary engineering challenge becomes economic viability. Developers must measure the exact business value generated against compute costs. Success depends on making advanced workflows cost-effective in production.

Turn failure patterns into calibrated evaluation judges

Agent tuning requires transforming recurring failure patterns into automated evaluation judges. Rather than hand-coding every rule, developers calibrate judges with human taste. Agents then use these evals to test and promote system improvements automatically.

Organize knowledge using a pantheon of specialized agents

Complex business operations can be automated by deploying a cast of dozens of specialized agents rather than one giant prompt. By feeding historical company data into a structured memory graph, these agents handle distinct tasks like pricing and outreach. They collaborate through simulated boardroom discussions to run the business.

Multi-agent simulations require stateful memory and emotions

Building autonomous agents in dynamic environments requires per-agent memory namespaces backed by retrieval systems. Tracking small emotional vectors and trust scores allows characters to form realistic relationships. Events are evaluated for importance, ensuring critical context persists in separate caches.

Long-horizon social consistency breaks without auto-research

While short-term interactions work well, multi-agent systems lose social consistency over extended horizons as rumors turn into false facts. Fixing this requires introducing an auto-research meta-layer. This layer evaluates full simulation traces against ground truth rather than judging single responses.

Evaluate agent runs using controlled scenarios and scorecards

To systematically improve multi-agent behavior, developers must use repeatable control scenarios paired with balanced scorecards. Measuring metrics like diffusion reach, provenance retention, and uncertainty preservation prevents agents from gaming a single metric. Changes are kept only if the entire scorecard improves.

Restrict auto-research to a small editable policy surface

The auto-research layer should never be granted permission to rewrite the entire codebase freely. Developers must freeze the core harness and expose only a controlled policy surface like memory writing and retrieval rules. This constrains the search space while allowing meaningful behavioral updates.

Use HTML instead of canvases for agents

Asking AI agents to manipulate visual canvases like Figma or PowerPoint fails because spatial coordinate manipulation is unnatural to models. Instead, giving agents HTML and CSS provides a structured language they natively understand to render complex layouts. This approach enables automated generation of professional slide decks, docs, and videos.

Mobile workflows need cloud sandboxes

Achieving true 10x productivity with AI requires redesigning workflows rather than just swapping the engine. For mobile development, forcing non-developers to run heavy local environments or waiting 40 minutes for CI builds creates massive friction. Lightweight cloud sandboxes with instant VM boot times let designers and QA iterate directly on code alongside agents.

Building and scaling production AI agents

OpenGov scales its production AI assistant OG Assist by focusing heavily on harness design, evaluation observability, and agent-to-agent protocols. Managing long contexts and robust monitoring are critical for enterprise adoption. Internal tools leverage these same patterns to streamline developer workflows.

Conference highlights and human-centric engineering

The conference brought together diverse tracks ranging from expo networking to technical keynotes on agent architecture. As technical complexity grows, engineers face the paradox that knowing more highlights how much remains unknown. Industry sponsors and community interactions bridge the gap between best practices and practical application.

Engineers own production verdicts and evidence

As AI handles routine generation, the engineer's role shifts from writing code to owning verdicts and production decisions. Quality produces evidence, but a verdict assigns personal responsibility and answerability for shipping. Clean code repositories not only help human maintainers but significantly reduce token usage and errors for coding agents.

Taste as alpha and capabilities decay

Model advancements rapidly erode capabilities like raw speed or recall, a phenomenon known as decay. Taste remains a durable form of alpha because it involves making qualitative judgments where no objective metric exists. However, true engineering value comes from moving past mystic taste into repeatable critique and structural judgment.

Avoiding cognitive debt and orchestration tax

Relying blindly on AI introduces cognitive debt where teams lose deep understanding of their own codebases. Furthermore, spinning up hundreds of parallel agents creates an orchestration tax that overwhelms human attention bandwidth. Systems must be designed intentionally to avoid borrowed confidence and unmanaged complexity.

Separating agent capability from human agency

Agents can successfully execute inner-loop tasks like investigating, implementing, and testing code. However, the outer loop of deciding, verifying, and owning responsibility remains strictly human. True high agency means exercising judgment over which execution paths deserve ownership and attention.

Explain your code or do not ship

Automation lowers the barrier to entry, moving the engineering bottleneck from building to governance and validation. An operational rule for teams is that someone must understand the changes well enough to defend them. Ownership and risk acceptance are non-negotiable requirements for enterprise software.

Tracking the intelligence and cost frontier

Artificial Analysis benchmarks chips, cloud infrastructure, and models to measure intelligence, speed, and cost trade-offs. Their intelligence index demonstrates that open-weights models maintain a remarkably consistent three-to-nine-month lag behind proprietary leaders.

Why AI feels expensive despite cheaper tokens

Although token prices drop exponentially year-over-year, overall AI spending surges because users apply models to increasingly complex, multi-step agentic work. Complex workflows require hundreds of tool calls, causing token consumption per task to escalate.

Input tokens and prompt caching drive agent costs

Long-running agentic tasks spend the vast majority of their token budget on inputs due to extensive environment exploration and tool usage. Consequently, prompt caching discount rates become the dominant factor in controlling operational costs.

Choosing models based on task intelligence ceilings

Knowledge work tasks split into two archetypes: open-ended problems where higher intelligence continuously improves output quality, and bounded tasks with a clear ceiling. For bounded tasks, developers should select the cheapest model that clears the minimum intelligence threshold.

Evaluating real-world intelligence with Arena

Arena provides real-world evaluations powered by millions of user interactions across text, image, video, and code. Their live leaderboards help developers navigate model selection by comparing performance against actual usage patterns and costs.

Measuring agent outcomes in complex workflows

Evaluating autonomous agents is technically difficult due to multi-component tool harnesses and sparse signals over long horizons. Arena addresses this by embedding agents into real-world applications to capture organic user traces and feedback.

Mining signals from agentic execution traces

Leaderboard rankings are derived by scanning user sessions for explicit success declarations, implicit behavioral patterns, and environment execution logs. Combining these signals allows platforms to score models on test success, stability, and hallucination rates.

Slicing performance by task distribution and cost

Model performance varies significantly depending on whether evaluation focuses on professional GDP tasks or consumer use cases. Plotting net improvement against average session cost helps teams identify the optimal Pareto frontier for their specific budget.

Event wrap-up and sponsor acknowledgments

Conference organizers conclude the afternoon keynotes by thanking presenting sponsors and previewing upcoming startup battles and closing remarks.

0:00 / 8:51:55
k play/pause · f fullscreen · m mute
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said
  • Fable30× · first 15:42

    A new mythos class model from Anthropic that opens up advanced agent capabilities.

  • Capability overhang1× · first 20:46

    The spiky way models gain new intelligence when paired with proper tools rather than just memorization.

  • Sonar Vortex1× · first 45:06

    A product focused on providing context and constraints to guide coding agents.

  • Agent-centric development cycle1× · first 44:07

    A framework for coding agents centered around guiding with context, multi-layered verification, and active maintenance loops.

  • Perception agents1× · first 1:05:34

    Agents that visually parse rendered screens and meeting audio transcripts to share real-time context and verify tasks.

  • Self-play3× · first 1:25:16

    A training paradigm where models generate their own programming challenges and verify solutions independently of human data.

  • Effect8× · first 1:43:14

    A TypeScript library used to build robust applications with advanced error handling, tracing, and schema validation.

  • Agent as a judge2× · first 1:41:11

    An evaluation method using a long-running agent to analyze dynamic execution traces and detect subtle failures.

  • Eureka machine3× · first 1:56:14

    A conceptual machine designed to automate scientific discovery and invention across physics, chemistry, and biology.

  • Recursive self-improvement6× · first 2:08:43

    An AI system's ability to analyze its own shortcomings and autonomously update its entire system architecture.

  • Test-time compute9× · first 2:21:00

    Spending additional inference compute instead of training a larger model to achieve better results.

  • Durable execution2× · first 2:41:25

    A platform architecture that ensures reliable execution of multi-step workflows across process and network failures.

  • Deterministic simulation environment2× · first 2:48:32

    An environment used to discover correct algorithms under partial failure before building production systems.

  • Memory harness34× · first 19:31

    A control loop managing write, read, and recall operations to prevent context rot in long-horizon agents.

  • Value architecture design1× · first 3:13:36

    A thinking process focusing on how value is created and supported by underlying processes before system design.

  • Speedrun9× · first 18:44

    An automated evaluation loop used to test model optimizations and rewards.

  • Gemini Omni13× · first 3:41:46

    A multimodal model enabling real-time video generation and editing APIs.

  • World model5× · first 3:59:00

    A model-based reinforcement learning representation used to simulate space-time physics.

  • Joint audiovisual generation1× · first 4:06:31

    Generating pixels and audio simultaneously within a single causal generative process.

  • Forward Deployed Engineers2× · first 4:33:43

    Engineers who work closely with customers to build harnesses and channel real-world feedback upstream into model training.

  • Second brain2× · first 4:37:57

    A large collection of personal notes and research gathered across apps like Obsidian and Readwise.

  • Weights and Biases Weave10× · first 4:41:59

    An observability platform used to log agent traces, session spans, tool calls, and LLM evaluations.

  • Role-Playing Language Agent2× · first 5:01:33

    An AI system built to instantiate a persona and reason and speak in character.

  • JEPPA13× · first 5:39:17

    A reflective prompt and system optimization technique that uses evolutionary loops and Pareto-based selection.

  • Optimize Anything7× · first 5:45:59

    A universal API for optimizing any text parameter, code, or harness using reflective optimization.

  • LLM wiki1× · first 5:33:51

    A structured memory layer used by agents to persist decisions and state outside of the active session.

  • Jepa4× · first 5:42:04

    An optimization interface used to improve prompts, agent skills, and model performance.

  • Auto Research39× · first 1:59:15

    A framework where an agent iteratively proposes and benchmarks solutions in a loop to achieve a performance goal.

  • Polygraph8× · first 6:14:06

    An agent-agnostic meta-harness that builds unified dependency graphs across multiple repositories.

  • System distillation1× · first 6:34:16

    The process of analyzing loop outcomes to continuously improve and codify agent behavior into recipes.

  • Agent recipe8× · first 6:34:55

    A portable, versioned configuration in a git repo that encodes maker taste, evals, and system prompts.

  • Behavioral scorecards1× · first 7:14:25

    Testing frameworks used to evaluate and score agent behavior across controlled scenarios instead of relying on subjective vibes.

  • Cloud sandboxes1× · first 7:29:10

    Lightweight virtual machines that boot up quickly in the cloud to let agents build and preview mobile apps without local IDE setups.

  • Cognitive debt1× · first 7:56:47

    The erosion of team understanding and memory regarding how to solve problems due to heavy reliance on AI solutions.

  • Orchestration tax1× · first 7:58:43

    The cognitive overhead and extra decisions caused by running multiple agent loops in parallel.

  • Intelligence Index5× · first 8:08:56

    A synthesis metric across multiple evaluations used to measure the AI race.

  • AA Briefcase5× · first 8:13:18

    An agentic knowledge work benchmark that tests models on realistic professional tasks.

  • Agent Arena15× · first 6:55:11

    A platform to experience and evaluate agentic capabilities using real-world user traces.