Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic

12,318 words · about 49 min to read · YouTube’s automatic captions

Applications often sit inside the Pareto frontier due to misconfigurations

0:12 We can kick it off. Thank you all for coming. I'm Lance. I'm member of technical staff at Anthropic. And I actually want to share some very simple things with you today. A few simple ways to get the most out of frontier models. So we typically think of cost and quality as a trade-off. So in order to get, you know, better performance, you have to spend more money. And in order to kind of reduce cost, you have to scratch sacrifice performance.

0:40 And so you can kind of think about it on any given evaluation, you have models that kind of create this Pareto frontier. And you can imagine this case, Fable is kind of up on the far right of the curve, and then you have Opus, you have Sonnet, then you have Haiku. Now what I want to share is actually that many times you're not actually on this frontier. Your application is actually sitting inside this frontier that you could be on.

1:05 And it's because of minor misconfigurations. Which I'm going to show you today how to fix.

1:11 [snorts]

1:11 And we've seen this repeatedly. Improve performance and reduce cost for many customers using some view very simple things. So first, I'm going to cover five themes. And then we can make it interactive. The slides won't take an hour. Not even close. So I question answering is actually much more

Remove legacy prompt boosters and verification rituals

1 Highlight · Frontier models make legacy prompt boosters obsolete

1:31 fun. First, audit your prompts. This is a kind of important point that prompts accumulate debt just like code. So if you recall several generations ago, we used to use things like think step-by-step. We used to use verification rituals, double-check your work. We used to use thought from this boosters, be maximally thorough. Sometimes we would use kind of scratch pads, tell the agent how to think, give it places to think.

2:00 You don't need that anymore. Frontier models are very good at at thinking, choosing when to think. They don't need to be told what to do and like kind of how to verify their work. They naturally verify their work. Um they don't need told to be thoughtful. They're already very thoughtful. Um and worse, when you do these things, it actually increases cost and can reduce performance in some ways I'll talk about briefly.

2:28 Another interesting one with earlier

2:29 models, if your prompt had contradictory rules in them, earlier models that weren't as good at instruction following would kind of ignore one of the two rules. So it kind of still worked. But with frontier models, they'll often get confused by contradictory rules. So I'm going to give you one very simple thing to do right now if you have any code that you want to check. In Claude code, you can run this command checkup prompt audit. You can also run Claude API prompt audit. These are two different skills in Claude code. They link to the same code.

3:00 It's just checkup is typically how people do checkups on their Claude code configuration itself. So your skills, your prompts within Claude code itself, the Claude API skill is a separate skill bundled in Claude code that's meant for a kind of code that uses our APIs. But both of these commands use the same principle and they kind of link down to the same underlying skill itself. It's just two different ways to call it.

3:23 [snorts]

3:24 These skills will audit your code, anything in the working directory of Claude code. So that could be skills, that could be tools, that could be prompts for these anti-patterns and will remove them for you. This has become kind of very popular. Many people we all Many people use it within Anthropic. It's kind of become somewhat popular, you know, on Twitter as well. Um it's very simple and often it can improve cost and performance. So here's an example uh where it This is an evaluation that I built.

3:54 It's a synthetic evaluation of customer support tickets. There's like 50 tickets. And what I do is I start with Opus 48. And I basically simulate a migration to Opus 55, okay? And in this process, I inject each anti-pattern individually and measure the result. That's this middle point, okay? So, what happens is it's taking the average of all the anti-patterns across the benchmark, basically. The key point is this.

4:25 When I do the migration to Opus 55 from 48, initially with these kind of anti-patterns injected, we do see some cost savings and a little bit performance savings. You can see it goes to the left and it goes up a bit. If I run prompt audit on top of that, I get a further bump both in performance and cost reduction for a bunch of different reasons. Because this command is going through and it's removing verification rituals, boosters that are actually causing Opus 55 to expend excessive tokens.

4:58 So, this little simple command often will basically reduce token use, saving cost, and it can improve performance in cases where things like contradictory prompts were confusing frontier models. So, this is a very simple command that I recommend. Um and we've used it very extensively within Anthropic and also many customers have enjoyed this quite a lot. It's very simple to run. You run it kind of in any code using the API or even on your cloud code configuration to make sure your prompts are kind of well-tuned for frontier models.

5:28 That's point one. Now, point two is make sure you're using the prompt cache. This is probably pretty familiar to many

Maximize prompt cache hit rates for multi-turn agents

5:35 of users, but I do just want to make sure it's extremely clear. So, agents, for example, operate across many turns. Every turn adds just a little bit. For example, one tool call. Much of kind of context is repeated across turns. You don't want to pay for it twice. That's why you can use the prompt cache. So, prompt cache is built at like a tenth the the rate of base tokens. So, it's much cheaper. Varies a bit per model. I think 10 a a tenth is I think the current pricing, but you can check the pricing page. The point is it's much cheaper.

6:07 It's also much faster. So, it's very important in multi-turn applications to

2 Highlight · Prompt caches require strict prefix matching

6:11 be making heavy use of the prompt cache. Now, the problem is prompt cache are a strict prefix match. So, any even a byte change in your prefix will invalidate the cache. So, you have to be careful in building agents or multi-turn applications that you don't break the prompt cache. That's the key point. And Claude console, if you're using the API, has nice diagnostics for you. So, if you're building applications with the Claude API, you can see for your application your prompt cache hit rate, which is very important to track.

6:43 For example, with Claude code, we track the prompt cache hit rate very very religiously. It's very important for the economics of the application, obviously. And it's also in your cases, if you're building apps with the Claude API, it's very very important to make sure you're using making heavy use of the prompt cache. Now, in general, how to organize your prompts. You know, you can think about basically, um system instructions,

7:07 tools, things that are globally cached across all of your sessions. You can think about Claude MDM memory. That's kind of cached per project. Session state is kind of per session, and then messages kind of incremented each turn. So, that's kind of how you want to structure your prompts. And the key point is this, run this in Claude code. So, if you have a project using the API, you want to make check and make sure that you're using you're making proper use of the prompt cache. You can run this. It'll run diagnostics to make sure that you're using the prompt cache effectively. It'll find opportunities to reorganize your prompts to make sure that they're organized such that you get a high prompt cache hit rate. That's really the key point here.

7:42 [sighs]

7:43 And this is kind of a fun exercise. So, myself and some folks on our Claude AI team ran um ran this command on a bunch of internal benchmarks that had kind of default harnesses. We found opportunities to reduce costs by up to 72% with no meaningful change in performance, just by running this command, because some of the harnesses on these Evals were not using the prompt cache properly. Really simple command. You can run it in your API code and it'll just look for opportunities to make sure you're saving as much cost as you should be, i.e. by using the prompt cache. It'll also check for other things. It'll look for um I'll I'll talk about it a little bit later, but it'll also check your effort levels, it'll check your prompt. So, it it'll do a number of other things, but in particular it'll look at your prompt cache to make sure you have a high prompt cache hit rate.

8:26 Again, this has, you know, optimized prompt cache has no change in performance. It's like effectively just free cost savings. So, it's something to be very serious about when you're building applications with the API in particular. Um it's very something to very to very

Design evaluations with clear headroom and low variance

8:40 seriously make sure you're utilizing effectively. So, I want to get into to some other fundamentals. Um we talked about prompt caching, we talked about making sure your prompts are kind of well that are that are well-tuned. You can audit them using prompt cache. Now, let's talk about a few other things that are very important for working with frontier models. So, Claude, frontier Claude in particular, is very good at self-correction.

9:09 So, if you give Claude a well-structured evaluation, it can basically just hill climb on that in that eval and I'll talk about that in a minute. But important in that is good eval design. And you have to give Claude an evaluation that is climbable. And so, actually we spend a lot of time thinking about eval design and making sure Claude is set up for success when building an eval and setting up hill climbing. So, some general features of good eval design. This is This is basics, this is nothing [clears throat] related to, you know, Anthropic, this is just kind of fundamentals, but it's good to revisit briefly.

9:43 What you should see generally is that these are a number of different curves looking at basically tokens per attempt. This is kind of like thinking on the X and then score on the Y. Frontier model at the top, mid-size model on in the middle curve, small model at the bottom. So, think about that as like Opus, Sonnet, Haiku. You should generally see on your evaluation as you step up model and performance goes up. That's point one. As you increase thinking, performance should go up. That's point two.

10:11 There should be headroom to improve at the frontier. So, that's point three. So, basically shouldn't be saturated. And there should be low run-run variance. Very simple points, uh but actually very important to get right. And actually even a lot of very popular evaluations that we see have many tasks that are ambiguous, effectively limiting that headroom at the frontier. And this is a problem I see all the time. You build an evaluation, it turns out you have a bunch of ambiguous cases that can't be solved anyway. So, it looks like, you know, oh, I've 20% more that I can get on this, but no model can actually achieve it.

10:45 So, you could spend many tokens [clears throat] hill climbing on an evaluation that's effectively saturated. It's very important to get this right.

10:52 [sighs]

10:53 Try this command. Uh Claude API build eval. This is actually a very nice command because it's not opinionated at all about where your data lives. It doesn't care. You can point this at whatever tracing solution you have. It's just a skill in Claude code. It's more employing the principles I talked about to test and sanity check the evaluation for you. So, you can kind of give this skill, hey, here's a problem I want to solve, here's what I'm interested in, here is the raw data, for example, here's the traces that I want to work with, and this command can help you build an eval set. It'll walk you through a process and step you through building it. It'll it'll help you you'll show you the data, it'll show you the evaluation, it'll sanity it'll sanity check the grader the criteria with you. So, it'll kind of walk you through the process of building eval based upon some of our, you know, some of our best practice and we have a nice blog post on this as well that that I'll share.

11:46 Um and it pairs very well with hill

Split evaluation data to prevent hill climbing overfitting

11:49 climbing. So, once you've built a good evaluation that has headroom at the frontier, low variance, scales as expected with thinking and model, you can then climb on it. Frontier Claude models are very, very effective at hill climbing. So, autonomously looking at an evaluation and autonomously kind of improving on it without you in the loop. So, you build a good eval, Claude can hill climb on it. The problem is there's many ways that this can kind of go wrong. And in particular, you can end up overfitting on the evaluation itself.

12:22 So like some classic examples would be um like the task in your benchmark have particular needs that are bespoke or specific to those tasks. Like maybe one task needs OCR, so it happens your harness to solve the eval adds like an OCR tool. But in production, you don't see that problems. So you end up overfitting on the eval distribution, it doesn't apply to production. This is very common. Um yeah, like distinctive phrasing specific to your eval examples.

12:52 For example, another classic is like it just kind of it like the prompt when you're basically hill climbing overfits the specifics of your eval distribution, it doesn't apply to to your to your test distribution. So overfitting is the major challenge concern when you're hill climbing. Imagine you spent a lot of tokens hill climbing an evaluation. It looks really good after hill climbing. You run it in production and performance is not better at all. That's frustrating because you spent a lot of money. You hill climbed and like, oh, it's not applicable to production because often you'll you've just kind of overfit to the eval distribution.

13:27 So try this command. In Claude code is Claude API hill climb. So if you build an eval, it'll hill climb for you. And it tries to combat this with a few tricks. Now it's of course not perfect. You have to audit this carefully and and and indeed be in the loop and and make sure you really understand what's happening. But it can help you.

3 Highlight · Preventing overfitting during autonomous hill climbing

13:46 And in particular, what it does is a process like this. It will look at your evaluation and it will split it into test and train sets. This is kind of very classic um kind of classic principles from statistics. Create a test and train split. What happens is when hill climbing, this is just on Claude code by the way. So Claude code basically has this skill. You run it on an eval. It will only ever look at the train split and reflect on those failures when it's making decisions about what to change to improve on the eval.

14:18 So, what's nice about that is at every round, it never sees a test split. And it will basically check performance per round against test and train. So, what happens is a little it's a little early signal for overfitting. If you see train improving but test flat, it's a signal of "Ooh, you might be overfitting." So, basically it applies this check and a number of other more subtle things check to make sure while hill climbing's happening, you aren't just overfitting to your eval distribution, you're actually hill climbing in a way that will apply to production.

14:51 Assuming you built your eval distribution well, and that's always something that we can address in that build eval phase. Now, I'll share some concrete examples that I've used here. So, this is actually the Claude API skill itself.

15:04 So, if I back up a little bit, this is a skill in Claude code. And it is a skill that gives Claude full awareness of our APIs. Uh so, basically our SDKs, APIs. And it has to be grounded in our documentation. So, I built an eval set based upon our documentation. And I tested the skill. The skill initially was not very good at the eval questions in my set, 66%. So, I set up a hill climbing loop with giving Claude code access to the skill text and the documentation itself, and let it basically just iteratively look at the failures in the eval set, and try to figure out why the skill was misrepresenting what's in the documentation. And there's all sorts of little tricks, and I could, you know, talk about that more in more detail. But the point is, Claude can autonomously find, in this particular case, the right skill text to improve significantly on the evaluation. And I talk about this in the blog link below.

16:03 Um but again, this is a good example of a kind of a well-scoped evaluation, a simple service I'm modifying a skill, and an autonomous hill climbing loop that resulted in a much better performance just when I set the system up this way with a carefully scoped eval, a surface that's easy to modify, claw can just rip through it and like significantly improve performance. This

Use automated hill climbing to aggressively reduce costs

16:22 works well for prompts, it works well for skills. Simple text-based surfaces are very very good for this kind of hill climbing process. Now, this is another kind of interesting trick that I've seen a lot of customers like. You can also apply hill climbing to cost reduction. So, what's kind of nice about this is I've added guidance to this hill climb command. If you tell it to reduce cost, what it'll do is it will basically try to find a configuration that's that is at is at least at parity in performance and for sure reduces cost. And it'll search things like effort, model, and your prompts. Here's a simple example on that same um customer support benchmark.

17:02 I start with Opus Opus 48 high with some baseline prompt. And I basically tell the hill climber reduce cost. So, what it does, it basically will search through model effort combinations and basically pick those that continuously reduces cost over the rounds without degrading performance. In this case, it moves to Opus 5 low. We see a big bump in performance and minor reduction in cost. The hill climber then autonomously tries flying at 5 low. We see again degrades performance a bit, but big cost reduction.

17:35 And then it it then decide tune the prompt and we kind of risk rescue the performance. So, we basically got a 90% reduction in cost with notable improvement in performance autonomously using the hill climber. This is another very nice way if you have an evaluation, you can use the hill climber to basically say, "Okay, keep performance fixed but reduce my cost materially by tuning things like effort, things like model selection, and my prompts." Another very nice use of the hill climber. And actually, what's nice about this is you can do this if you have a saturated eval. So, you can basically say, "Okay, say my eval saturated, fine. It doesn't matter. I'm not I don't care about performance. I just want to reduce cost." So, then you can just tell the hill climber, "Okay, keep performance fixed and just try to burn down cost." And it can basically search for configurations and often it'll find very interesting things.

18:21 Like in this particular case, there's generally an intuition that moving to higher models at lower effort is often cheaper and more performant than lower models at high effort. It's like giving someone who's smarter can like try less to actually get a better objective or or or to kind of solve the problem more effectively. Um and so it ends up actually using fewer tokens and as a result ends up being cheaper. That's kind of a broad intuition here that we've seen quite a bit. And that's baked into this hill climber skill.

18:49 So, um another nice trick. Now, I'm going to open up the questions

Architect long-duration agents with recoverable components

18:54 pretty soon. I'll just kind of close with some other comments. And this is a bit of a nod to the magic agent stuff you heard this morning. I just want to re-emphasize this a little bit. I do think um as part of unhobbling the frontier, it's very important to design harnesses that can take most advantage of the latest Claude models. And in particular, it's interesting if you look back the last few years, time is on the X, and this is task duration for human on the Y. You've seen these curves. They're just meter charts.

19:25 It's interesting to think back, you know, Claude Opus 3 was capable of only like 4 minutes of autonomous work. And in that kind of regime, there were very limited products you could build. It's really auto completion or chat. Now, we saw over the course of 2025, models become more capable. They can do like hours of work. Like maybe an hour to 2 hours of human equivalent work. Claude code obviously kind of um gained significant over the past over 2025.

19:51 And we're kind of entering a new regime now where these um task durations for models are are exceeding like a day to 2 days. I think this is kind of outside the scope of what meter can even measure now. But in this regime where you have models that can do the equivalent of like, say, days of work, what harnesses do you need? Like the considerations change quite a bit because you want models that can just do things autonomously over very, very long periods of time. Now, how do you make sure they're secure and reliable? The considerations change a lot.

20:21 And I hear you heard about that morning, but this is actually the motivation behind one of our newer APIs called Cloud Manage Agents. Um and the principle here, I wrote this

4 Highlight · Decoupling the agent brain from sandboxed execution

20:30 engine blog back in April on it, is decoupling the brain from the hands. And the idea is pretty simple here. You have an agent harness, which is effectively the brain. You have hands, which are sandboxes. That's where the agent harness writes code and performs actions. And you want those to be separate things. In particular, you want that you want it to be the case that if the harness dies or the sand box dies, the system can recover. Imagine you have an agent that you spin up overnight to just run and the sandbox dies midway through the night. It's very frustrating. You lose state, you lose your session log, and everything.

21:02 So when we design this, we designed this for a world where Cloud would be doing work for very, very long durations. And so it's recoverable. So that means the session log is an independent object separate from the harness, separate from the sandbox, that is append-only, that is recoverable. So if the harness dies, the sandbox dies, you can always recover from the session log. Sandboxes are also recoverable. So if a sandbox dies, you can boot up a new one. If the harness dies, you can boot up a new one through this orchestrator.

21:28 Another point here is safety. So one of

21:31 the important things that Manage Agents does is it actually never gives Cloud direct access to secrets. Secrets are held in vaults. Um and that's actually stored in that kind of tools box you see at the top. So vaults ensure that Cloud can reference API keys or secrets, but it can't actually see them. Because obviously, if you have an agent that's working for, say, days on end, it's a little bit spooky to give it direct access to very kind of sensitive secrets. And so we ensure the security is is kind of very is is a very important consideration when we build Manage Agents.

22:02 So reliability and security become much more important as agents can operate for hours, days, and maybe soon weeks. And this morning you probably heard a little more about this, but we have a nice uh recent blog that I put out with CJ that walks through uh an example of basically building an agent that can serve as like kind of a an an agent automation uh for regular context updates. It can reach into different sources that you specify, it can write updates to Slack. Very simple automation example, it's highly customizable, but it shows how all these different primitives come together. It brings together a scheduled deployment, all this all runs on Anthropic infrastructure, it secure- it securely stores secrets, it has memory, and um it's a very nice kind of end-end example.

22:49 Now again, the final point is you can just run this in Cloud Code. So, run Cloud API magic sandboard and then give it the link to the blog, and it'll just boot up this tutorial for you, it'll spin up the example, and you can just play with it. Um so, the really the key point here is um there's a number of ways you can kind of harness and uh make sure that Frontier is effectively unhobbled. Uh it is very important to audit your prompts.

23:17 It's very important to make sure that you're effectively using prompt cache, especially building agents for long duration, to make sure that cost is managed. It's important to think about harnesses that are designed for long duration in terms of reliability and security. And it's also important to think about evaluations and hill climbing. And when this all comes together, you can have agents that you can give a task, for example, an evaluation, and let it run for long period of time and hill climb on that evaluation, come back, for example, in the morning, and it's it's significantly improved.

23:43 And that's really the way that I see many of my colleagues working now. People are spending much more time on eval design, and basically just specifying tasks that Claude can effectively hill climb on, and doing this in a way such that agent is highly secure when doing this, and is highly reliable for long duration jobs. And let's see where we are on time. With that, I think I'll just take it to questions, because it's more fun when it's interactive anyway. You can follow me on all the the socials here.

24:11 And uh thank you for your time and

Use binary rubrics and pairwise comparisons for LLM grading

24:13 attention.

24:25 Sure, start there.

24:27 This is a very good question. The question was how automatable is eval design? So actually I was engaging a bit with with Hamel on Twitter. Extremely thoughtful with respect to evals. And some feedback he had on our skill was that it should indeed walk the user through the data first, which I actually agree with. I'll make it some minor modifications to the skill to do that. So I think that uh eval design, I'll go kind of go back to it.

24:56 Claude can assist you when building evals significantly. So for example, in the example he gave when his walk-through, um he was looking at like a it was like a customer support app or something. And the first question was like, okay, like it's a bunch of traces. What are the failure cases here? What should I even like promote to an eval set? So Claude's really good at kind of identifying, ooh, okay, here's case where there might be errors. It's effectively like acting as like an online evaluator, but looking at your offline logs, scanning them saying, okay, here's some problematic examples.

25:29 And Claude was very good at that in his example. Then it is actually important and quite useful if Claude then shows those raw like the raw data to you in a viewer. So I'm going to add that to the skill. I have a PR up and I'll add it. Currently, Claude goes directly to analyzing the data and like building the eval and showing it to you. I do think it's important and helpful for there to be an intermediate stage where you look at the data first. But I think having data annotated by Claude where Claude's giving you some ideas about what to look at is very important. So kind of like Claude assisted evaluation and scanning of the data is like I think like I I think a nice marriage.

26:06 Um and then once that's done, you can kind of promote like kind of examples to your eval set. So I think there is still quite a role um for the human in the loop there, but Claude can assist quite a bit. Hill climb with human feedback. Um so hill climbing only requires that you have an eval and you have some signal that Claude can use to judge good versus bad. So if you've taken human feedback and you've rolled that into an eval signal, for example, then it should be perfectly feasible to hill climb on an evaluation structure that way.

26:37 Yeah, this is a good one. So, for hill climbing, what's like the best surface to start with? So, I kind of have that here as like the thing we edit. Yeah, so so I've actually played with this quite a bit. Text is nice, in particular if things like prompts and skills are very nice because it's a very it is a very clear place that we know sign that we know significantly alters and affects model behavior. So, it's like a very constrained surface to modify, and we know it's highly coupled to the output, typically in terms of like what you're evaluating.

27:08 Uh like here's a good example. Um like when I'm doing skill hill climbing, I'm testing something that is directly reading from the skill, so I know there's clear coupling. Now, I'm going to contrast that to I've also done hill climbing with agent harnesses. But, what happens there is it can be a bit more hard to interpret. So, you're modifying the harness code, and so the the the the changes per round are much larger diffs. And um there's just more surface area for confusion. You can be making changes that are actually decoupled from the output signal you're measuring. So, it's certainly possible, and indeed people have done quite a bit

5 Highlight · The Overfitting Trap in Prompt Hill Climbing

27:40 of work hill climbing on agent harnesses. I just think it's a little bit trickier, and text-based services are often a good starting point, prompts, skills, um because they're so clearly coupled to model behavior, and they're easy to modify. The catch actually is that is that things like prompts are very easy to overfit. I've seen that in in many of my uh hill climbing efforts as well. So, like um I don't know if I have a good diagram, but um you know, you have some eval distribution you're hill climbing on, even with these regularizations in place, like you you the eval test train split and so forth. Um prompts are very easy to like wiggle Cloud's very good at wiggling the prompt into a shape that does well in your eval that isn't really representative of what's in production.

28:22 So, you have to be very careful with prompts, but they are very easy to to modify. So, it's a good starting point, you do need to be careful with overfitting. Uh let's see. Turning a prompt into a skill. So, they're kind of compliment like um maybe do you mean if you if you're like going through some

28:38 standard operating procedure, when to like promote that to a skill? Okay, yeah. Um that's actually kind of interesting. I think often I and Claude is quite good at judging that. So, basically you can basically go through a workflow and then you can ask Claude, okay, turn that into a skill. I do that all the time. So, Claude is pretty good at like catching, okay, you're doing some repetitive thing, make it a skill. Uh I actually think you should just have Claude do that. Um just tell it, turn this into a skill, and it'll do it. So, usually in the in the case of evals, the setup will be I'll have an existing skill that I've kind of defined and maybe it has some boilerplate starting text and I'll say, okay, I want you to make this skill improved. I want to improve this skill text to basically perform better on this eval. So, that's usually the way I set it up. Like I have a skill boilerplate as a starting point. I know I want this thing to be a skill. Like in this in this case, right? Like I had this skill for the Claude API. So, and it was basically you can imagine it's basically distillation of our docs. It's like

29:39 go to this page if you want to learn about this, go to this page you want to learn about this. Um it had a bunch of little gotchas and issues, and so my eval performance was pretty low to start. But I had a starting point, and that's what Claude was able to then modify. So, I think usually with skills you're kind of starting with some

Binary rubrics and pairwise comparisons ensure reliable evaluation

29:54 baseline. And it could be like a very crude default, but then give Claude something to like start from. That's usually the way I would do it. Right. So, verification in um domains that that have kind of open-ended criteria. So, like yeah, I think in general heuristic evaluators are preferable, of course. So, if you can have define like a a a deterministic check for good or bad, that's obviously preferable. So, in the maze that doesn't work. So, what I found is um well-structured rubrics that produce

6 Highlight · Designing Crisp Rubrics for LLM-as-Judge

30:30 similar uh grades. So, okay, a few different things. Build a rubric that you as a human can reason about first. So, it's like let's say we're grading a piece of like text. Is this good or bad? Build a rubric that like I could look at as a human to like look at some text and say, "Okay, I can grade that." That's point one. Point two, don't use arbitrary scales. I see this all the time like grade it from zero to five.

31:00 It's very ambiguous. Make it crisp and simple like ideally a binary classification good versus bad. You if you want to evaluate multiple things, you can set up like multiple binary classifiers rather than the worst thing is give Claude like from one to 10 rate this. That's extremely noisy. Another sanity check, test across different models. Have Sonnet, Opus, and Fable all grade using your rubric. If they all disagree, that's like a bad sign. Another test, run replicates. So, basically um like I

31:30 show here how you want low variance, run replicates with your rubric and make sure it's tight. So, if you do all those things, you can definitely make it work pretty well um for yeah, like text-based answers that are squishy. Like there is no very definitive like way to heuristically check that. Uh but LLM as judge with a good well-structured rubric can be quite effective. Yeah, that's actually a good one. So, actually I I I I should have mentioned that. Uh pairwise comparisons can actually be quite good. It's much easier for humans even models to kind of compare two artifacts than it is to grade an artifact. Like you can typically answer, "Oh, I like this webpage better than this one." versus "I'm sure this webpage grade it between zero and five." It's much harder. So, pairwise comparisons are pretty effective and actually I think that is a good that is a good way uh if if your problem allows for that, that actually can be a effective way to deal with um more ambiguous gradings. Well, I think it goes a bit back to just looking at

Build eval distributions from first principles, not model failures

32:26 the data carefully. Kind of relate to the question asked over here. Um when I say the distribution, um I more mean ensuring that um you've sampled some set of cases for your eval set. And making sure those cases represent what you actually care about in production. That's really what I mean by distribution. And so, there are some interesting tricks here. Like one problem that happens is adversarial sampling, where you select like this is maybe a little bit confusing, but I'll try to explain it clearly. You select cases that are specific failures from a from a particular model class. So, models are jagged. And so, you just look at, okay, like Sonnet 5 can't do these examples. I promote them to my eval.

33:11 Opus 5 may maybe it'll solve them all just because intelligence is jagged. And so, like a particular pocket of one model might just get solved by another. So, typically, a good recommendation is promote eval cases promote cases in eval set that are hard or interesting from first principles, from your own intuition, from looking at production data, not necessarily completely sampled from the failures of any given model. Cuz that can result in kind of this odd distribution that's shaped to the valleys of one particular like jagged model's intelligence that don't apply when you upgrade models.

33:44 So, like can you use bucket-based evals or do you have to use binary classification? I'm being a little bit overly prescriptive. You obviously can, but just as a general rule, I see people going too far in that direction. So, there'll be some very arbitrary scale between 0 and 10 with unclear grading and rules that even a very small model can't resolve. So, you obviously, if you have a well thought through scale, you obviously can do that as long as it satisfies we talked about. It has like low variance, it's reproduced across models, a human grader can kind of distinguish between these two grades, that's totally fine. I'm just saying binary is like a shortcut of like if to simplify the problem, usually. And also for grading, you often want to use smaller, cheaper models. So, you really don't want Fable doing your grading and depending on your cost sensitivity. So, um, those are just the recommendations.

34:28 But obviously, if it's well thought out, that's totally fine. You can think about using, um, yeah, like a multi uh, like a multi multi-category classification. Yeah, that's a good point. Like when when to build evals for what types of problems. Like, is it worth building it for my like personal like use cases or we only doing it for things that are kind of production grade? Um I think it depends on um I let's see. It probably depends a bit on the importance of the task. So, if it's even it's a personal task, you feel that it's quite important, then it would be perfectly reasonable to need eval for it. I think that's totally fine. I think people do that. Kind of varies by person. I personally haven't necessarily built many personal evals for like my personal automations, but um I think if it's a task that's highly important, it's in particular um I okay, I'll give a good example. Like if say people work in finance and like you've automated parts of your your personal workflow, but quality is extremely important. Like, this is a

35:29 work output that's going to share externally or something, that would absolutely need evals. Even though it's something you've built personally. Um if it's my personal assistant, that's like, you know, it's like a a meeting it's like an app that I built to help me manage my calendar. Uh I may or may not build evals. It depends upon the context. But I do think obviously if it's something that's important for your work, it's like, you know, producing deliverables that are, you know, externally viewed or doing something very sensitive task, you obviously want evals. Even if it's a personal agent.

35:57 I actually worked at LangChain before Anthropic, but I I I I can't advocate for any one or the other. Um Yeah, I I I plead the fifth on that. Um I mean, look, I've I've used many of them. Um I think actually if you're if you're working at relatively small scale, you obviously could roll your own. Like, you don't necessarily need a fancy observability system. In fact, we're going to be adding to the skill the ability to to um to to uh just like a simple data viewer or we're actually adding to the skill.

36:24 So, Claude can kind of Claude's very good at kind of building

Clustering remaining failure cases helps diagnose stuck hill climbers

36:28 you UIs and visualization surfaces. So, they can kind of roll If you have like a very bespoke problem you want to look at the data for, it can kind of roll you whatever app you want very easily. And and not very expensively, either. No, this is a key point. It's not guaranteed to converge at all. So, I've seen many cases in which the hill climber gets stuck. And actually, we built You can read the blog, but we built a fallback in the hill climber where if it gets stuck for multiple rounds, it'll actually go and basically try to cluster the the remaining failure cases, and it'll try to look for. So, for example, here's the problem.

36:59 Oftentimes, you'll run the hill climber without having run like a build eval command or with without having built a very like carefully constructed eval set. So, the hill climber will go and look at your evals and try to find are these cases ambiguous like we talked about here. Maybe that last 20% are all faulty grader examples, ambiguous ambiguous instructions. This happens all the time. This is like very common. So, climber is just stuck not because Claude can't do better, it's because the eval is poorly structured. So, the hill climbing skill will try to find that for you.

37:31 Uh so, we added that. In fact, when I did this hill climbing I talked about here, um this second stage actually went it kind of got stuck, and it went and basically clustered the eval cases and looked at, "Okay, why the failure cases? Why are they failing? What's going on?" It found a bunch of ambiguous cases. It rewrote them. And And anyway, so it'll actually do that for you, but that happens all the time. It's absolutely not guaranteed to converge. And usually it's because of failures in your eval or like kind of misconfigurations in eval.

37:58 Uh so, that that's really interesting. I You have a lot of respect for the DSPy team and and Omar and and and and and um and colleagues. Uh this this actually just is a skill built like bundled with Claude code. The skill is also open source. So, you could consider taking that skill and and like vendoring it in any other surface. See if you want to could probably work with DSPy. It's just some very simple general instructions for setting up a hill climbing loop following basically these principles.

38:26 Um so, um that [clears throat] it could be compatible with DSPy. I haven't used DSPy, so I don't know like the details and how it's structured, but um it's possible this skill guidance here could be

Prioritize stronger models at low effort over weaker models at high effort

38:39 compatible with it. If you want to like effectively implement this or run it in DSPy, the skill is open source. Uh you said choosing effort level. Yeah, this is a classic one. Effort's confusing, isn't it? Yeah. I'm also confused by effort. It's Yeah. It it is it's non-obvious. Um so um this is a so what's nice is if you have an eval uh the hill climber can absolutely try to configure effort for you.

39:13 Now, I get it. It's like not that easy to have an eval. So, like it's not a

7 Highlight · Stronger Models at Low Effort Win on Cost

39:17 it's not applicable to every problem, but if you happen to have an eval then this hill climber can search through your model effort configurations and it follows a very simple principle. The principle is just simply that typically the stronger model at lower effort is is like more effective uh or it sorry is more token efficient and therefore cheaper than like a weaker model at high effort. That's like the general principle. That actually turns out to be true a lot. So, it'll kind of search through the different models at low effort first and basically continue that search until you see a degradation performance below the baseline. So, for here I don't show it, but like when I tried Haiku for this was 4.5. Haiku we now have 5.5, which is quite a bit better. Uh 4.5 at low effort was a significant regression, so we admitted that that data point. But basically if you have an eval hill climbing is a very good way to try search for a good model effort combination.

40:05 More general effort guidance if you don't have an eval is obviously tricky and subtle and like I think um people typically um I I will say the guidance I recommend generally on effort is try stronger model low effort in general. Opus 5.5 low effort. Try that. Basically. Uh better models low effort are often more token efficient and therefore cheaper than than weaker models at high effort. That's like the key point I'll I'll leave you with.

40:32 So, not medium?

40:33 Uh try low to start. Um you can always dial back up. But basically I like very smart model not trying too hard. That's a very that's often a sweet spot for for um kind of kind of cost performance. Obviously, you can dial up effort, but I think the the counter is people often start with something like max effort. This extremely token and like token hungry.

40:59 So, it is actually wise often start like small smarter model low effort and kind of kind of go from there if you need it. Yeah, actually, we talked about this in the blog here. I was open that up. I don't have internet. That's fine. I'll just tell you then. Yeah, so it's basically telling Claude how hard to think. So, low effort saying like don't think as hard as you scale it up it's saying think harder.

41:27 So, and that's kind of why more fundamentally intelligent model at low effort can actually be more token efficient and therefore cheaper than a cheaper model in terms of base tokens at high effort cuz that that for example, Sonic 5 at high effort is spending lots and lots of tokens thinking where smarter model low effort will spend fewer tokens thinking at a higher baseline of intelligence. That's kind of the intuition.

41:52 Yeah, so the question is about setting budgets in hill climbing. So, basically when you when you run this skill and set this up, it'll ask you for a budget. So, um it will manage that for you. It'll track the budget you allocate for the hill climbing round. That can definitely happen though. And that's actually why this is a classic example. That's why designing a good evaluation is really important. If you have if you have a poorly specified evaluation, imagine for example, you have an eval where that final 20% is impossible. You burn a huge number of tokens trying to solve those final you know, 20% of problems that are effectively not solvable.

Combine human taste with Claude-driven trace analysis

42:25 Kind of an eval building, the question is about kind of human loop kind of human Claude collaboration. So, I think I think that um humans still have the role to kind of steer the goal. Like what are we actually trying to evaluate and why is it important? So, it's kind of taste is important. So, like what are you actually trying to see answer? For example, I'll give you I'll take an example from like Hamel's public kind of ant or a public dog fooding of our um build eval skill. So, it's like okay, it's a customer support app. So, let's say that's your application. What matters? What's important? What do you care about? Those are things that a human can guide Claude. Now, once those things are clearly specified, Claude um can look at like raw traces and determine effectively like okay, where is the app likely falling short of my goals? So, that's I think kind of nice collaboration you can have in place.

43:22 And and and that's actually when like kind of looking at the data is very helpful. So, Claude can kind of help to steer you okay, I've looked at the data. Here's a bunch of cases that look problematic. And that's that's just like having an online evaluator. So, you like imagine you have a tracing system, you have an online evaluator. It has some like grading for like customer satisfaction. It's flagging like here's problematic traces for you. The same kind of thing. Like you're telling Claude, you're telling online evaluator, here's what I care about, here's what I want you check. It can look at the data, it can elevate a bunch of cases that look like these look problematic. Then given those cases, you can like review them, understand them, and then say okay, yes, build me an eval based upon these. That's typically a very nice marriage between like Claude and human when it comes to building evals.

44:03 Yeah, question is like who's responsible for building evals? I mean, it depends on the problem, it depends upon I mean again throw back at everyone ever everyone is empowered to build evals. So, it's not really delineated per team. It's typically based on ownership like of the product and um I think in particular when you have skills like this, it make it easier to build evals to help climb, it really opens up the number of people that can feasibly kind of build and design evals, which is actually quite nice. So, we've seen like Anthropic many many many different types of roles build evaluations. Um because Claude's so involved and then has can help so much.

44:36 Um much of the tedium of like um you know, building an eval runner and stuff, that's like Claude can handle all that. So, really it's about what we're talking about before, the human taste guiding the process. What do I care about? And that's very that many different types of roles can can kind of serve that guidance or provide that steering. Um let's see if I have a good one. Uh Right. Um Uh, off top of my head. Um I've seen Okay, I have a good one. I have seen um evals for generative UIs.

45:14 Um, so that's kind of a good example. Uh, actually I think in that case it was a pairwise judge back to the point that you were making. Like so it's basically comparing like configuration A B, um what is better um between like two different generative UIs and like the greater would would do kind of pairwise comparison. I think then those results were rolled across many trials and scored with something like an Elo score. So that's an example of like generative UI with a pairwise evaluator I I've seen before. There's many other examples I can't think of at the top of my head. Um, but people use LM as judge all the time. Um, for like retrieval tasks or like writing tasks.

45:55 Um Uh Look, categorizing traces, looking at traces and saying oh this is bad or good. So like it's extremely common. Yeah. So okay, that a lot of the motivation for the hill climbing work is kind of it is kind of flowing from auto research. Auto research is motivated heavily by the fact that models are getting very very good. They're very good at self-correction. If you give them a well-structured problem uh they can hill climb on. So I think Karpathy's case is a good example of I think I can't remember the precise problem. I think that general setup work with many different formulations. Like it could be um you know, like drive down loss or something. Um, so again there's like a very clear verifiable objective and clock can look at the objective and use that as a guide to hill climb. Um, I have Yeah, I didn't show it here but I have some other examples doing auto research um in this kind of hill climbing process and it would it would this kind of um This command in particular is actually designed more for eval-based hill climbing where you have like an eval set

46:55 with a with a split. Auto research was looking more like you build an artifact and like grade it, like look at the loss or something, but the the setup could probably very be very easily kind of modified to work with auto research with the very same intuition. The point is that models are good at autonomous work over long horizons when given a clear um a clear objective or verifiable that that they can reward that they can kind of measure and use to inform what their next move is, they're very very good at autonomously solving problems.

47:24 So, it's driven by the same intuition as auto research.

47:28 hill climb and then extend into auto research.

47:31 Uh the the methodology for hill climb would So, it's just a it's just a question of a problem formulation. This just to be clear. So, like the auto research from formulation was um was initially focused more on like on like designing, for example, an a like a network with, for example, um solving some problem where you have like a a verifiable objective like loss. That's really it. The the hill climbing process though was like very similar where like Claude runs some experiments. It measures loss. It looks at the results and then adapts, does something else.

48:01 So, it's the same kind of hill climbing loop, but like in that particular case it's just applied to a slightly different domain. So, it's like you could probably take this skill and basically say, "Okay, set this up for if you had like like a the nano GPT repo, you could use this skill with it, I'm sure." It would Claude would figure out how to adapt things. Just If you're interested in that, just just like try it. But the the core process is the same. The core intuition is also the same that like models are very very effective at using um at self-correction and hill climbing.

48:30 That's the key intuition. Yep. Yeah, I kind of try to do that in like this example where I'm looking at cost and um performance. So, you're like kind of jointly optimizing across those two. And yeah, you can just tell Claude the multi-objective. And usually it will just kind of measure multiple objectives and and take that into account. Um that's certainly feasible. And this is a good example of that.

48:55 Have you checked like what's like the max you like

48:58 Uh I mean, yeah, I haven't pushed that far. I think but I've typically looked at as cost and performance. And like speed's another dimension. So, you like cost performance speed. That feels like a pretty reasonable like trifecta of things that you would like kind of jointly measure. Um, yeah, I mean, I think that um, as you kind of scale up, it just hobbles the process of hill climbing a lot cuz there's lots of different objectives that Claus is trying to balance when making these changes. So, I think it'll just it'll make the problem harder, but

Retire legacy prompt anti-patterns for frontier models

49:24 like it is doable. Maybe you could think about kind of changing one thing at once. Doing doing them serially. Yeah, that's it that's a very good question. Uh, so the prompt on this stuff, which actually of all the things I talked about is the simplest thing to try and often is like extremely it's like do the simple thing first. It's very simple and often very effective. Uh, this general the general guidance in the skill applies to um, the the model delineation's a little bit on like it's not precise. These are well-known anti-patterns that definitively frontier models like the 55 series don't need anymore. How far back do you need to go until these patterns are still like appropriate? It's like not quite clear. I've tried to do some testing. It's maybe like Opus like 45 is the point at which these things start to like become less relevant. So, it's a little bit squishy. I don't know like precisely the model at which like you need to remove these, but my view is basically when you're doing a migration to the latest to like to like Opus 55, just do this. Like definitively like the latest

50:26 frontier models don't need these patterns. Yeah, yeah, okay, good point. So, the skill does have some model specific caveats. So, the skill takes that into account a little bit. If you're for example going to like Haiku, I get it. Like maybe Haiku like um, isn't quite as autonomous. Maybe it does need slightly more boosters. I don't know if that's true. I'm just kind of yeah, kind of kind of suggesting something. Um, it it'll take that into account. So, it does have some model specific guidance built in. It's like smart to figure out like okay, here's the model you're working with and here's the anti-patterns that we know are are not needed anymore. Um, but it's it's a good call out.

51:00 Yeah, uh, uh,

51:04 Yeah, um it it is not a Yeah, I I should I should get back to to like to be really clear about this. Um it is not a hard cap. It is basically telling Claude emphasizing how much thinking to do to solve the problem. So, like for example, even if you're on very high effort, you give it a simple problem that won't just arbitrarily spend huge amounts of tokens verifying it. There's some Claude has some thoughtfulness about like what problems require, um you know, all this extra thinking. So, like um And and I don't know if I have any evals that show that. Uh yeah, maybe not. Um but it is suggested thinking. So, that's the kind of way I think it's basically telling Claude how hard to think about a problem. The problem comes up when you have like a harder problem, like a frontier coding problem. That's where at for example hot extra high effort or max effort, you can see huge differences in token usages from like medium or low without much difference in actual outcome because Claude will do more work to verify the result for example. So, I

52:05 think out in the frontier regime of problems, that's where these effort differences can really result in like big big token spend differences. And that's where we need to be more careful. Yeah, I mean, this um this is So, if you kind of think about what's happening, um it's not dissimilar to to training. And basically you're giving a model some some reward or objective, and it it is indeed trying to make modifications to the system to improve the objective.

52:34 Um so, yeah, philosophically that is similar. And in fact, even where we're used, you know, we employ some simple principles like test train splits that are often employed like at training time. But the nice thing is these are kind of run time modifications you can make. So, this is basically a way to employ some of those principles, but basically like once a model has been deployed and like you're modifying services like prompts or skills, um you can employ some of those same principles that we use in training, um but like in your particular domain.

53:02 Uh that's kind of the the big idea. Yeah, so like is Claude aware this being evaled?

53:07 Um, right, exactly. That I think that's something we've published on a fair amount. That is a concern um, in the that that is certainly a concern kind of in in training uh, in different stages of training, absolutely. What we're talking about here um, I I do think um, it could be somewhat of a concern. I think the bigger concern than what we're talking about here is typically more overfitting. So, Claude may know it's kind of being evaluated. Um, but it will find ways to hill climb the objective based upon what's been provided in kind of this eval distribution that has. The classic example being it'll find um, kind of prompt tricks and tuning that aren't applicable to production that indeed improve performance significantly on your eval. So, I've seen that happen that happens all the time. And it's um, it's not necessarily that Claude's like aware it's being evaled and saying like

54:07 it's it's kind of some um, kind of unexpected behavior. It's more that like okay, you've given it a prompt and Claude can find ways to shape this prompt to like make number go up and it'll just do that. Um, that's why we use things like test train splits and try to regularize. So, I think overfitting is more of the practical concern that we're going to think about here. Um, but it is true that eval awareness is like a broader phenomenon that we we have discussed in in some in some of research blogs. Yeah, that's a good point. Um, kind of what's like the minimum kind of data set that's that's useful.

54:38 What is like the split between test and train? Uh, I haven't looked at that in depth. Um, frankly, just ask Claude. Like um, when you set this up and you run this command, Claude has some heuristics to reason about the size of the eval set that's actually helpful given the signal or like what you're trying to measure. Claude can kind of reason about like how many examples you should have. And then there's some guidance in the skill for this. So, it's something you can kind of uh, work with Claude to kind of con- tune. But I haven't looked at like spoop or super specifically like uh, different like eval set sizes and like um I think the central concern I would have is more um it is probably overfitting and um and like variance. So, if you have a very very small number of samples, like okay, is that really representative of what you care about? Um uh it's a lot easier to overfit. So, like they may not be out of distribution for your production like what you actually care about. Um they may be easier to overfit. Uh they may be noisier. Um so, anyway, like I think those are the problems that arise when you're in like the very very small

55:38 sample regime. Yeah, okay. So, the question is about hill climbing on tasks that are like very very long duration. So, kind of that's funny cuz that's actually why I mentioned I typically do like simple surfaces like prompts because they're like very very quick to change. Uh you are right though. There's some cases you're in a regime where like your changes could be very very long duration. That effectively makes hill climbing much harder cuz like it's a much longer duration task then. So, it's a lot longer time. That's actually where that gets into our final point about kind of harness design. If you're going to do that If you're doing like very very long duration hill climbing, you actually need harnesses that can like remain reliable for like many many hours to days. And so, we have we have that obviously internally. We use things like manage agents.

56:19 Um I think it just becomes a bigger infrastructure lift when you're doing that. It can be done. Um it obviously is harder because of what I just mentioned, infrastructure issues, failures, um and just like a lot longer time to see your signal. Uh single turn versus multi-turn evals. Um I mean, yeah. When you're building agents, you do typically need multi-turn evals. Um Claude can help you do that. Particularly like the build eval command, it can look at traces. It can If you give it a bunch of traces, it can help you promote um it can help you promote particular traces to your eval set. Um in terms of grading, um I mean, yeah, there's a lot of different philosophies on this. You can have evaluators looking at the entire trace trajectory. You can have evaluators looking at like the final agent outcome.

57:04 Uh there's a lot of different philosophies. The build eval command has a lot of guidance. So, like if you have a particular like agent you're interested in, if you basically give this command access to the traces, it

Decouple the context object from the harness for robust long-term agents

57:13 should be pretty pretty effective at kind of shaping and designing the eval for you. And if you can tell it I want multi-turn and here's what I'm considering. Um cool. Interesting. So like for context engineering, what promotes of reliability a lot versus what's like overhyped as a technique? Um Uh let's see.

8 Highlight · Decoupling Brain and Hands for Context Reliability

57:33 Um So okay, the key thing I think and I've actually I've mentioned this in the past, we mentioned in the paper too. I think the recursive language model work in RLM idea is quite good. Um and we actually employ that idea here. I think a very important thing in terms of context engineering reliability is decoupling brain and hands. So basically designing your system such that it um the con- the key point is this, the context object is independent from the harness.

58:02 So the harness is basically your cloud can effectively query this context object that lives independently. That's beneficial for all these different reasons. One, the context is always recoverable. So typically imagine alternatively you just have like, you know, a message history, you do a compaction event and everything you compacted is gone. You can never recover it if you haven't like backed it up in the way like this. That's why I actually like this architecture a lot for reliability in terms of context engineering. It does exactly what the RLM work advocated. Like it's basically splitting the context object from the model and the context object comes linked the model can interrogate through different mechanisms. And that's exactly what we do here.

58:38 So interrogating that session log is basically done by the harness through different tools. Um and that's a very nice architecture I think. So I think that's like one of the tricks that I I found to be very nice for reliability, recoverability, um

58:51 and in particular building like long duration agents, it's hard to know what will matter in the future. So when you do a compaction, um you don't necessarily know if you need to like go back and interrogate that context that you're summarizing. And when you have it backed up in an object like this, you can always do that. So I think that's a trick that I I I a lot. Uh and the RLM work was was very nice in that direction. Yep.

59:17 Yeah. Well, I think [clears throat] that's why, you know, we spend a lot of time on um, good eval design. Um, yeah, these models, um, are non-deterministic. Um, in many cases the operating regimes where outcomes are not easily verifiable, like not everything is a coding problem. Many cases you want agents to do things that are more judgment-based, more free from open-ended outputs. In those regimes, you have to be very smart about your eval design so that you're giving Claude an objective that I can climb. And I think all the things we talked about in terms of like kind of verifier design, like good objectives, clear grading scales, multiple models can validate the result, low variance, all are like super important when designing like the problem that you want Claude to go and solve. So, I think, as kind of mentioned it in this talk, when it comes to unhambling frontier models, I think spending a lot of time up front in a very very good eval design is a way to make sure that you're getting the most

1:00:18 out of the model because if you said tell the model, "Okay, optimize performance on this like very poorly structured evaluation." It won't get anywhere. And it's not the model's fault because you haven't thought through the eval, it's because cases are impossible or ambiguous. So, I think part of the message here is to just be thoughtful and spend a lot of time up front in the eval design such that you kind of set up frontier models for success and effectively unhamble them. That was kind of the the namesake of the talk.

1:00:45 Maybe, um, I we're probably at time and everyone's probably tired. I was at a

Continuously feed production failures into offline regression loops

1:00:49 wedding last night, so I'm very tired. That was a funny timing, but it's very nice to be here, of course. And actually, it's been uh, it's been a lot of fun, very good questions. Um, maybe one more question over here. Mm. Yeah, so I think that's a good question. It's about like, yeah, models are getting better all the time, so like, how do I modify my evals to keep up? And I think simply it should be the case that as models get better, you should see performance on eval going up like we see in this diagram. If you don't see that, then I would go back and question the eval was structured. Um it is true like Yeah, we see this all the time. Models are saturating eval very quickly. So, we do need the development of new and harder evals that like keep up with the material. And actually, that's a good point. It's actually pretty hard. It's hard to come up with evals that really challenge like the best models today, like the Fables Opus 5.5 even.

1:01:39 Um so, I do think it's a it's a it's a it's a recurring challenge um to design good evals that test the capabilities of the best models. It's absolutely a challenge. Um but I do think so, if you have evals that you built sometime ago, you should be seeing this trend on them. So, as models get better, you should be seeing performance going up and up and up. If you don't, I would question things about the eval itself. That's kind of the key point I would make.

1:02:06 I mean, I've actually seen a lot of interesting work on this from the various trace and companies. So, like the the the like workflow I've seen quite a bit of is like you have agents that are deployed, they're like producing traces, you have an online eval you're looking at those traces and scoring them. Score Trace that score poorly are like then pulled into some offline eval set, you kind of then hill climb on that. And then that loop continues. That's a good That's like a very good pattern, I think. It's very reasonable. Uh and so, what's happening there is you're typically like hill climbing on these cases that you're failing on in production.

1:02:41 And then you're like, you know, putting those in regression tests once they all pass, and that like continues until in production you're seeing very few failures. So, that actually is a good general way to structure your kind of eval loop and in many of the, you know, the LangChains, Rises, Braintrusts, all these companies kind of have product offerings that offer this kind of this like outer loop, which does seem quite quite effective.

1:03:08 You do. So, the challenge there becomes okay, well, what are your online evaluators? So, like okay, how do you know what good is in production and what are you promoting to your offline eval sets? And Um, yeah, it's not easy at all. Um, I think that is Yeah, that's like there's like a lot of product work in that particular domain now. As far as I haven't followed that area too closely. Um, like what all the latest um, kind of updates that the various like tracing observability companies are working on, but my understanding is that it's like a very very active area of like product development among that class of companies.

1:03:43 So, that's pretty interesting. Uh, okay, with that maybe we'll let everyone go. And um, very nice to see you all. So, feel free to reach out to me anywhere and thanks for the time.