Anthropic's Secret to Better AI Coding Agents

10,036 words · about 40 min to read

Introduction to Cloud Code and agent customization

0:00 Um, my name's Daisy. I work at Enthropic. Um, I work on this, uh, side project that turned into something big, uh, called Cloud Code. Um, I don't know how who in here has used Cloud Code before. Holy crap, that's a lot of people. Okay. Um, that's really exciting. But I will get to go through some of the early slides first. um uh a little fast and we'll get to some of the the bigger later slides. That's that's really cool.

0:34 Um yeah uh I got the amazing privilege of joining anthropic right before cloud code went out. It was actually a a internal product called Claude CLI um that I first contributed to and literally the day after I contributed my first PR the uh the um name was changed to cloud code which I think was a much better name for marketing than cloud CLI even though us silly engineers thought CLI was going to be a much better name.

1:06 Um uh in that time I got to lead the design and implementation of plugins and agent teams. uh a few other small features here and there but uh plugins especially have been my passion in terms of both in terms of building community and building customizability. I'm really interested in fullscale software engineering not just agentic programming and I think I hope you'll understand the difference between those two at the end but I I really want people to be able to do their jobs with this and not just build cute prototypes.

1:37 Um my background is in programming languages. Uh I was involved in the C++ standards committee for almost a decade. I was a chair for a couple years. Um I've been very deep in the trenches of uh difficult programming. I used to work on uh parallel and distributed programming models for supercomputers. So trying to figure out ways to make it easier for people to write code that needed to scale up to an entire data center. Um, and now I'm working on cloud code for kind of the same reasons. I think it's uh I want non-technical people and semi-technical people to be able to build real production uh software. And I I think that the this talk is more about figuring out ways to enable that for people around you. Um software engineers more and more are going to be about software engineering more and more is going to be about enabling that. Um, so yeah, where we're

2:37 headed here. Um, well, I have a section that is about getting on the same page about what agentic programming even is, but I guess everyone in here has used cloud code. So that is um, we'll see how that section goes. If I get a lot of nods, I'll go really fast and I'll spend more time on the the later stuff. Um, we're going to talk about customization of cloud code. Like why do you need to do it? um how the context window is the kind of the basic primitive that you are working with when you do this customization. Then we're going to dig into how plugins are created in cloud code, what they're composed of, how do you build out this knowledge base in your monor repo or in your real largecale software engineering environment that um enables Claude to act as a software engineer. And I'll talk through a few of the ways that we actually run cloud code uh within Enthropic and and uh even just on the

3:38 cloud code team and and kind of what's

The evolution from chatbots to autonomous coding agents

3:40 coming next. So yeah, let's let's start like a little bit basic. Um some of you may understand this. I promise you it's going to get more technical, but uh like what is the difference between a chatbot and an agent? Uh so when we first started to get LLMs working, we were building chat bots with them and it really the interaction looked something like this, right? It was the human turn and then there's the model says something and then you go back to the human turn and then the model says something and you go back to the human turn. In 2024, we came up with this thing called um uh tool calling um which allows the the model to basically just ask the computer to do something. I'll go through the specifics of what that looks like, but probably most of you have seen this happen. Um and at first it was just kind of like the model could go do something, could go get the output, and then could make a decision based on that and then go back to the human. And kind of we slowly

4:41 started adding more and more tools like it could do a second tool call based on the first tool call results and then it could do a third one and at some point we crossed over a boundary where we decided we needed a new name for the thing and starting call started calling them agents. I think this is the easiest way like I don't like the debate of what's an agent um or the discussion. It's a very buzzwordy thing, but this is basically what led to the term agent is that you went from no tool calls to one tool call to a few tool tool calls to at some point it's got enough autonomy that we're going to call it an agent. Um and uh coding agents or agentic programming harnesses or whatever term you want to use for them um like cloud code, like codecs, like cursor um are basically agents where the tools we gave them are like normal programming tools uh shell commands, file edits, uh CI related tools, the ability to run

5:43 code, the ability to compile code, right? Um, I mean, all of those kind of fall into shell commands, but it's like we're basically giving the agent the ability to do things that a programmer can do.

1 Highlight · The surprising simplicity of agent tool loops

5:53 And I I'm saying programmer, not software engineer. We're going to talk about what the difference is there or the gap is there. But, um, tool calls are shockingly simple. When I first started getting onboarded into this whole AI world um, from C++ world, I was kind of shocked at how like primitive it all is. It's like literally you tell the model that it can write JSON and something will happen when it writes a certain JSON and then you give back the results of doing the something to the model and stick it in the context window right after that. And that's like that's the entire thing, right? This is how the model runs something in bash is it says that the tool it wants to use is bash.

6:41 Here's the command. Then the harness goes and runs this the the command and then puts the output in a tool result block, right? Um it's it's like super easy to get your mind around and and everything in agent agentic harnesses is built around this idea. Um, and the ability to like act on this output and then create another tool use that is based on it and then continue that loop is what an agent is and everything is built around that, right? Um, the big workhorse that really makes uh coding agents work is the edit tool. Um, and

The mechanics and scaling trends of agentic edit tools

7:24 the edit tool this is literally the the schema for the edit tool that is in cloud code. um is uh so quad by the way cla code made all of my slides. So when I wanted this, I said, "Hey, can you put like a tool call from your system prompt into here?" Um, it Yeah. So, um, it has to give the old string, gives the file name, the old string, and the new string. And it's like a Yeah. It's a find and replace. Uh, there's no cursor, no selection, no quoting. Old string has to match bite for bite. if there's multiples of a old string, it either has to say up front that I expect there to be multiples or it just fails. Um, these are like really really primitive tools and I think there's a lot of iterating to be done in that space that I'll talk about later. But I think um, yeah, just to give you some idea of where we're coming from. Um, one of my favorite

2 Highlight · High reliability of strict string matching in edits

8:22 things to do with this is since Claude made these slides, the tool call to make the previous slide looked like this. um where it is writing JSON that contains some JSON that references an old slide. The models are shockingly good at this. Like not making a single token mistake in this entire thing is pretty shocking to me at least. It the the mental model that I have about how agents work and what they're good at and what they're bad at really still doesn't even jive with this. I'm like that is not something a human could do. character for character if they had to. So like this gives you some nuance into like where the agent what the agent is doing, what the agent might be good at, what it might not be good at. It had no trouble producing this. No errors, no multiple attempts. Probably the last time I saw a string replace error or an edit tool error was something like I don't know

9:22 last February of 2025 or maybe April of 2025. The models have progressed a lot past that because this is an easy thing to

9:33 train. Um yeah. So um and then you have these increasingly like because it can handle this complexity, right? You can you can make the tool calls increasingly complex. You can make it be able to do more and more complex things and that leads to these longer time horizon tasks being possible. And there's this this chart from um an AI research organization called Meter that some of you who's seen this before? Give me some idea. Okay, a few people. um as to how it's it's been called the the uh Moore's law of agentic I don't know the Moors law of agents or something is that um roughly speaking the time horizon for the task that models can do uh with a 50% success rate is doubling about every four months. Um you'll see that this starts this kind of stops at the beginning of this year.

10:33 We're now two model versions later, Opus 47 and 48. And it's because the um the institute actually decided that there's kind of the error bars are too big as to what counts as a 16-hour task at this point. And so this this chart is almost no longer useful, but it's been a pretty steady trend um over the past uh three years. I don't know about putting a 2019 uh dot here, but 2022 through 2026. This gives you an idea of where we're headed and what the trend looks like. Um, and so like you can start by assuming that this is going to plateau at some point. It it will, but like I think that if you had back in the 80s or 70s bet on Moors law uh plateauing, you'd be in a very different spot than if you hadn't, right? So I think it is worth considering the possibility at least that this trend continues at least for another year or two. Um and you know

11:34 exponential trends are pretty crazy in terms of the way that they change how we work. Um and this is starting to show up in in hockey stick like charts. This is uh from April. Mozilla uh Foundation put out a chart of the um security vulnerabilities, security bug fixes that they shipped in the month of April with uh with access to our latest model. Um and it was literally more than they had had fixed in the past 15 months combined. And that's keep in mind, right? Like it's not like they didn't have AI tools in the past 15 months.

12:17 This is like the difference between the most recent AI tools and the ones that they had in the past 15 months. Um, and yeah, um, this gives you some idea of the practical uses here, right? This is these are this is fixing real bugs in software. And I don't know about you, like fixing bugs in software is a use is useful work, I would say. So when somebody tells me, "Oh, I don't know that agents can do useful work." I like to point them to something like this.

12:47 It's like, "What if you never had to fix a bug in your software ever again, right? Like I don't know anyone who likes recreational debugging." Um, I like making things, right? And debugging is a thing that ended up having to do because I like making things. So like for the rest of this talk, I'd like to consider the hypothetical of what does engineering look like if and when agents can write better code than you, right? Um like and keep in mind I am a person who used to give talks on like C++ metrogramming minutia, right? Like I was so proud of how good of a coder I was. Um, and the past year and a half at Enthropic has really convinced me that uh it doesn't matter.

13:38 Um, and I'm not going to necessarily sit here and try and convince you that this hypothetical will become a reality. Totally valid opinion if you don't think that it's worth uh contemplating. But but if you do, like I'd like to build on that hypothesis. What does agentic programming look like? What does harness design look like? What does context engineering look like in a way that we can build this into something that can really manage monor repo sized uh software projects?

14:08 Um cloud code just happens to be the one I build in my day job. C-Ilot cursor codeex all all of this information should transfer. Like I am not here I'm an engineer. I'm not here to give an advertisement. I'm here to give an academic talk about the discipline of agentic software engineering. Um, so if you if you hear me say Claude a bunch of times meaning like an LLM, I'm sorry.

14:35 That's just because I'm used to saying that in my day job, but I don't actually mean that, right? I really am interested in the direction that software

Why agent customization requires accessing real work environments

14:42 engineering is headed. Um, okay. So, let's talk about why do you even need to customize the model in the first place? Right? If we are headed towards AGI, quote unquote, it's generally intelligent. Why do I need to tell it anything at all? Um, and there's there's three subpoints here. There's it needs to have access to the things it needs to access to do its job. It needs to know about all of the institutional knowledge that's available to you or to any other employee of your company. and it has to have the tooling to get all of that information into the context window correctly. So, we're going to talk through what that looks like. But the the fundamental thesis that I'd love for you to walk away with from here is that if Claude can't do everything you can do, it can't do your job with you, right? Um, out of the box, it really only sees like a repo and a shell. And like we don't even let it go out of the folder that

15:43 you started in by default, right? And there are good security reasons for that. But like I don't know anyone who's a professional software engineer who's like I am going to spend my entire day without looking at any information anywhere that's not in this folder. Um it's is kind of fine for zero to one projects. I think this is part of the reason why people see so much more success with like rapid prototypes and zero to one projects. Like part of it is that that is really easier, right? Like even if you're a regular software engineer, that's that's a human software engineer that that is an easier thing to do. But part of it is that like the constrained context, the the the boundaries we put it in uh negatively impact the agent even more. uh it's rarely enough to do high quality software engineering at really large scales and um I mean another thing about this that I don't think people think about enough is the fact that most professional software engineering doesn't live in the source code right

16:45 source code is about programming software engineering is about much much more than just programming right um so like giving claude or giving your model access to the things you have access to in order to do your job will help it do your job or do your job with you better. Um like we are really not about trying to replace ourselves necessarily. We are about trying to multiply ourselves, right? And there's some subtlety there that in terms of like how scary the impact is on society that I won't go into, but like um for the sake of an academic talk, let's talk about like I think every one of us would like to have five of us at our company and would feel very good about the um productivity that you get out of that. So where does your work actually live? Um like where are most of the decisions made? Right? the decisions aren't actually made in the source code unless you have a very

17:46 different uh software engineering environment than the one I'm used to at least, right? You you have a team chat, you have you have CI where you find out whether or not things work, you have dashboards that show you what's going on in production. Uh you have internal docs, design docs, things like that. the the tip that I give people and it's like actually a an experiment worth trying. Although like depending on how much time you have for this kind of thing, try doing a full full day of work without leaving the um without leaving the terminal, right? And if if you can't do your full job, then Claude can't do your job with you, right? Uh, if you get a Slack message and you can't respond to that Slack message by telling Claude what to say for you, then Claude can't answer that Slack message for you or Claude can't act on the information in that thread, right? Um, if you have a CI failure and you're like copy pasting the

18:47 CI failure into the prompt, that that's something you're doing for it and it can't do for you anymore. Um, yeah. So uh the second aspect of this is is knowledge in general. Um and when I say knowledge I there is some of some aspects of this that are like oh this just wasn't in the training data. Um, but I think people tend to over access people tend to overindex on wasn't in the training data and and the reality is that there are things that are just like can't be trained into the model because different companies do them differently, right? Like your codebase's conventions are something that the model needs to know about but can't be trained into the model because different companies do different things different ways. your institutional memory, things that your team tried two quarters ago and never saw the light of day. Even if we had a perfectly trained model, we would need a

19:47 way for the model to access this information. It's not even public, right? Um things that changed last week, right? Models have a training cutoff. um things that are just yours, internal APIs, internal vocabulary, definitions for terms that your codebase uses and that your design docs use and how to find those definitions. Um so customization is knowledge customization is is closing this gap between what the model knows inherently and what your team knows.

20:22 Um the fancy term for this if you want to sound fancy is called in context learning which is a really fancy term that's uh basically means text files. Um, so the model weights uh are frozen at the point at which we release the model, right? You're not going to be changing those. And especially recently with the windown of fine-tuning interfaces and fine-tuning APIs, there's just less and less fine-tuning out there, almost all of the customization that you're going to do happens in text space, which is actually really cool if you think about it, right? Because like programmers know how to control text, right? And so this gives you a lot of control over how you can uh improve the model without having to know anything about how LLMs work in terms of like weights, right? You don't need to know any of the math or anything like that. You know how to control text.

21:16 You can put text into the model and you get different output, right? Um it's not deterministic, but you can improve on the output of the model based on what you put into it. Um yeah.

Tighter feedback loops beat smarter models for codebase performance

21:30 uh tooling. So, we already talked about what tool calls look like, but I think many of you will have noticed that this is like incredibly primitive, right? It's like literally Claude is roughly using Who here has used ED, the Unix utility ED to like edit anything before? Who here knows what it is?

21:56 Okay. Um, everyone I assume knows VI, right? If if the the the the bottom line of VI, if that was the entire editor, that's ED. So if you had to do the entirety of your editing by like find and replace that that is roughly where agentic coding this is roughly where agentic tooling is uh today, right? And no one has really figured out what does an IDE for quad look like? What does real time feedback for the model look like, right?

22:32 The gap between ED or VI like you know raw VI and VS code is the opportunity here and that's that's all happening in text space which is like very exciting, right? We're not like feeding things back into the weights in order to get the model to stop making mistakes while it's programming. we are actually just feeding text back into the model. So the the example I like to give is like what what do red squigglies look like for agents, right? How does a agent know that it's done something wrong in a tool call? Well, how do humans know that they've edited something wrong? Right?

23:09 If you if this variable doesn't exist um or isn't valid to reference in this context, right? You get a red squiggly,

3 Highlight · Using post-tool hooks for immediate error feedback

23:19 but obviously models get raw text. they don't get rich text like this, right? Um, and they don't get feedback as they type. So, we have to add that. Um, the mechanism we have for that in claude code is a called a post tool use hook, which is when Claude does the tool tool call that we showed earlier. As part of the the tool response block, we will tack on some information that you give us, right? And that information can look like this. But this is already something you know how to produce, right? Like your codebase already knows how to produce this because uh you used to produce it in order to like mouse over this red squiggly, right? That text exists, right? It's it's a nudge right at the moment of m the mistake, not when the model goes back and compiles later and has to think about, oh, what did I actually mean there? Using up more tokens right?

24:17 Um, and there's lots of things you can put here, right? Type checking, uh, linting, uh, you can flag things that you would you were trying to get it to do in your cloud. MD, but it it didn't do. We're going to talk about how the mechanisms that we provide for for this later, but really the fastest way to make an agent better at your codebase isn't necessarily a smarter model. It's a tighter feedback loop, right? it's telling it that it's made a mistake sooner rather than later. Most of these are scripts you already have.

24:51 Um, another perspective on this is that uh for those of you who've worked in developer tooling before or developer experience before, there really are two kinds of tools. There's ones that kind of compensate for a lack of intelligence and there's ones that scale up with with intelligence. This is for humans or for models or for whatever, right? You can give a junior engineer something that stops them from editing certain files or stops them from changing certain things or running certain things and at some point they will become senior enough that you trust them to do something right and you have to give them a new set of tools for that. But if you can give if you can have a tool that gives a nudge or that gives a reminder that scales across because you know senior engineers still make these mistakes from time to time but they also know when it's not a mistake right so these like you want to focus on being well being AGI pill sorry loaded

25:54 buzzword but um it really is focusing on building the second kind of tools for your agents right thinking thinking about not just what's going to happen, what's going to help the model now, but also what's going to help the model two

Managing the fixed constraints of the context window

26:07 generations from now when it is better or stronger in some way. Um, okay. The context window is a box. The context window is this space that you have to do this customization, right? The context window is where you can put your text. Um, over the past year, the capabilities of models have kind of exploded. We went from like autocomplete, fancy autocomplete to like kind of longtime horizon autonomous work. But like the context window sizes have not changed.

26:47 Who's comfortable with the idea of a context window? Okay, that's I think that's most people. um I will go through it in more detail but it is just like the set of tokens that a model can see in order to predict the next token right and that's limited fixedsized thing um and context windows really haven't changed that much like the first 1 million token context windows were like late 2024 um and the state-of-the-art Frontier models were 1 million token models in February of 2025 And right now the state-of-the-art frontier models are roughly 1 million token context windows, right? That hasn't scaled up nearly as quickly as the capabilities of the model. Um this means that as the tasks grow, you have to get smarter and smarter and smarter with what you actually put into this context window. Um your entire like we figured out very early on that you can't

27:47 just throw the entire codebase in there, right? But like documentation and um uh yeah internal codebase and reminders and all these other things you have to cram this all into the context window. So um it's becoming a full-time engineering discipline. And I think there's definitely an aversion to the title context engineer that I see around a lot from uh software engineers.

28:19 I kind of want to assert that these are the same thing, right? As uh agents get better and better at writing software, doing real software engineering, uh teaching them how to do it is going to become the main discipline of software engineering. Um and figuring out how to get good at that. Secretly, it's always been the main discipline of software engineering. It's just just that you had to get to a senior enough level where your full-time job was making sure that more junior engineers can do their jobs. Now, it's just that they're all agents, right? And it you have to be able to do it at every single level.

28:57 Um so, yeah, the context window is where we have to put everything that contributes to the prediction of a of the next token, right? Everything that um Claude can think about in a given token prediction or when it's creating a tool call when it's making an edit has to fit into this space. So you have like your system prompt your tool definitions audited MD skills any files you've read any tool results all of that. And the more stuff you put up here up front the less room you have to do actual tasks um further back. So every customization competes with the space to do work and your whole codebase doesn't fit. Um and and it you can't just kind of naively dump everything into the box or you can but like that's not going to perform nearly as well as very carefully selecting the relevant context.

29:52 Um the analogy I like to use for this is that it is like running npm on an Arduino, right? When you are running a package manager on your computer, you can do all kinds of things to like deal with traditional um management problems, right? If you have diamond diagram or sorry, diamond problems, diamond dependency problems in in PM, you can literally just import two different versions of the library to resolve it. You can't do that when you're customizing a context, right? you have to decide what is the more relevant piece of information intelligently and put that into the context and leave out the things that aren't relevant. Um, so the the customization really does have to be don't pay for what what you don't use. Zero overhead principle um or you're going to run out of tokens very very quickly.

30:48 There's one more constraint that I

The KV cache constraint complicates caching strategies

30:50 actually really want to go over um because I don't see this in a lot of talks about this kind of thing and that is the KV cache. Um the way that next token prediction works in modern LLMs. Um this will probably change at some point but I it's been about to change for a year and a half now or two years now and I still haven't seen a lot of progress on the problem space. um it's a lot lot cheaper if all of these uh tokens are the same as the previous time you try to predict these tokens, right?

31:28 So like in computing in general when you're looking at the shape of problem we've been talking about so far, right? Where you have relevant information and less relevant information and a really confined space to to put all of that into. You do something that looks like an LRU cache, right? This is how CPU caches work. This is how uh web caches work. Right? We we we kind of this space has been explored. But the problem is that because of this constraint where like um all of the things before your token have to be the same in order to predict the next token.

32:08 um like this prediction costs 10 times more than this prediction and if you do that you know every tool call you will end up with a massive massive token spend. So very early on um if I understand correctly at least cursor used to do this with cursor rules. They would kind of swap them in and out based on what was the most relevant rule for the particular thing that it was working on and then it would evict an old rule that wasn't related and and very quickly they found out that like that costs a whole lot. Um, and so this problem is more difficult and more nuanced than just doing an LRU cache of the relevant information and calling it a day. Um, so that that's the other like constraint here that's like difficult to explain without going deep on that. But this the caching of the prediction mechanism enables this to be very very cheap

33:09 whereas this whole thing is very expensive. Um all right so I'm going to dive into the

Evaluating plugin abstractions for large monorepos

33:17 so plugins are the mechanism that we provide for customization especially at enterprise scale um of cloud code and I want to talk through each of these uh the the plug-in abstractions that you can build plugins out of and talk about like what happens when you throw this into a monor repo with a 100 million lines of code right where or if you're not a monor repo company what happens when you have 5,000 different repos that all have information that you need to know in order to do something in one or two of those repos.

33:52 Um, and so the question there, right, is does it scale? Right? You if you have 5,000 different repos and they all want to provide three or four different skills to explain how to work with that their own part of the repo, right? Um, what happens? what happens if each one of them provides their own MCP server. Um, so we're going to walk through that.

MCP servers and skills face scaling limits

34:15 We're gonna walk through that in the context of four specific plug-in primitives. There are other plug-in primitives, but a lot of the logic kind of transfers there. Uh, anyway, we'll dive in. Um, who here knows what MCP is? Okay, I think most people have heard of it, but I don't think most people have thought about it in terms of customizing uh agentic coding uh environments. So if you are going to build some integration with the um with anthropic in general, you probably should send out a public MCP server, right? Or you should put out an MCP server for that integration. But this is about how do you customize your codebase to interact with other developers at your company. Uh MCP is this protocol that exposes it. It basically just adds more schemas for tool calls to your system prompt, right?

35:11 It gives a a way of doing that. It's a JSON based protocol. Um it gives a way like a very lightweight from the client side way of the um client to say here I want extra tools to deal with this thing, right? Um it is transport agnostic. It has a lot of nice properties for consumers that are not necessarily great for software engineers. They can be, but um pretty rarely. So when is it the right tool?

35:41 Yeah, if you need to design this for any client, it needs to work with a chatbot. Um the server side owns the authentication, which is like a really big portability win, but then you have to actually set all of this up to work in any environment. And the fact is that like you're not programming in any environment. you are programming in the development environment of your company, right? Um, if you already have a command line interface for something, it's probably a lot better to create a skill that explains how to use the CLI for for your uh product. So, does it scale? This is like what we're looking at. What happens if you have a thousand MCP

4 Highlight · Why MCP servers don't scale without help

36:21 servers in your monor repo and you try and just load them all into context? Well, every tool has a name, a description, and a schema that sits in the system prompt so that the model knows how to use it. If you have 20 MCP servers and they each have 15 tools, that ends up with a context window that's mostly full of um tool descriptions and you have very little room to work, right?

36:48 Um so no, it doesn't scale without help. What we have been working on recently is a thing called tool search which loads the names of all of the tools up front. Uh so it doesn't still like completely scale because if you have I don't know literally a million tools then you're going to use your entire context but it scales a little better than um name plus description plus schema. So we we give the names and then we give Claude a tool to search for tools which yeah um turtles all the way down. But um the problem is that that like there's not a lot of free lunch here. Um the more descriptive name you give or especially if you include like some set of some subset of the description, the more likely Claude is to remember to search for the tool.

37:42 Um and if you don't give much if you have a like random tool named query then uh cloud will very rarely search for it at the right time. So you do still have to think about how many MCP servers am I going to give if you use cloud code you should be thinking about how many MCP servers am I giving an access to because that is going to dilute the amount uh dilute its ability to find the tools it needs to do its job. So, it it does scale up a

38:13 little bit with tool search. Um, but not great. Uh, who knows what skills are. Okay, cool. Um, skills are just folders, folders with a markdown file in them. Um, the basic idea here is that a a skill is a um kind of a lazy system prompt, right? It has a bunch of instructions and it has some sort of summary of those instructions that the model can use to decide when it wants to expand this part of the system prompt.

38:44 Right? This is how a lot of kind of scaling you have a like a a small thing and then you can turn it into a big thing on demand. This is how a lot of scaling in software engineering has always worked, right? Um and uh yeah, so you can you can also have um other resources in the folder with the skill and claude will know how to get to them. Um but the uh I think the thing that has resonated with a lot of people about this is the fact that it has such a simple form factor, right? It's just a folder. There's no special protocol.

39:21 There is like front matter fields to the skill.mmd file, but there are things like description and then you put this like short description that tells the model whether or not it wants to expand the um the full prompt. Give me a second. I'm going to take a drink of water. It is warm up here. Are people following more or less where I'm going? Okay, I'm seeing a lot of nods. So, I'm taking that as a somewhat good sign. I guess the people that I have lost are asleep so that's okay too.

39:54 Um it's it's a lot right like this is a software engineering is a field that has had that's been around for 60 70 years and we're having more change in the past two years than the rest of it right in in some ways at least uh or at least arguably u we can talk about whether or not we have but all right um so skills so we so skills are these lazy system prompts right it has a description of when Claude wants to use the thing, it expands to a much larger amount of text. when Claude decides that it is relevant. Um so does it scale right the the whole body is pay so we are getting like don't pay for the body if you don't use it but um and so like you know at a small codebase scale you know maybe 500,000 lines I would call 500,000 lines of code a small codebase um and I think many of you probably work in enterprises where you would call that

40:55 a small enterprise codebase right um the the body is paid per use. The description is always loaded, right? The description always goes in the system prompt. So if you have a 100,000 skills and you load a 100,000 descriptions into the system prompt, you are going to use your entire context window, right? So it it still doesn't completely scale up to a large enterprise like Google's codebase or Facebook's codebase.

41:26 Um, and I think for me at least that's the interesting question about how do we get these things to do software engineering. They need to be able to scale up in that way. Um, skills also currently don't have a way to do hierarchy. We're working on that uh hopefully out soon. So, it does scale uh kind of it's kind of the same problem that we talked about in terms of uh tool search, right? tool search has this um like description this this description line that if you make it longer and longer it it like triggers better um yeah sub agents what's up we working on that yes so the question was do we need a skill search tool we're working on that it's um it's a little harder um because it requires the model to know what it doesn't know versus tool search the model is much more likely to

42:29 know what it can't do. Does that make sense? Like it's a lot easier to tell that you can't do something than to tell that you don't know something. Um but it yeah that is a a direction we are going. Sub agents are relatively similar to skills. I'm going to go through this pretty quickly because I actually want to get to the nice stuff at the end. um where you you basically give it a short description that gets put into the system prompt, but Claude instead of Claud expanding that text into its own context window, it'll expand that text into a separate context window and then get back a summary of what the sub agent did, right? It'll expand that text along with like its own prompt into a a sub aent context window, a separate context window, and then gets back a summary that that is that is compact, right? So it's in this sense it actually scales better but across a um monor repo a large monor repo where you have hundreds of these or thousands of these or tens of thousands of these you're going to start using up large fractions of your context window on these description

43:30 strings right so um but the yeah the takeaway for understanding the difference here is that skills are in context sub aents are out of context I think we think that the the difference is fundamental enough uh to keep them as separately named things, but there's a lot of similarity there. I think we're still just we're still figuring out how we want to name these things, how we want to separate these things, what the um ontology of all of this is. I don't think anyone really knows what the long-term state of this is, but hooks are my favorite

Hooks provide zero-context-overhead customization

5 Highlight · Why hooks are the ultimate scaling abstraction

44:07 abstraction to talk about in this context because hooks do scale. Uh spoiler, right? So this is this is like the kinds of abstractions that I think we generally want to be building are the abstractions that don't put anything at all into the context window uh unless you somehow determine it is relevant. Right? So hooks are an abstraction that we have uh in cloud code that will when a certain event happens like a tool use or a user submits a prompt or the model stops um working for one reason or another um or the the whole context compacts. You can run a specific script and there's like there is a protocol for like what the input to that script looks like and what output you have to deliver and based on how you deliver that output we'll just we will put um some context into the window into the context window for you or not. Um, but I mean the the way to think about how this scales, right, is if you have if you are working on a Rust

45:09 project and you have 10 uh JavaScript skills, you still pay for those 10 short prompts related to JavaScript even though you're not doing anything related at all. And you have to remember to manually toggle them off to not use that. Right? If you have 10 skills for linting your JavaScript code and you are working on a Rust codebase, you don't pay well, you pay for like running the script that immediately sees you're not writing JavaScript and then goes away, right? You don't pay for anything you don't use except for like CPU resources, which I think we can all agree are a little bit less limited than context space.

45:54 Um, yeah. So, it runs outside the context window. You can do them, um, you can do them as scripts. You can also actually put an agent in here if you really want to. Um, probably use that sparingly because you'll use a whole lot of tokens really quickly. Um, does it scale? Uh, well, we already talked about this. Uh, it in only injects something when it fires, when it matches. the the pattern that we have is like call the script, determine if it's relevant immediately, and quick exit if not, and otherwise do some processing to figure out if there's anything relevant that you can tell the model. Um, this is where the red squigglies live as we talked about earlier.

46:35 Um, what are some abstractions that I don't consider to be good parts of

46:39 plugins? This is like a pretty quick aside, so I'm going to go through it, but the one that I I really like to talk about here is cloud.MD. I have had so many people ask me why do plugins not have a claude MD abstraction and I think hopefully if you followed the talk up to this point you'll pretty you should understand this right it it it is always it is the worst at uh don't pay for what you don't use right because if it if we were to provide an abstraction that allowed plugins to unconditionally inject a large amount of text at the very beginning of every context then we'd limited to five or 10 plugins being up being uh active at any given time.

47:23 And and the biggest problem with that is that like it doesn't look like it's that expensive, right? If you're writing out your plugin, probably the first thing you're going to do is create a cloud. MD file that explains what the plugin is. Like it's so low cost on the implementation side and it's so high cost um on the uh sorry, yeah, it is so high cost on the use side, right? So it it you can still do this, right? You can still set up a session start hook that injects a whole bunch of tokens into your context right at the beginning of your of of every single session, but at least that makes it obvious that you're doing something for that costs everyone no matter what.

48:05 Um, memory. Memory is another one that I like to say kind of establishes a different boundary of what I would call a plugin, right? Memory is like this model curated. Who here is familiar with model memory or agentic memory? A few people uh spoiler alert, it's text files. Um all of its text files. It just you just tell the model to write a text file and then like read the text file at a very specific point later, right? But it is because it's model curated. I don't consider it to be a context engineering primitive. Right? We really want to focus on what are the primitives for sustainable reusable context engineering. And I think I'd like to keep that separate from what we call a plug-in. So even if they are shaped like skills, even if they are things that are that are updated and contain the same kinds of knowledge of as skills, I really think that the future of software

49:05 engineering at at ultra-large scale is going to have to be very discriminating about what we call context engineering and what we call memory. Those two things are need going to need to function very separately. So I throw that in because I think I think I've I hope at this point in the talk I've given you the tools to understand why I believe that.

Scaling developer throughput through multi-clotting

49:26 Um all right the the part where we talk about some of the things that the cloud code team uses these days and where I see this going in 2026. Um the only way to scale up from here the only way to speed up from here is to do multiple things at the same time. Um, how many people would say that your regular workflow uses more than Well, let's say how many people would say your regular workflow uses more than one clot at a time, more than one session? How about more than five, more than 20?

50:01 Um, okay. So, some somewhere around there. But you can pretty quickly say see how if you could get to the point where you're comfortable managing 20 clouds at a time, you could get close to four times as much work done if as if you can manage five. So I think a lot of the kind of work in 2026 is going to be figuring out abstractions that give developers the uh cognitive space they need to context switch between these things quickly. Um get work trees are the simplest way to get started on this.

50:34 put each session on a different work tree so that you're not the clouds aren't stepping on each other. I don't know that I need to go too far on that, but one of the things that really helps with this when you have multiple sessions who has used slashname or slashcolor, it's okay. It's uh three people should be everyone. If you've used everyone who raised their hand about multiple clouds, this this like works wonders. It's super super simple, super super easy. just label what it's working on. Um, give it a color. When you switch to the window that's doing it, your m your brain will see the color. It'll trigger some memory of what's going on there. Um, there's some research to show that humans like recall things related to color or non-colorblind humans uh recall things related to color faster than they recall things related to pros.

51:29 Um, my actual setup looks like this. I actually have these longunning agents. Um, this is a little outdated because I um didn't feel like I could operate efficiently with just 16 permanent work trees. So now this goes all the way to Z instead of A through F. But each one of them is separate. They're kind of long running and they're set up. They're they're parked in one spot and there's an agent that knows that its responsibility is to maintain that work tree long term. So um and for me at least having an agent with a persistent name and a persistent memory um it kind of manages its own like scratch files within its own uh branch of the work tree. Um allows me at least as someone who is has experience being a tech lead or a of a large organization where you have a bunch of different people coming to you over the course of the day telling you what they've done. having persistent identities really helps me.

52:29 Um, so I think it's worth worth trying. This is like diagrammatically what it looks like. I have like long lived work trees that are that have names corresponding to the agent name. Uh, and they all track upstream main and they know that they're supposed to manage that tracking of upstream name. So they don't step on each other even from a um, so it's like I mean it's like they're separate developers, right? Um anyway, uh yeah, I have a whole talk that I'd like to develop out about like how does agent identity work with cognitive load? Um so I don't know. I think that's a very interesting thing to explore over the course of 2026.

6 Highlight · Scaling via agent teams and cross-session messaging

53:11 Um agent teams. Has anyone here tried agent teams before? Oh, that breaks my heart. We need more. Um, so agent teams are are ways the the basic thing here is just giving the model ways to talk to other agents. So if you're running two clouds at the same time and you tell one of them one thing, you can tell it to tell the other one the same thing if it needs it, right? Um, send message tool is the the baseline primitive that gives Claude the ability to send one send a message from one session to another.

53:45 Um, you can have a a team leader that delegates to teammates. You can also just have uh a bunch of agents that are all peers of each other. And if one of them discovers something, you can tell it to share it with the others or you can tell it not to. This is all under your control. We are moving towards I actually don't know if it's been launched because I've been out of office for a couple of weeks but it is announced enough um that we're moving towards a direction where if you are signed it in a certain way and you give it permission then any session on any machine that you have can talk to any session on any other machine that you have um hopefully um so that you just have the ability to tell the model to talk to other instances. Uh, and this is like another example of like give Claude access to all of the things that you have access to, right? Because it needs to be able to see the things you can see. And if another agent is one of the things you can see, then you should be giving it access to that, right? Um,

54:45 slash oh, babysitting. Um, slashloop is something we shipped maybe two months ago. Um, it is uh literally a cron command or cron tool for the model that it can schedule a message to just show up every 10 minutes. It has the ability to turn itself off at some point. So if it's like, okay, that's enough. I've seen this every 10 minutes for 3 days now. I'm just going to turn it off. um uses some some amount of tokens obviously, but um it surprisingly minimal too compared to human having to go and check CI and then copy and paste and get the thing started up again. I find often that the problem is that the model goes to sleep before it's finished the job. And this really helps with that. It also I mean I think it also gives Claude the tools to make sure it doesn't do that, right? And we see that increasingly the models are better at understanding the fact that like if

7 Highlight · Auto mode enables multi-clotting safely

55:42 something is going to come in as a result in 3 hours that it should set something to wake itself up in three hours. Um auto mode. Who's using auto mode? Should be everyone hopefully. Um yeah this is really what enables this multi-clotting, right? This is how you get up to 20 or 30 agents, right? So you're not just sitting there hitting inner all day. Um it is better than YOLO mode. We have a sophisticated set of uh safeguards that go through several screenings of of models and classifiers that uh check whether something is dangerous but before blocking it or allowing it. It looks through all of your prompts and sees if you've actually told it to do something or not. And if you haven't told it to do something that seems dangerous, it'll stop it from doing it.

56:33 Um, this is what makes sloop and agent teams and overnight runs actually work.

56:38 It's a it's a little expensive. It it costs uh anywhere from 10 to 40% more depending on the model. Um, I am running out of time and I'm sorry, but um, this is really what I'm getting at here and I think I've said this a few times, right? 2025 was about getting information into the model to get better results because it Agentic programming was in its infancy and we were constantly compensating for the model making mistakes. 2026 I believe is going to be a lot more about getting information from the model to the user.

57:16 Right? The quality of the information that the user gives the model is going to matter less and less as the models get stronger and it's going to be less and less of a bottleneck. Um, and this year it's going to be a lot more about how quickly can you context switch, how quickly can you scale up your workflows to get information about what the model is doing. Uh, Claude put it this way. Um, your attention is the smallest box in the system, which I think is cute. Um, so cloud agents is actually one of the things we've launched along those lines if you haven't tried it or it's also called fleet view. Has anyone tried this yet?

57:56 One person. Okay, everyone should try this. It is It is amazing. Um the team member who put this together literally hit I think a thousand PRs one week using this um or yeah it is um significant way to scale up your attention right because it is showing everything that's running all in one place. We have little classifier models that give a very short description of like what the model needs, what it has just done. Um, and these are like, you know, 10 different cloud code sessions all in one view. Um, and yeah, the lightweight classifier and you can jump in between sessions from directly in there. No more like 20 different tabs.

58:44 uh remote control, another fantastic thing that I use all the time. Um I have these kind of persistent agents running on a server in the cloud and I can uh interact with them on my phone. I interact with them on cloud code desktop. Um I wrote portions of this talk literally while walking over from my hotel by interacting with the agent on my phone that was responsible or sorry on my dev box that was responsible for this talk.

59:11 The three things I do want to reiterate before I run out of time. I have 30 seconds. Um, give the model access. Think about the box and think about what you're putting into the box and how that affects what you're getting out. Uh, and pick abstractions that scale up to large scale real world software real world software engineering. Um, thanks for hanging out hanging with me. I'm sorry that my uh stumbling over my words was a little worse than usual. It is also after five for me. But I hope you all enjoyed it. I'll be around. Uh you can always hit me up on Twitter um if you have questions about this. I'm happy to answer them. Um yeah, thanks for your