Dexter Horthy: Why We Stopped Trusting AI to Write the Plan

11,595 words · about 46 min to read

Shifting away from unreviewed AI code generation

0:00 We wanted to be living in the future. And so we said, cool, let's try this. Let's pull requests. But like, we'll just say like, cool. If you read the plan like the code is not as important. Yeah, we did this experiment. We ran this for like 4 or 5 months before we realized, like, oh, this code base is actually unusable now. Like, it became slower and slower. The change over here cause a regression over here. And then we decided, no, this is bad. And then we literally rebuilt the whole product from scratch. And the first two weeks of that, I co-founder Kyle was literally in VS code typing every not even cursor like typing every character by hand to plumb out the data architecture for the new version.

0:32 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Podjarny and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. Yesterday you were writing code. Today you're prompting agents.

0:59 Tomorrow you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads and VP's packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub, cut through the hype and get to what actually works. This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future.

1:27 Three days of real talks, real tools, real answers. Whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. Our early bird pricing ends on September 30th, so make sure you lock in your place before prices increase. As a podcast listener, you can get 15% off your ticket using promo code POD15. We have so much planned and we can't wait to have you join us in New York.

1:59 Hello everyone! Welcome back to the Native Dev. Thanks for tuning in. Today we have a really fun conversations about software factories and how to do them or not do them. And how did we think and we didn't think and all that. And to sort of dig into the timeline and sort of really kind of informed opinions about this. We have Dex Horthy, who is the CEO and co-founder of HumanLayer. Hi, Dex. Thanks for tuning in here. Talk to be here. Dex, you've done a lot of super interesting things beyond HumanLayer on it, but I think there were these sort of 12 factor agents that you wrote before trying to codify and explain how do we should we build agents on it today?

2:35 I think you're sort of starting to refer to like 12 factor factories on it. It's a bit of. Oh, you got that already? Well, we just sort of know that you've done it. I actually don't know. Maybe we'll dig into that a little bit on it. And, and I think also what's sort of super interesting is that you've been sharing kind of strong opinions, loosely held a little bit or maybe like, changing over time. They were strongly held, but pride out of your hands. Yeah. Over time on it.

3:00 Which which I think makes for sort of sort of good insights and conversations to have. So, thanks for tuning in. And maybe I'll, I'll kind of dig into it straight away, which is you've you've sort of changed your mind substantially and declaratively around

The evolution of the planning metagame

3:17 kind of the way to do a gentle development of it. You know, really around specifically the kind of lights off development. Right, versus versus not. So what happened there? Like, what did you believe before and what's the what's the turnabout? Yeah. I mean, the journey kind of starts around, you know, around when opus four came out, and when they launched opus for, they also launched the Claude Agent SDK, which means you could use it was it had a very small interface at that point.

3:45 Right now, the Claude CLI is basically like the JVM of of agent coding. It's got like 200 flags on it or. Yeah, it's a freaking battleship. But, but it was the first time you could use Claude like small agent loops as part of a broader, more deterministic piece of software. So we were building like terminal multiplexers for Claude Code of like, hey, what would it look like to build a tool where you can manage a bunch of sessions in parallel or like manage, get an inbox of like, what are all the approvals waiting on you and things like that.

4:16 And we spent a lot of time trying to get how how do we get these agents to work for longer? It's very interesting. The, the biggest change between summer of 2025 and now is actually like the planning metagame. I don't think a lot of people realize this. A lot of people just write plans because they put plan mode in the thing, or they read something online and they found some skills and hey, we're going to make a plan. And I think depending on where you are in your journey and like how the models have changed in retrospect, I realize that, like writing a plan file in the summer, in like July of 2025, the number one most high leverage thing that that got you.

4:56 Was it was a really reliable way to get a model to work for longer. Yeah. Well, okay, we're going to do everything on this plan. And to kind of follow instructions I think a little bit at the time. Right. It was well, it didn't always follow instructions, but you had also had things like a spec kit, for instance, who though was trying to like really micromanage the, the agent and sort of like say, okay, not only is this the plan, but at any time, just come back to me at the end of this step and I will spit out the precise set of additional plans that you're going to do.

5:26 But I think it got them. Yeah, it got them to sort of kept things a little bit on track, or at least more than it was if you didn't plan. And it was a way to like also like make the work a little presumable. Right. It was, it was, it was the first version of like first the compaction of intent of like, hey, I'm going to take this whole thing that I want to build, which could be specified as like every line of code I want to write. Or you could put it into this file, a couple of hundred lines of markdown. It's like, here's what we're going to do.

5:52 And then you could keep that up to date with what was done, what changed. And so you could resume from a clean context window later, which again also in July of 2025, like the models were not that good at long context. I coined the term called the dumb zone or whatever it was, which at that time was around 100,000 tokens. If you went over 100,000 tokens, you got way worse results. Obviously, that number is higher now. Yeah. Although there is probably still a dumb zone, like there's still a there's still a spot in which, like I said, like I reuse that term and I feel like it has a, it has an element of like, you know like too much focus is no focus, right.

6:24 You know, when you have sort of too much information in it, then finding the area of attention is still something where if you're intentional about it by yourself, making the decisions around what's in the context and what's not, then you can you're more likely to achieve the goal that you're trying to achieve. Yeah. So anyways, getting back to like our journey is like we had just come away from Sean Grove had given this really interesting talk. I recommend watching it. He was a researcher at OpenAI at the time, and he was talking about basically like the spec is the new code.

6:53 And this idea that like the way we think about assembly now is the way we're going to think about code in the future, and the spec will be the thing that is verifiable and executable and compiles into code and that like checking in the code. He made this he this really good analogy of like, hey, you write a bunch of Java code and then you compile the, the jar, right? And the jar goes into production or whatever, whatever your compiled asset is.

7:20 Right. And then you commit the code to the repo. It says in the future, like you'll basically just commit the specs. But right now, all of the interesting stuff from how people were using coding agents in June of 2025 was basically, you send a bunch of prompts and then you get your results out, and then you commit that, and then you push that and that becomes part of production. Yeah. But like that, all of the interesting, useful, more high leverage stuff is actually all the prompts you sent in that conversation that should be recorded somehow and that like having a long chat with an agent and then throwing away the chat and shipping the code is going to be the new version of like, what if we wrote, you know, a bunch of Java code and then compiled the jar and checked in the compile jar and threw away the code that we wrote, right?

8:07 Yeah, yeah, I think, I mean, I very much kind of remember that it needs to be sort of held, frankly, like this conversation is not just sort of past tense. Right? Like, I think today there's a lot to be claimed. And I know this is very much in your world, which is like, those are the very artifacts that you're supposed to be reviewing and discussing and preserving on the, at the time, but I think so. So, so you were like, touting, okay, we want to do, like, context engineering and kind of capture all of these sort of intense on it, but I guess what was the what was the advocacy then like at the time when you were guiding people to say, hey, be amazing with sort of agentic development, like, what is it that that you should do?

8:48 And I'm interested because, like, I, I know that you kind of went from like almost it feels like you went from like a context engineering kind of advocate to, to almost like a, you know, an antagonist to the, to the group on it that it's like a little bit more like you really need to be a more, more defined and more reviewing.

The hidden cost of unreviewed AI architectures

9:06 And I've heard you say now that you should review the code for now. So I guess what's the what has changed? Yeah. I mean, this thing happens with models as new models come out and get smarter. You never know when the singularity is. And the only way to like, stay ahead is like, try to live in the future. And so I said, okay, this is the thing people are saying might work. And people in the lab, you know, I think there was a thing that was happening last year a lot where it's just like the labs have way more alpha than any of us.

9:32 So if they say, this is a thing you should do, then you should go do it. And so the OpenAI said, oh, this is working for us. Then, then we wanted to try it. We wanted to be living in the future. And so we said, cool, let's try this. Let's try like WhatsApp pull request, but like we'll just say like, cool. If you read the plan like the code is not as important.

1 Highlight · Unreviewed AI agents build overly complex architectures

9:48 Ton of ton of issues with that. First of all, the plans contained all of the code that was going to be written. And so you read this plan of every, every single. It was literally like in-line diffs. Read the plan and then you would go you would go write the code. And then if you didn't read the code, you would basically, if you wanted to also review the code because there was drift or you, you know, you were larger. We were three person startups, so we could have this kind of option. We were market fit. It was like, okay, go, go go go go.

10:13 But we kind of recommended this to some larger teams, even teams like 50 developers. Then they would say, yeah, we were at the plans and then the person went and did the implementation and then the implementation would drift. And so they would have to review the code to they weren't ready to let go of reviewing the code, which good for them. But it was actually they were actually doing more work. They were reviewing basically the same, almost the same shape of code twice. Yeah. Depending on the sort of the, like, the detail level of the sort of the spec. But either way and combined with like some sort of sense of discomfort because now you're reviewing someone else's code and so you're kind of a familiarity and control over it.

10:45 It's oftentimes like harder to apply your opinions when editing than it is when you kind of write it in the first place. Right? Once. Yeah, once. Once the code is written, you're kind of committed to a path. And it's like much more work to change it to the point where sometimes it just makes sense to throughout the implementation, like drop a new prompt in or a new or redo your plan and then and then go again. But, yeah, we did this experiment.

11:11 We ran this for like 4 or 5 months before we realized, like, oh, this is this code base is actually unusable now. Like, it became slower and slower. There was like, every single change caused, you know, change over here, cause of regression over here. I think this is also one of the things we're like, what if we let the model own the system architecture too, like human in the loop reviewing it was like, hey, how would you solve this? Well, well, make a Golang daemon. And then I would use a Unix socket and then I would connect these things. And then when we run the mic piece and we ended up with this like system of like six different services where you have like a desktop app that talks to a socket over Rust and then that that's like a Golang daemon on the back end, which then launches and manages Claude Code sessions, which then launch MCP servers to handle the approval system.

11:51 And then those MCP servers call back to the front end and like put things in a SQLite database via Unix socket.

11:57 And then that shows in the end, the user approves something that goes back to the MCP, back to Claude, back to the Golang daemon, and back to the UI. It was just like this incredibly complex system that the models developed. Are we if you if you let them decide how it should be architected. How much is that, though? How much is it that the models don't know how to do it versus you just sort of saying like, hey, figure out an architecture, like how much of that can be prompted? And I know we're sort of still in a little bit of like a past ten.

12:23 So maybe some of this, like we need to talk about like current perspective on it. But I think there's a difference between saying, hey, models can't architect and models can't architect yet, or they can't architect if you don't give them enough guidance. Yeah. And I think, I think like fast fast forwarding a little bit. So like we ran this experiment and then we decided no, this is bad. And then we literally rebuilt the whole product from scratch. And the first two weeks of that, my co-founder Kyle was literally in VS code typing every, not even cursor like typing every character by hand to plumb out data architecture for the new version.

13:00 Yeah, exactly. But I'm really glad. Like those those patterns have stayed really, really solid. And like, especially for starting a new project, the decisions you make early on just constantly cascade into the future. And we've we've changed some basic things, but the core ideas behind it are still really strong. And we built on like ElectricSQL and durable streams and have this kind of directional data flow where like clients don't have to like there's no signaling logic from the back end and there's no like, signal receiving logic on the front end.

13:28 That's all just handled by the sink engine. Which which was a much better way to do it. And I think any, any piece of software built today should use like a real time sink database where clients can just, like, connect and get updates live. But, anyway, so we basically were like,

Keeping AI code simple to avoid debug traps

2 Highlight · Building systems simpler than your peak intelligence

13:45 okay, we need to read the code, we need to understand the architecture. And then also like, there's so many slop patterns that get in and that these models will, left unattended, will create a system, I don't know, there was this thing, I think it was Donald Knuth or one of these like OG like C language guys had this, had this, had this sort of principle that was like, in order to debug code, you need to be like much smarter than you were when you wrote it.

14:12 And so if you write code that is like the smartest code, the most clever thing that you can come up with, you are by definition not smart enough to debug it. And so the thing you're building always has to be more simple than, than, than than your actual peak. Peak intelligence, which I don't know, I think just in the news last week was like Steve, who spent the last year running around every podcast on the internet and saying, like, hey, and I love Steve.

14:38 Steve's brilliant guy. Huge fan. But I am going to call him out for going running around the internet and telling everyone, hey, if you're still trying to read the code, you're not going to make it. And, like, just let the models cook kind of thing. And then last week kind of said, hey, I used fable to build this thing, and fable made a system so complicated that it itself could not debug it. And you really have to keep an eye on these things and, like, understand the system you're building.

15:04 So, so sort of the. Counterargument to this, like I, I kind of understand and I relate and we sort of still have like a Tessl as we sort of build and we are sort of a team of sort of 20 odd people on the engineering side.

15:16 So, you know, we do need some sort of supervision, by the way. Like, still compliance systems require kind of a name to every sort of code review and all that. So we have a lot of like, AI system code review and some changes are allowed to sort of go autonomously and there's some supervision of complexity, etc., etc.. So I wouldn't sort of say, hey, entirely lights off. But I think the counterargument to reviewing the code is, is that you're stuck, that, you know, when you when you review the code and you apply a change, right, or you sort of give more specific instructions or open vs code and actually kind of handcraft some sort of portion of it, then what you're not doing is you are not building a mechanism or a system or an automation that enables the creation of code that is, that is to your preferences, right?

16:06 That is when you talk about those specs, like I, I always think of specs as this kind of this slider between determinism and adaptability. You know, it's sort of like how many degrees of freedom do you want to give the, the, the, the LM the agent to, to build with like these are the decisions that you have made and within those you should be allowed to play or like where you're enabling the agent to play.

16:32 And you need to define the sort of the outcomes that you want to see out of it. And you should be comfortable with sort of not caring as long as the outcomes are, are achieved. And so if you're if you're kind of building towards that, you would like to achieve that result, you have to codify your intent. And the way that you would do that is not by reviewing the code, but rather by continuously interacting with the system.

16:59 There's like a balance to it, like this idea of like plan before you build.

Using planning as leverage to reduce rework

17:03 Like the only reason to do planning or alignment or whatever it is, whether it's you in the model or you in the model in your team is like, because you have this factory and this is a I. Right? But there's all these pieces and I've talked about this before, so I'm not going to go in depth on this. But you have people making stuff and then someone builds it and then you have a review thing and then you have a rollout and you have monitoring and you have all this stuff. But basically this idea here is like what people learn 30, 40, 50 years ago is that this build step is like can take hours or days, and the review step can also take hours or days.

17:34 And so we as humans before AI, we're like okay, we're going to talk about the architecture, we're going to do this sprint planning thing. We're going to align with everybody on the team to make sure that this is at least directionally correct, in the hopes that we can decrease the chance of rework and reduce the review time, because if we talked about it ahead of time and we've all done this, everyone's read a perfect PR where they're like, yeah, we talked about this for an hour, and now this thing is 1000 or 2000 lines of code, but I, I it's, it's following our patterns and everything is correct.

18:00 So it's easy to review when people say we are drowning in pairs. We have too many pairs. My pushback is like, no, you don't have too many pairs,

3 Highlight · Balancing planning effort via expected pain

18:07 you have too many bad PR because you're letting people turn off their brands when they're coding. And so like the the other side of this though, is like, you don't want to over plan, right? So like I can yolo a two sentence prompt and there's a 50% chance I'll have to rework it. When I say rework, I'm in like I'm going to go back and forth with the model a couple more times. Please don't yolo a two sentence prompt and not read the code and send it to someone else to review. I'm assuming you're going to actually polish it.

18:33 Or you could like handwrite a detailed spec. You could sit there in a meeting and spend a bunch of time and spend five hours writing a spec, and maybe now there's a 10% chance you may still have to change some stuff. But, but like, it's it's less. Or you could do the extreme of, like, write all the code by hand and then, well, this is a 0% chance you'd have to rework it because of something the agent got wrong because you wrote it. You may still have to rework it. And we've had code review for 30, 40 years, but, it's you're not going to have to change it because of something to agent because you left the agent some, some, some leeway to, to, to make decisions on some of the smaller details.

19:10 And I think about this in terms of expected pain, like, what is the chance you'll have to change something later? And how painful is that? Like, I never argue with a model about like how what color a button is going to be or something like that, because I know that if it's not the color I like, I'll just send one prompt and it will fix it. But if like the database schema is wrong, then we're probably going to have to throw it out and start over. And so I want to get aligned on that first. Yeah.

19:37 And so the idea is like you have to build this intuition of like based on the size of the thing and the complexity of the code base and how good the code is today and how many bad patterns are in there that the model might follow, like where do you want to land on this curve? And that's that's what I think about like today, planning is all about leverage. How do you find the sweet spot for the going to get it wrong?

19:57 Sometimes. That's why I say like get a lot of reps. But this is this is kind of how I think I think I think it's similar to what you said, but just kind of visualize in terms of like what our plans for is to, is to kind of at every stage of your accumulator, we have a bunch of different planning workflows. We have, like just do a design dock and then go ship it. We have like design and then the order we're going to do it. We have like a PR flow that's like 3 or 4 stages, but depending on the size of the feature, you want to have one or more stages where you are just like slicing off 50% of the potential bad outcomes, you're narrowing the outcome space in a way that is like a good use of your time.

20:34 Yeah. I think, well, I have a whole bunch of like, thoughts on it. Like, I think direction, we're sort of aligned on it. Like you have to control the a little bit the or you have to decide what do you want to control for and what do you not want to control for on it? One I have a set of questions here. One is like how much of that planning is around kind of guiding the decisions versus expressing your intent? Because I think when you're when you're when you're planning, you say, well, I want the architecture to be like that and implicitly as well, because I want to make sure that it, you know, scales cheaply.

21:05 I want to make sure that it is, you know, resilient to like a additional kind of a whatever data fields on it, you know, whatever it is. But you don't say that. You sort of say, I want the architecture to be like that. And so you're, you're you're sort of limiting to your own creativity of how you would implement, but you actually never capture your, your kind of actual intent of why is it it's one of these, you know, ask why 17 times to to get to it.

21:32 So I guess when you're when you're going through that planning, how much are you trying to extract and do you separate the, what do you want to achieve versus how the implementation should look? I mean, I think there's a reason why Matt Pocock skills and specifically like the grill me skill has the most it's in like the top 25 stars on any GitHub repo of all time.

21:59 And it's like the top three stars on GitHub for repos that like mostly just

4 Highlight · Using AI to interview users about intent

22:03 contain skills is because like, people are bad at specifying their intent and and I think the the a good AI assisting assisted planning process involves like having the model get enough context to understand what's going on and enabling a user. The fastest way to get a good description of your intent is to just like off the cuff, say kind of what you want. I'll turn on voice mode and I'll ramble for 60s and the things I know I want, like, oh, okay, we're going to this has to scale to end users or this cannot use this pass that has to use this like has to use reading and not the Postgres database because I know the speed like anything, you know you should specify.

22:42 But the, the, the onus of like, hey, I need to sit there and just like think for six hours and decide exactly what I want can it's it's much easier and much faster to be able to take a very rough like large, unspecified thing under specified thing and use AI to, you know, relentlessly interview the user as, as to, hey, look, here's what you asked for.

23:06 Like, here's some alternatives. Are you sure you don't want these? Or like, hey, you asked for this, but there's a ton of open questions. They're like, here's what I think they are. And like walking down that decision tree. Yeah. So I think when. You're when you're so I think we're in agreement over there that you like, it's hard to extract sort of things out of it. But when you, when you think about like reviewing the code and, and the likes, I guess what I, what I struggle with a bit is this okay,

Back pressure and verification drive AI autonomy

23:29 like if you extracted the intent correctly and you have defined your kind of verifications correctly, right? You know what you're looking for, which are are both big ifs. They're both hard to do on it. Do you still need it? Like the models are getting pretty good, you know, let's put aside you know that is hard to do. Let's put costs aside for asset. Right. Let's assume you can kind of iterate and you can define because you define what good looks like on it.

23:55 Like do you think like is it what's the limitation over here. Is it is it actual like models or just not good enough yet in your view? Or is it a I don't know if you have like a guess as to like when would they be or is it a or is it more or is it something else. You know that it's like a that, that is sort of that like that causes you not to sort of say like what you really need to do is like invest in the capturing of the intent and the verifiers, and instead you're saying you should review the code.

24:29 I, I. There's like this there was this post from the Haskell community, like six months ago

5 Highlight · Using back pressure and specs to drive agent autonomy

24:38 that was like a sufficiently detailed spec. Is code indistinguishable? Same thing we have. Yeah, but those are like there are implementation specs like these are they're talking about cases in which I am dictating an algorithm to you. Right. Yeah. And so yeah, if you give I mean, the reason why I think specs are so powerful and people maybe get a little bit like one shot by a good spec with verification is like if you give the model a good way to get feedback on what it's doing and to like hill climb against some objective that it itself can measure, whether it's qualitative or quantitative.

25:08 Yeah. It will move mountains for you. And it's again, just like plans were a great way to get a model to work unattended for longer. A good spec and good. We call it like back pressure. Right, is how do you give the model the ability to get feedback on what it did without a human having to do that part? Yeah. You can get a lot more done and you can do more in parallel because that thing's going to go work for two hours and go kick off another one. Right. I think the so I think I think that I'm entirely aligned with.

25:35 So like I think the difficulty as I interpret what you've sort of said and sort of maybe like a bit of like to my sort of self view over here as well, which is like in a perfect world, you'd be able to express intent very well and you will to express your verifiers right of like, how is it that you would verify that it's right? Very well. And when you do, you don't need to review the code. Like if you're if you're able to do that, then you will achieve something in which you can allow autonomy, right. Like what is the level in which you'll allow?

26:02 Like what do you need to see to be able to allow autonomy? Yeah.

Maintainability lacks a fast oracle

26:09 I mean, I think it's a great way to kind of like, prototype and understand what's possible. Like, I'm working right now. I'm just a little experiment, and I don't actually think this is like a good. There are. I'm working on a, like, code search harness basically. And I know like just more film built a custom model just optimized for code search, called warp crap. But what I have been working on is like a code search harness for Jeff.

26:35 So, like, can you get Jeff to select the next correct action, like basically using Jeff's classifiers as tool calls of like, cool. Next, I want to read this file or the final answer like, hey, yeah, now we have enough context to answer the question or whatever it is. And the problem is Jeff only has a 32 K or 60 4k like input context window. I forgot I think 60 4k is. Yeah, it's some asterisks on it. But that means that you cannot put an entire code base in there.

27:03 Right. Or you cannot say, here's all the information we have so far. And so you have to get really creative about how you split things up and how do you merge them back together. And so I'm kind of taking like an evolutionary algorithms sort of approach of like, hey, cool, go do a bunch of subjects, have them try different things, try them hill climb different dimensions of the Pareto frontier for cost and speed and accuracy, and 45 different code search evals right now. And more coming.

27:29 I'm not reading any of that code. And if it actually works and it seems good, then I will probably like, take the, like root of it and be like, cool, like explain this algorithm to me. It turned that into a spec and then re-implement that with actual like clean architecture because there's just slop everywhere right now. Like the code is just like doing all this crazy regex matching and splitting and, so I think, I think not reading the code in order to prove what's possible and to prototype is, is very feasible.

27:58 But, I mean, I think everyone says, I don't know, there was this thing that people said in like the sonnet 3.5 opus for kind of Days, which was like, yeah, sure, the code is slot, but I'm not going to read it because like, by the time I need to fix it, that'll be GPT seven's problem. And the challenge is, is like, folks, we're at six. And every really good builder I know has mentioned that like

6 Highlight · Why maintainability lacks a fast oracle

28:21 like, you know, Astro got much better at computer use. It got better at video game slop. It got much better at blender, but it didn't actually get that much better at writing code that is maintainable. And I think this is because, to train a model at something, you need RL environments for it. And to have a good RL environment, you need a oracle. You need a thing that can verify whether the the product meets the needs of, of the user and maintainability.

28:51 You don't find out something as unmaintained until two months later. There's no fast oracle for software maintainability. I think as I was riffing about this with Addy Osmani, who was at Google for a while. And that was kind of his take is like, yeah, maintainability has no fast oracle, and it doesn't seem like there was this benchmark. Have you heard of Slop Code Bench? I'll give you the very high, very high level, which is like you give a model of code base, you give it a challenge, you say, hey, go, go write this feature and then you have the code base and then you say, cool.

29:19 Now go write this feature and it has to inherit its code base at every step. And it's like, now go write this feature. And like basically what the benchmark shows is one is very under saturated. Like I think GPT 5.5 in May got a 14.8%. And they're getting ready to publish the results for Astra. I think I was helping them run it with some inference and stuff, but like, it's not my benchmark, by the way. This is like a lab out of University of Wisconsin.

29:44 But we've been like, you know, providing providing runs and doing stuff like this. And Astro got in the first run we did was like 16.3%. So like the models are getting better in this dimension of like, can they maintain a code base over time? But not as fast as everyone thinks. Like I said, we said GPT seven would solve all of this. We're GPT six and like we are getting, we are RL ING and hill

30:09 climbing and other domains. But like the frontier of like model, maintaining code base over time does not appear to be getting better that quickly. Hey everyone! Hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests, so we can have the most informative conversations about agentic development, whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

30:35 But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

Codifying developer taste and opinions

30:53 All right, back to the episode. I love this conversation. I think about this all the time. I feel like the counterargument, to an extent, is sort of this notion of sort of of maintenance agents. Right. And I, I think in general, like through the Tessl journey. Right. We started talking about sort of spec centric software development, spec driven software development, and then we moved into this sort of context and skills. And to me, the the sort of the premise of that evolution was that is maybe the acceptance that when you think about an amazing developer on your team, they're not amazing just because they know what you are building, but also because they understand how you are building.

31:30 And really, to me, context is sort of the breadth of it, right? I think like you look at skills, for instance, they will have, I think three primary categories. They will have specs which are more about like how does your product or another product work on it so they can build it. They will have workflows around how do you want to sort of perform some action on it? And then they will have opinions, they will have policies, best practices and like choices where it's not necessarily correct or incorrect.

31:57 It's just your choice. We got in the room and we agreed this is our security policy, right? This is our sort of a balancing act. Or we say, I don't know, we had a couple of these at Sprout Social, my first, my first gig out of college and they called. They just had a list of things. It was like, do not wrap APIs in other APIs, exactly like minimize code traversal depth. And my boss at the time, this guy Allen, crazy smart dude, he had this thing of like three. He called it three copies.

32:23 Press hard is like at the top of your end point. You always have like a okay, log a thing and then send a metric and then do another thing. And every engineer wants to like collapse those into a function like an abstraction, like send metrics. And his take is just like, no, just duplicate the three lines of code everywhere. Because as soon as you put an abstraction over it, it ends up like leaking a bunch of stuff and it becomes very complex and it gets a bunch of responsibilities.

32:48 Just call the three different methods in line, and this is what you do when you go to the car shop or the car mechanic, right? You, you they say three copies pressed hard, you write down, you sign the thing, and then one goes to management and one goes to, one goes to processing. And one, one you keep. And they've just figured this out that it's like, no, we're not going to take one thing and then like make copies of it or whatever. It's just like, yeah, it seems a little weird in archaic and like, okay, you could be smarter about it. Anyways. Yeah.

33:13 These are the kinds of opinions that, like, are very hard, this taste that engineers have over five, ten, 20 years. Yeah. We haven't figured out how to put that in a model yet. We just haven't figured out how to give models that that taste. So I guess I guess what I'm asking is, okay, so you can't put that in the model. And I don't know if they ever will because like models, almost by definition they are generic intelligence. The one you use, then the one I use are the same. And so even if they are. More. Or less smart, they're, they're, they're the same opinion.

33:41 You know, they will they will run. You know, design is probably the easiest examples of that. You know, like like you might like dark mode or you might hate dark mode, right? Like, you know, you might sort of choose that. And so but but you can't put it into the model but you can put it into context. And so you can define and if you've captured and that to me is another planning aspect, right. Like you've planned how you want to build. And now you can run a whole bunch of maintenance agents that would run on that code base, that would review code that was created by an implementer agent who took in more like feature requirements.

34:13 Right? Like sort of requests on it. And it will ensure and you codify your taste, your preferences, I guess. Why isn't that the way to achieve a non or a low slop code and kind of invest in that path versus reviewing the code? Yeah, I mean, I could see I could see a world where, I mean, the way this usually starts is people just add one end up adding one after another after another of like bullet points into their cloud empty.

34:42 Other agents MD of like always do this and never do this. And then you end up with a bunch of instructions and like most of them, don't get followed because it's too much context. And so like, yeah, you could probably I mean, so we do this thing internally at HumanLayer where we have, deterministic detectors for some of them, some of them are non-deterministic but very simple of like, we don't like this pattern. The desired end state looks like this. Yeah. There's a bunch of code that looks like that. Yeah. Whether it's like react doctor deterministically pulling warnings out or oxlint pulling warnings out, it's like every night we have 4 or 5 different, people call them agents.

35:17 It's the same harness with a different user message, basically like five agents. If you if you if you must. Have sort of a skill powered workflows or sort of prompt power workflows. Exactly. And they run on crons and GitHub actions. And every morning we wake up to five pull requests that make our codebase a little bit better along some dimension. Yeah. I think the challenge is, is like, you have to know exactly like you said, you have to know exactly what you want.

35:44 You have to codify all of your opinions somewhere. You probably can't be in your agents empty. Maybe you can put them in a scale, or you can break them up into a set of skills. But the the more opinions that you have, the more clever you have to be. This is context engineering. This is always the thing is, like the more clever you have to be about like splitting them up and parallelizing and kind of like orchestrating multiple different things to go do this. Yeah. So I get the point around sort of the needs to review the codes today because of the limitations.

36:10 And as I mentioned in Tessl, we still sort of review the code, but to an extent, sort of the bet that we're making in our own internal software factory is that really it's all about extracting correctness out of observations. And so what we do is we take today maybe some amount of sort of a risk or burden that really forces the process of not writing the code and sort of reviewing the code, and then you interact with the systems.

36:35 There's exceptions to that. There's asterisks next to it, but by default it has to be kind of a feedback that you provide about what is wrong, you know, as part of pull requests, as part of systems and sort of these sort of continuously, improving skill powered workflows. So like these feedback loops that, that really help feed back into the system. And I think there's a good case to be made that today we are maybe making more effort in sort of in sort of investing and honing those skills on the, on the sort of the bet, which is always TBD that it will it will set us up better, and we're sort of seeing some compounding speed out of that.

Preventing slop code in software factories

37:19 So I don't know how how you think about that and how does that align or conflict with like reviewing the code. Yeah, I mean, we are very bullish on this compounding thing. I think probably like the anti-pattern, I've seen a lot of orgs and most of them are probably larger than yours or mine is is someone gets excited and they say, I'm going to build the software factory. And they kind of throw everything out and they say, cool, let me design this whole thing from scratch, and it's going to have this compounding thing, and we will never have to read the code and whatever it is.

37:46 And like, I think that's like any piece of software, the big design upfront is like is always going to lose to the like, iterate fast and get 1% better every single day, or even 1% better every single week. So I think like the idea of having having a process

7 Highlight · Why software factories cannot tolerate bad code

38:03 thing is, is like, you cannot afford to put slop code in your code base because every, every time you let bad code in, you are making the entire system worse. People talk about software. Yeah. Yeah, people talk about software factories. But the there is a big difference between a traditional factory and a software factory. And that like in a factory, if a car comes out at the end of the line and it's bad, like, that's cool, whoever was going to buy that car or whatever is going to is going to deal with it.

38:29 But like if you ship bad code in your software factory, it degrades every future piece of work that goes through that factory, because there's more bad patterns for the model to see. So like, you do want to continually being making progress towards, you have to want to have some ramp where you can make the factory better every single day or every single week. But it's like the number one priority is like, stop the stop the bleeding or stop the, you know, close the windows on the house before you start conditioning.

39:01 You have to. Yeah. Yeah, exactly. But no, this idea of, like, cool. You know, we have humans in the loop whose job it is to make sure that the code is good and that nothing goes into prod or into the code base. That is like going to make the quality worse and going to make future quality worse, because the agent will see that and think it's okay. But a byproduct, the kind of like exhaust of that process is you get a ton of like real examples that you can use

39:28 to feed improving the factory and make it a little bit better. Like anytime a human has to put a comment on an agent written PR and say, hey, this is wrong, we don't do it this way, agent goes to fix it, it's fine. You need to be like siphoning that off every time someone sends. Even in a prompt, you should be reviewing the session traces. Maybe not reading them verbatim, but analyzing the session traces of everyone on your team and saying, where are people getting frustrated? Where are they telling the model they did something wrong, and then figuring out how to improve the entire system, whether it's the base skills, whether it's the code review skill, whether whatever it is, so that, you know, if you're not getting better, then you're getting worse.

40:04 And so, yes, I 100% agree on this idea of compounding engineering or, or whatever you want to call it, of you. You do always want to be working on the thing that builds the thing. And if you do a good job of this, then you're sliding scale of like you're spending less and less time actually reviewing individual code and more and more time doing things that are going to improve you. You're basically shifting from like caring about your position to caring about your velocity. Yeah, yeah. From kind of building the car to building the factory.

40:30 Right. You're sort of producing that. And how so? So that's a really good point, though. I think one takeaway from what you're saying is like, yes, you should improve the sort of the context that guides the factory and the production, but you should also remember that one of the primary, if not the primary context that eventually guides the factory is the actual existing code base. And so if that code base is full of slop, then then the reality is that that will be a pattern because the agents somewhat legitimately default to mimicking the, the, the patterns that they see, that they see within the code.

41:05 And so if you if you let that go sideways too much and it's sort of a counter, a counter view to, to to saying that tech debt is deflationary and that you can accumulate that today. I mean, that would maybe be true if literally every pull request that you're creating, you don't take in any code, you just sort of produce it from scratch. But that is not realistic because you don't actually have that. So, yeah. So really good point. Maybe.

41:34 So going a little bit more future, looking a little bit and think about what are the artifacts that might stick around.

The evolving source of truth in software

41:41 So like how long do you think code will still be the source of truth. Like we're talking about rebuilding, replacing, editing increasingly, but we still end up kind of capturing the actual decisions in the code. Do you foresee that two years from now that is that is still the case, or do you think new artifacts emerge that are the primary and the code is more like the bytecode that you've produced, right?

42:09 Yeah. I don't I don't want to predict the timeline. I think a lot of people's predictions have been either way too way too conservative or way too liberal. And, you know, a lot of work with, you know, fortune 500 companies, publicly traded companies. You know, large private companies. And like the, the, the the reason why they love working with us is because we're like, cool. Like do not mortgage or code base for the future.

42:34 Like, yes. Like there may be like a 1% chance that you miss out on, you know, some crazy, crazy upside. But we are going to let you move as fast as possible without without giving up control or giving up taste. I was at a dinner in San Francisco probably six weeks ago. And, you know, we were talking about software factories and loops, engineering or whatever we were calling it back then. And this question was like, cool, raise your hand of AI.

42:59 Is writing 100% of your code like, everybody hand up, raise your hand if you're still reading the code. And it was like 50% of the people. The other 50% have been like, no, I made 70 linter rules custom by hand, and I caught every single anti-pattern and whatever, whatever your approach is, that gives you the confidence to ship it. But then the question came is like, how long until we don't review the code? And I was like, yeah, maybe one two years. It's possible that that could be the case. And then there was a really interesting question.

43:26 I was like, do you ever think there will be a time when you don't review anything at all? And, one of the founders there, who is a veteran of a lot of this stuff, kind of said one of the most interesting things that I think about almost every day now, which is like there will probably always be alpha in reviewing something. We don't know what it is. I can't say what it's going to be.

43:51 It's probably going to change over the next two years. But if you don't review anything, then you're just letting the AI cook and then you're going to get the same product, the same email, the same company that everybody else gets. And so there has there will always be something if you want to rise above the median. And not just I think I think like the, the generic like token form of just like, hey look, yeah, we spend tokens and we create value is like very undifferentiated.

44:21 So I think, I think there will always be a place for smart, creative, thoughtful, caring humans to be reviewing something. For now, I think it's still the code, but eventually it might be something else. I don't know when or what it's going to look like, but there will always be something to review, and there will always be a thing that you can add to the add to the picture. Yeah, yeah, I think we're sort of seeing a lot of that with the with like a Claude presentations, you know, for a while, like, you know, everybody was amazed by how, how, how great it was that Claude like, at the breakthrough, like it was creating these good looking presentations.

44:52 And now every single presentation looks like a Claude presentation. And you, you just sort of yearn for for something that looks a little bit different. But how. So so I agree with that. I think I think you have to differentiate. You have to capture those. I do muse, though, about a better way to capture preferences. And I, I think a lot about sort of outsourcing. Right. You have a lot of like great companies that have sort of outsourced their whether it's their sort of sort of software stacks or even more so, like a lot of manufacturing companies that they don't actually manufacture everything they design the thing that will be manufactured, someone else is producing it.

45:33 And so they they really kind of define the outcomes, but they work with these entities. I guess, does that analogy kind of resonate with you, which is like create artifacts that document preferences, opinions, taste? And, and for the most part, outsource implementation to the agents. Yeah. So two things. Yeah, I think I think absolutely like people say, oh, the new mode is taste.

46:02 I think that's really blurry. But like I do think it's safe to say like the most you can have is understanding the problem better than anyone else. Like knowing what features to ask the factory for. Understanding your customers better than anyone else is incredibly valuable still. And that's again, that's that's where your you can context engineer the agent in a way that none of your competitors can. If you understand the problem better. I'm about taste like part of me feels that taste is just a preference that you haven't bothered writing down.

46:31 You know, it's just like a know it when I see it type of thing. Is, is, is something that just okay, like get the agent to see it often enough and it will capture it. It's still it still is a differentiator. Like it is still the case that if your written down taste is better than my written down taste for some purpose. And so for those purposes, your product or your output will be better than mine.

46:59 And so there's still there's still value in it. I still think taste should be codified. And I think still it is still think that it is the differentiator. But it is it's not this elusive human versus AI difference. It's just a voice choice piece. Well, what what about you, man? Like what? What do you think the future of reviewing things or building or factories? Like what what what spicy takes to you have or or mild takes?

47:27 I don't know, like, I think there's, there's, there's the market for spicy takes his way to frothy right now. And I actually love a more middle of the road of like look yes I think this is directionally correct, but actually like it's not as crazy as everyone says it is because everyone is incentivized to, you know, engagement farm on the internet and you can actually you get paid money if you get people excited about something, regardless of regardless of whether it's true or not. But what are your spicy or or otherwise takes on, like where this is going?

47:54 I've got a fairly kind of a skill or context centric view of the of the world. Like, I think the fact that, like, I think software engineering is a creative profession. Like, you are creating software and the fact that it has engineering in the name is a mistake of history. Like it just happens to be that that you had to be an engineer. You have to be an engineer to be able to create those. And really what we want is to be able to capture taste and preferences and balances.

48:23 And I think those will be the kind of units of code, if you will, that you are, that you are building. I said, this is my definition of what secure is, of what good design is, of what even the specifics of subsequent like. You can still get granular or not in terms of its preference and even even some elements of like how controlling you are, whatever. Whether you're Apple and like you want to design everything down to the sort of the last bit and gram, or whether you're sort of fine with some more sort of generic OEM component in there.

48:53 And so those are your choices, and those units will still be developed in the sense that you still need to know that which ones you are using and which ones depends on what. And you still need evals and tests to be able to know that when you modify them, they do not regress and it's still need to observe them and sort of see how they work and duplicate them and align them when you have multiple team members creating them. And so like, I guess my belief is that software development would revolve around skills and that the role of code will be will diminish over time, that it has to be sort of duplicate.

49:24 I don't know when it goes away. It's more about, we've referred to at the beginning of Tessl, we've had the sort of the notion of a spec and a shadow spec, sort of the spec being the thing that you create and this sort of shadow spec being the capturing of the decisions. What is the difference between the shadow spec and code? I don't know, it's like degrees of freedom for it. And today you already do that, right? You already use Java and you're delegating garbage collection.

49:49 And it happens in different ways on different systems. And you don't care because you trust that it is good enough. And so so I think it's sort of going into into that world. And the conversation we have right now is really about what is the journey. Towards it. Right. You know, and what should you build today.

Toward open and composable dev tools

50:07 So you're not, you know, left stranded for tomorrow, but that still actually gets you somewhere today. That balance has been tough. Yeah. And so I won't I won't say software engineer. Maybe I'll just say builder. But like, as a builder, as a person who wants to create things of value, I think right now your, your two biggest bottlenecks are how do I let the model pull as, as efficiently and easily and like, mentally like pleasant for me the human.

50:35 How do I enable the model to brain drain me as quickly as possible on all the things that are my tastes and preferences and things? And then I think, I think we're in alignment there. And then if you're if you're still reading the code, there's like a corollary of what is the fastest, most efficient, most visually effective way for a model to tell me what it has done. Like, yeah, maybe I might not read every line of code. But, between if I'm reading every line of code now and someday I'm reading a spec like between there, there will be, you know, ladders up of like, cool.

51:08 Don't tell me the code, but give me, give me something that is, you know, 50% faster or 80% faster than reading every line of code. That gives me almost as much confidence or almost as much understanding. How can you what are the like patterns where we can leverage? How do humans think and understand things to both get that into the model's context as quickly as possible, and then let the model context engineer the human of like, what's the fastest way that I can explain to this person what has been done so they can be confident that it works?

51:36 Yeah, yeah. No, that's that's well said. So maybe one last question a bit along those veins. I think the, the advice you give today is different than the advice you gave 12 months ago. Right. Sort of seeing the the evolution of the surrounding. What's your bet on what aspects of your advice will change 12 months from now? Yeah, that's a great question. And there are some things that we got right almost, almost 18 months ago that I have not changed. Right.

52:03 Like the context engineering thing I think will always be true first, as long as we're using, as long as we're using transformer based LLMs. That's just like the math of how they work. And you will get better quality if you use less context. So you'll always be able to like engineer things to get engineer things to get better results.

8 Highlight · Software factory stacks will decompose into open components

52:20 Will change in the next 12 months, I think. I spent a lot of time thinking about infrastructure. I think the entire software factory, this is going to be like AI advice, but I think the, like, entire software factory stack will decompose into good interfaces and open components, kind of like what we saw happen in the Kubernetes world in the last, like, you know, ten the last decade of like basically like the more open and portable you can make these ideas, the more likely they are to get broad adoption, and you need a certain amount of momentum to actually become the default solution.

52:59 Just like what happened to VMware, just like what happened with AWS, what happened with Kubernetes? And so I, I think a lot of these like walled garden, like you can build a software factory entirely on Claude. You can use Claude Code as the harness. You can use Claude automations as your orchestrator. You can use Claude's managed code sandboxes, managed agents, whatever they call it, as your runtime environment. And you can kind of do this whole vertical stack. And I just I think that the advice I will keep giving people today is like, I think dev tools and dev tools stacks are probably going to stay or become even more open, open source, open protocols, whatever it is.

53:41 And so I would I would encourage people to prepare for that.

53:45 And you can build anything now, don't, don't totally know about on the code, especially if you're building critical systems. Amen to that. In terms of the sort of the open composable systems, I had a, early on on the podcast, we had Matt from Netlify, the founder of Netlify, and he was kind of making this passionate case, which I relate to, which is that the web, you know, like had elements of it that was sort of closed that eventually the sort of the open web one.

54:13 And I think we're all better off for it. And I think similarly, in the kind of the world of agents, the closed echo systems are very tempting because they they naturally align and all the different pieces can come together on it. But, you know, sort of this belief that eventually open and composable leads to greater innovation and greater flexibility for organizations. And so I think this composability will will win over the ordering. That's a that's a whole different ball game.

54:39 Yeah. What what's what which chips are going to fall in what order. Or dominoes I suppose. And yeah. How how much of, how much of a lead do these sort of frontier? The factories, the devons, the OpenAI, the Anthropic of the world. How much of a lead do they actually have? And how how it's like for them, it's like they are they're going to war every single day. Because I think that open is catching up faster than anyone is ready to notice yet.

55:10 Dex, thanks a lot for the great conversation here. Super, super fun. And, you know, deep insights. Also things in general for like sharing the whole journey here and the learnings and the practices along the way on it. I think we will learn as a community. And you're definitely sort of a chief amongst the educators over there. And that as much appreciated. Oh, and thank you man. I am huge fan of of of you and the team. And you guys were super early to this game. And you have I mean Patrick is like been a fan boy of his since he invented DevOps days in like 2009 or whatever.

55:40 So you guys clearly have a really good squad. And like I'm excited to see what you guys are cooking. Thanks and thanks everyone for tuning in and I hope you join us for the next one. Yeah. Native dev is brought to you by the package manager for Skills and Context. Your hosts are Guy and me, Simon Maple. Our producer is Tom Dowler. The AI native. Dev is not just a podcast, it's a community. And we host monthly meetups at the Tessl offices in central London.

56:05 Visit Tessl IO forward slash community to learn more and I hope to see you there.