The Centrifuge Method: Spin the Stupid Out of AI

9,370 words · about 37 min to read

0:00 whatever I produces should be good for the first try and never, never the case. So how do you get to quality. You refine. this thing is only good at doing one thing at a time. you have to almost give it the mission to disagree and fight that in your rules. So things is like, hey, tell me honest things don't lie to me. Don't try to please me. So when I started doing that, I noticed that I was like, oh, it's less nice to me. like I call it like a centrifuge. So basically that is like you spin so fast and stupid.

0:27 Comes out. Yeah. The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Podjarny and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

0:53 Yesterday you were writing code. Today you're prompting agents. Tomorrow you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads and VP's packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub, cut through the hype and get to what actually works. This November, we're bringing it back to New York.

1:20 Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers. Whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. Our early bird pricing ends on September 30th, so make sure you lock in your place before prices increase. As a podcast listener, you can get 15% off your ticket using promo code POD15.

1:47 We have so much planned and we can't wait to have you join us in New York. Hello and welcome to another episode of the AI Native Dev. And we are shooting here on location in San Francisco. We actually wanted to have a session with Lada Kesseler a number of times, actually. We tried to schedule it a number of times, and we bumped into each other in San Francisco at AI Engineer in San Francisco.

2:15 And so we thought, hey, why don't we just do this in person? So welcome to the podcast, Lada. How are you doing? Great. Thank you so much for having me. Oh, it's an absolute pleasure. Now, Lada, you are working as a principal engineer at Logic20/20, right? Tell us a little bit about Yeah. Basically I'm working at a contracting company. It's kind of a niche one. But nowadays we're all over the US and I get to go do a lot of various things.

2:41 Right now, I'm trying to do everything in my power to make sure that they are about a. Yeah amazing. And you, of course, gave a session at AI DevCon. Which one was that? That was in New York last year, in November last year. Really loved by all of our attendees and got some super high scores and chatting with Patrick afterwards. And it's like, we have to get Lada back. We have to get Lada back. So. So yeah, we'll see what we can do for for this year.

3:07 But it's wonderful to have you on the podcast. And by the way, we are. Today is the 1st of July. It's currently 10 a.m. just gone 10:10 in San Francisco and England are playing live and they're playing so poorly it's actually one nil to Congo right now. They're playing so poorly. We saw that's not what's the second half here. Let's let's have a podcast that would be more interesting. But we'll keep you up to date on scores as we go. Yeah. You're still watching.

3:32 Though I well we have the kind of score there, but we're not watching the game. That would be rude. That would be rude. So, so, so tell us a little bit about your involvement with AI. How how how are you using AI today in your day to day or your hobby Pretty much completely at this point really. Yeah. Not just software development.

Integrating AI across notes, code, and communications

3:52 So I do pretty much everything with AI, or the majority of things I do, I do it, with AI, so like it starts with code obviously. And then I do a lot of code that like the systems like code have potentially some agents in there as well, but not always. And what I'm actually passionate about is figuring out how to how to get the coding bit right. So and I find that oftentimes when you actually like working with a coding agent, you also get much better intuition to how to write a system.

4:21 So I think it's very, very much linked. But then things I do on top of that are like, so I do a lot of notetaking with AI, but like, I used to be very disorganized and like Obsidian and things like that didn't work for me very well. But now I have basically a folder where I have a Dropbox folder where I have my notes, and it has a system of organizing things for me. So I'm like, hey, process today's notes and goes and like, okay, ask me some questions sometimes or like organize it.

4:51 So now I'm like, oh, so well organized. I like. That I've totally just built this for myself as well. So I actually connected it with Slack, Granola notes. And I have completely opened up now. It knows everything about me and it tells me what to do. And it's like, what should I do today and things like that? So it's good to see it's not just me. So far. I'm restricting it from accessing my email. I'm trying. I'm trying very hard not to go this way because I know it's temptation is there, but.

5:16 I even allow it to write some of my emails. But I haven't yet got to the stage where I'm comfortable with it sending at all. And when it does write my emails, I tell it what I want to write and just words it for me. And then I go into that and kind of like that. But it's it's curious the path that we're all on here and how much we rely. You know, we're relying on this to almost be our you know, we initially kind of like focus on the code that we review very heavily. And we're probably going to see exactly the same levels of, of validation and verification in the, in the way that we present ourselves as well.

5:48 So if I write an email, that's me talking to someone else, and they're expecting that email to come from me. Right. And so I want that to be validated to make sure it's not just AI slop that's being sent, otherwise it's making, you know, it's giving me a bad reputation by me sending exactly shocking content to someone else. And I, you know, it's almost rude for someone to for me to expect someone else to read some AI slop that. Absolutely. I've generated. That's very rude. So let's talk a little bit more about that, because by default, agents aren't always the best at getting at nailing something that we, you know, want.

Configuring ground rules to override AI agreeableness

6:23 So how much does like the enablement of agents matter. Basically I think about it as like AI comes with defaults that are not very helpful. And you, you kind of have to adjust them in your systems. So one way to do this is basically so they all have like this thing that I call ground rules. Basically this is CLAUDE.md or something like that. And you can basically define here's my here's here's the things I want most important for me.

6:48 Like you need to be careful to not just put everything in your history and your TDD process, and you like everything else in there because it also has to not enough memory, right? And your ground rules there when you talk about your ground rules, how much of that is specific to the projects versus almost like more your organization or your preferred way of development that you can just share across all your projects? Right, right. So I have both levels, right?

7:14 So first of all, my my my big one, my my big one that is applied to everything. This is like the ground rules that are fighting the most, like the most problematic things that I see across all the AIs, no matter where they are. So several things.

1 Highlight · Forcing AI agents to disagree and be honest

7:28 So for example, if you think about like so we're dealing with a black box, we don't know what's inside. There's a mental model of some sort in there. When you talk to it it understands something, but it's completely invisible. You don't know what's going on. And it's also like trained on purpose to please you. Right. So what I have a lot in my ground rules is like, hey, you actually like, you have to almost give it the mission to disagree and and and fight that in your rules. So that's what I do.

7:57 One of the things is like, hey, tell me honest things like, don't, don't, don't lie to me. Don't try to please me. Does it does it follow those or does it almost like it was like that? It's a little like. It's much better. So. So when I started doing that, I noticed that I was like, oh, it's less nice to me. It feels less nice like this initial, like it's actually being like, oh, you're so great and so on. Like, I have allergy to this to like, I, I think I have I need therapy for.

8:24 This at this point. But but like I noticed the difference. It was it was substantial and now it's so I should say, like, before the latest Opus. It was pretty great. Nowadays it also tells me that it's being honest, which is really annoying. It's like. Oh, it's honestly speaking like. Oh my God. Stop doing this. So yeah, but and then I have productivity rules, and productivity rules are basically, hey, if you notice some problem, tell me now and don't actually commit to the thing. I said.

8:50 There's like checkpoints or does it naturally do that as part of everything. So so two folds right.

8:56 So first of all it's in my ground rules like this is like basic like here's our contract, here's how we work with Lada. And and also I have to reinforce it or I have to do checklists when I do the work on time. And I kind of feel like I need both layers depending on the size of context. The second thing helps a lot as well. And where do you put all that actually, did you mention you put that in your CLAUDE.md or. Yeah, the ground rules are better ground rules. I include CLAUDE.md in the user level for everything.

9:24 Yeah, but like my my CLAUDE.md, I know every line there. It's 72 lines and I, I'm very careful about how much stuff in there. Right. And then unlike system prompts that it feels like Anthropic just just puts a lot of things on the system prompts. It's a bit. Yeah. Not not great. But yeah, I also have things on like a slash commands. We're just invoking time, right? I sometimes have some things on the project level, but I'm careful what I put there again. And I'm more like, here's a menu of what you have.

9:52 And then here's where you discover more similar skill, like right somewhat or. Something like that.

Implementing TDD through lightweight agent skills

9:58 And, and it's more of a progressive disclosure in terms of skills. What kind of what kind of skills do you feel like you rely on most heavily in your day to day engineering? TDD, I think is the. Biggest one. So why is TDD or not? TDD is presumably a one liner in your CLAUDE.md and then oh, it isn't so. So no, it's a skill. No, no. It used to be a slash command. Yeah. So so things like so many people are saying, oh, TDD doesn't work with AI.

10:24 And the way when I ask them, how are you holding it. Like and they're like, oh, we just tell it to TDD and it doesn't cause it like it doesn't work, right. Well, it works really well. Is like, what, what I, what I have is basically I process, hey, do this, do this, then do this and this, and basically I just what I did, how I got to it is I basically TDD with it for, for like a session. And then I was like, hey, record everything we just did. And like I micromanaged it at first and then record everything they had.

10:53 Now I have it outside. Now I say, hey, clear session, let's TDD and I watch everything where it messes up and I correct. So you you ask it to TDD. I asked to. Do TDD. Okay, so I was expecting a one liner in your CLAUDE.md that says make sure you always do follow the TDD rules and then a skill that explains the rules. But if you if you're explicitly invoking it to your control. No no no. It's all a bit of both. So I have like basically I have some rules about code, but not too much in my in my main thing I mostly like I like simple things.

2 Highlight · Using lightweight frontmatter to trigger agent skills

11:27 If you can do things more simply prefer that we prefer that to hacky solutions and stuff like that. But with TDD, basically I have a skill, right? And the skill basically has this, this little front matter on the top. Right. And that's the thing actually that's written for the agent to for it to know when to invoke it. And the whole point is like unlike the old MCP that just, you know, puts all the instructions for itself in your context and just garbages it completely.

11:52 You have just this little thing over there. And then the agent sees it and is like, oh, I have a TDD skill. I'm supposed to do it every time I write code. Oh, so basically I have that and that's what makes it automatically do. That's right. That's that's the activation. Right. And that's the bit I think the description is the piece that most developers, when they look at a description, they think it's almost like that's for humans. And it's absolutely not for humans. Yeah. They're very lightweight description. And they completely mess up how that skill then gets activated.

12:20 Exactly. That's how the agent determines whether it should use that skill or not. Exactly. And when you have many, many skills also there to. To fight. To decide. And if they're very generic then they're going to overlap.

12:32 So so much I got no chance. Precisely, precisely. That's why I have like a skill factory basically that focuses on that explicitly because I, I find that Anthropic default one doesn't work very well. So and, and I it's like I see like many lines of in front matter it produces like all this kind of stuff. It feels like it's not it's not good. So I basically have like a one of the processes in the, in the skill factory is like, hey, iterate over this and in this context, like, this is what it's supposed to be like and try many times and like go into that direction.

13:02 Is it good enough? And then it shows it to you and it's usually much better than the defaults. So just iterative way of. Making sure. Yes, yes. It's concise. I call it think into a file. So basically this is a refinement loop.

Refining outputs iteratively using the centrifuge method

3 Highlight · The Centrifuge Method for iterative refinement

13:14 This is just super powerful. So the process goes like this. You basically have a goal, some goal with AI, doesn't matter what. Then you have it take one step towards the destination you want just one step, which is really important. And then you commit or you put it into a file. So for example, like I actually write with AI and I'm scared to say that because so many people are making a point to say that they are not writing the the I, because I think it's expected that whatever you produced at this point is, first of all, not your voice and slop.

13:45 Right when you say write, like author. Like write something, I don't know, like for things, for conferences, for. Example. Like a little blob for myself, I would I would write it with AI and and result is much better. But like it's so much my work. Yeah. There's so much more my work there. But like the way I get to that is like, hey, here's the thing, draft the thing, thoughts like maybe something I don't like this, I quite like this go into, right. Go think into a file. I call it go meditate. Yeah. Goes. And basically it tries the first iteration to a file.

14:12 Then I say stop, read it back. Here's the goal itself. Is it good enough or there isn't something to tweak. Write it back and so on. So I have like it goes on like I call it like a centrifuge. So basically that is like you spin so fast and stupid. Comes out. Yeah I love it. Love it. Yeah, that's really interesting. And I think actually I do I do exactly the same from the point of view. I use it to write a lot, and I feel guilty as well when I'm like,

14:39 you know, if I send this to someone, do I need to de-AI-ify the, the text so that it it feels like someone really thinks I wrote it myself. And you know, the it's the guidance that we add in and the direction that we add in and the reading it and telling it, no, this is wrong. That's right. And that's what is the important piece rather than the fact that we're we're writing every single word or not. And I think there's a, there's an amount of us as humans that need to kind of like get over that, that, that mental barrier or that.

15:09 And I think a lot of people, if they if they say they're not doing any of that with AI, they're probably lying to a little bit more in terms of. Maybe, maybe they're trying to make a point of not doing that for reals. And maybe that's a good thing. I think there's just like this, this expectation that I, I see a lot of people having, which is like, whatever AI produces should be good for the first try and never, never the case. It's just not the case, like, so let's stop expecting that, right? So how do you get to quality. You refine.

15:35 So this kind of processes of refining it together with human input a lot of times. So I have basically a lot of centrifuge and okay I don't quite like this. This doesn't sound good right. Go go meditate again and so on. And after like five loops of that, I'm at a place where it actually sounds much more like me than I would have done on myself. I think. I can, I can hear in Tom's head right now he's thinking, how do I put Simon into a centrifuge right now to get the stupid out of him?

16:02 But that's a that's another question.

Focusing agents on single tasks and verifiers

16:03 But let's let's talk a little bit about in order to get the best out of AI, it's super important to understand the problems, right? Understand the problems, and working around those problems or circumventing those problems will help us get that better solution. So when we think let's let's think more about coding now, what would you say are the most common problems in and around coding agents that we should understand better so that we can actually produce a better output? I think there's twofold.

16:30 So I think so going a little bit back to what we discussed about like the problems with AI. So there are some problems with AI. And I found that there's some problems with mental mental expectations of people around how AI is supposed to work. So like a lot of things I see right now, like especially new people, when they come into it, they're like, oh, there's ground rules. I'm just going to put everything I care about there. So let's just put my TDD process. Let's just put all my like, best practices and do this and make sure not to screw up and make no mistakes and do this.

17:00 So things that thing doesn't work this way in practice. Like I'm not training models. So I actually I'm getting more and more curious about how they work internally, but it feels like they have a concept of attention. And that's probably the biggest thing that I see people not understanding. So even like yesterday, that was an interesting, interesting, really, really good talk from HumanLayer, two talks from people. I really like them both.

17:26 And one of the things that I was disagreeing with was that the person was like Dex, not the person. He was like. AI if I knew how to write good code, it would have done it. And I think that's so wrong because what I see a lot is basically so people are trying to say, hey, write, write me, do this task for me.

17:54 First of all, this is a task. And also AI, hold my standards, and they add a lot of things on top of here. So in my experience that doesn't work. And I think why doesn't work is because this thing is only good at doing one thing at a time. So you can't expect to just enforce your standards like this way. So what you do instead? It works amazingly well. And like he was basically, here's something that he knows so much about this.

18:21 Like right. But he's like, if I knew how to code right, good code, he would have done it. Thing is, if you go and ask AI after it read something to find all the problems with it, it will. And if you go into different directions, you can get so much better.

4 Highlight · Using deterministic verifiers to catch code smells

18:37 Like reports about what's wrong and you get it to fix it there, but you don't do it in one go, you do it in many goes. So this resonates so, so much. Right? And it actually reminds me that. So this week we actually launched Tessl Agent. And, and I'll mention this for one reason that there's a cool, really cool thing that I love called verifiers in Tessl Agent. Yes. And you can, you can use Tessl to kind of like kick off code reviews.

19:03 We have a basic we're not basic. It's a it's a reasonable code review in there as well. And so you can use it to kick off your code review. In your code review, you might have 100 things that you're trying to check for. And if you say go and do a code. Review. It'll it'll do a reasonable job. But yeah, exactly. What if you look through every single piece, you might think, oh, actually, you didn't do this. Whereas if I was to say to an agent, just do that, it will do it and it will do it.

19:30 Well, now a verifier is very interesting because it takes a very specific thing and sometimes triggers deterministically on various files. It can be on a certain file to do this on or whatever, and it will say, did you, you know, can you check, can you verify that you're adhering to these standards? Maybe it even points to the standards in the in the skill, and it will go back and it will do that. And when you have each of those verifiers that are all doing that, you actually get a far, far better results faster as well. Yes.

19:57 Because it's, it's the verifiers are, like I say, deterministically triggered and they run very short, kind of like processes. Really, really interesting. That really backs up what you're saying. That's like very similar experiences. I think I'm going in the similar direction with friends, like we're doing something similar. So we have deterministic triggers that detect some code smells like long method or dead code. And then it triggers something and then that that thing can be pluggable. So it can be like hey, plug an agent to that to be like hey, here's a prompt to do this for long methods.

20:27 Here's a long method that make it better. And here's how recipes so one focused do this. Or it may be invoking a linter or a formatter. Right. So I'm trying to do as much automation there as possible as well. But yeah, and focus I think is. A big one of what you said. Yeah absolutely. Yeah. Let let's now maybe turn more practical. And if we think about, you know, various

Markdown sketch prototypes for unfamiliar problems

20:50 kind of methods of, of writing software, before AI as well, this is potentially a problem. We would, you know, even my, myself at times would just think, oh yeah, I know what I'm doing here. And I just write, just knock out as many lines of code and you know, so here's the app. I didn't even need to think about it. And sometimes that. Works. Right now with agents. We can do that like that. We can do it very, very fast and just say, this is what I want.

21:17 Just go and do it. But in larger systems, maybe we need to think about that a little bit more, maybe to plan it in architecture. And I know you do a little bit of architecture in your role as well. You play with the architecture as well as engineering. So, what's the what's the right process as a, as a, as an engineer to, to approach a larger, more complex task? Do we just go to the agent and do like a, you know, rapid, rapidly, rapidly developed application first prototype, then do architecture?

21:49 Should we whiteboard first? What's the what's the approach you use? Oh, it depends so much on what's going on. Right. And how much I understand. The problem is the biggest biggest thing here, the biggest variety. Like what would actually do. So for established projects I would usually go talk to AI and like prototype things together. So there's a, there's a whole prototyping session before but and then I, I. Tend. To thinking together or just like knocking an app out and learning. It's a lot of back and forth.

22:15 And I asked it to produce me artifacts. So like I, I really like right now. So it produced me artifacts and I can I can comment on each individual one, and then I can send all of the feedback to the agent and it's much easier. It's like a whiteboard a little bit. So more and more like, I want this whiteboard with agent and I'm trying to build one as well. But like that, that works pretty well on the established thing that I all things that I understand, I also have another process entirely for the things that don't understand,

5 Highlight · Using markdown sketch prototypes for unfamiliar problems

22:42 because I find sometimes it's like, oh, we're trying to do this thing. Neither the person I'm doing it with no me have a good understanding of the problem space yet. So not let's not even just go coding at all. Because like when if you start coding, you're like over committing to a specific files, just this one and you blind to the others varieties and you're like, you're so focused on solution, you're on that you like forget about like what are you actually trying to do? And maybe there's a better solution over here.

23:09 Maybe it's a completely different solution that wants to to come out, but you're not looking at the problem yet. So I see so many people looking at the jumping into solutions before thinking through the problem. So one thing I do with some people that was pretty fun is like I call it basically like a sketch prototype. I don't know, it's markdown. Basically, I take my code and I replace it with an agent and in no way put it into production, please. Ever.

23:35 But basically this is for very draft. So basically and the code is instructions in a file. So my whole system is markdown file plus agent. And the agent goes through processes and I get to experience the user experience from that. And I learn so much. So basically then I actually commit to code or like go into some different directions. That's been working for two. Well, in some. Cases. You are sent to the user trying to. Yeah, like to. Understand how. The user would go through that. And then from.

24:01 That, and the agent basically can be a stand in for code, for any code whatsoever if you want to. But like it won't be stick won't be good.

24:08 But I don't need good. At this point. So it's almost like you're mocking the you're mocking with an agent. Yeah. It's like a pretend. It's like your paper prototypes you used to do, but it's much faster and much easier. Yeah, yeah. Very interesting. And when obviously, you know, from an architecture point of view, a lot of people talk about and myself included, talk about how developers are kind of more going to be driving teams of agents and moving more into the, I guess, the architecture style role. Why can't an AI do good architecture?

24:36 so it might on a scope of like one specific task, do a pretty good job. But what I find is like there's two fold problems, right? First of all, oftentimes we are not able to keep up with it. Right. So question is like many people are doing different things, but many people don't read code and many people are, I think, finding a vacuum in their heads, like where are they used to be a problem in solution space. There's now nothing so so but but I know that if you just let something happen, it will probably degrade.

25:06 You know, we live in the universe where entropy is a thing, so and AI kind of does a decent job. Maybe if you do the the iteration on one task, but then it can't hold everything in its brain, it can't do all the things all at once. Right? We discussed. Right. So maybe you can iterate and like I haven't figured this out with it, I definitely so, but maybe you can like run several iteration loops and think about this, think about that on this level. And you have to do it on different levels of architecture that you can't just do it on one.

25:35 Yeah, even like on this scope, it's too much. So it's a problem of complexity management, I think quite a lot. And also like holding this mental model and I don't know who hold this mental model right now. So I, I kind of. Have to. And, and you see that long term short term when when like how how I know it's really hard to actually it's a it's a tough question because it's impossible to, to really guess what's going to happen in three months.

26:01 But I'll ask anyway because I can do that as a host. Like, how close do you feel we are to AI actually being able to do a good enough job? Which is the danger, right? Because as soon as it can do a good enough job. Yeah. I'm worried. About good. At the time. It'll, it'll try it. And do it. All right. I have no idea about timescales and what will happen. It feels like we're building a system that builds other systems, right? Yes. Yeah, it very much goes. And it feels so tempting and it feels so interesting right now.

26:28 It's like the most interesting problem out there, I think, at least for me to build that system. And like, can I bake my engineering practices into that system. So it does all that stuff for me. So I actually built things, but on a higher level, I think that was an allure that's been like people were trying to get to for a long time. I think it's actually might be reachable. I don't know if it's reachable. So there's some problems that I think are unsolved right now. Will they be solved? Maybe.

26:54 Especially if we find something else that works much better and can actually learn as. We go. Hey everyone! Hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests, so we can have the most informative conversations about agentic development, whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel.

27:19 If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode. So a quick game update. Harry Kane just scored in the 75th minute.

Reverse direction and use AI for vision

27:42 So it's one all England still have a chance. Go England. Let's get back to to practical matters. Let's talk about maybe some other tips of what you think developers should be doing. Could be doing that they tend not to naturally do in their agentic coding. Right? I think what I see a lot of people are like, I think it sounds like a conversation, like you have any conversation, so you kind of behave like in a conversation.

28:10 So like it asks a question like, oh yeah, sure. Let me answer you, AI. And I feel like many people are not thinking high level and like, hey, you don't have to actually go where it asks you to go. You actually can steer quite a lot. So I call it reverse direction. You basically decide like you're the decider here. You don't have to uphold, and also you don't like something that the eye does like. It gives you a list of like ten, ten questions. And some people sit there and answer every one of them.

28:36 This is so painful. Don't like one one at once or with all the the the files much punchy there. So this kind of steering, another thing I see people not doing is like using it almost like instrument of vision I find, because like what I do is it a lot is so I was just at a Craft conference and Gojko Adzic, I hope I didn't murder his name horribly just now, but he he has amazing talk, and one of the points that he was making, it's much easier to complain than to explain what what you want.

29:14 So like seeing something helps you understand what you. Want and it's just much, much easier. So like have it show you something and then it's much easier to see what you're not like. You get more ideas. You you can borrow different ideas, combine them,

6 Highlight · Prompting AI to reveal paths not taken

29:29 and you can go into a different direction entirely. But just just doing that and a lot of it is like I think about it as like, imagine I'm at a crossroad with many, many, many, many paths before me. Right now, they're invisible to me. But AI can make them visible to me at zero cost. Like show me that are in front of me. One thing that I really like is like, when you have a choice that you have to make, give me a choice.

29:58 Like choose yourself, show me with a little emoji. Like I have a star emoji. This is a chosen option versus simply and then chosen. Choose me. The paths not taken. Yeah I. I, I was doing it for a long time. Kind of manually. But then Arlo Belshee showed me this trick that he's doing it in one move. So basically this is like four patterns at least applied all together. And then then I can basically like, oh, this is the decisions before me. I want this one. No, no, AI you're actually wrong.

30:24 This one is good because blah and so on. So I can go and do that direction now. And I actually broaden my horizon. What I don't know before me looked into that briefly.

30:32 Not good. Yeah. It's very similar to kind of like we're doing a lot right now thinking about sorry to bring it back to Tessl, but and Tessl and we had this this change risk. And the key is it will determine or it will try to determine whether a change is high enough risk to be to be raised to say, hey, this needs human review or whether it can is good enough to do a review. And I think it's it kind of like makes me think about, you know, we use LLM as a judge for a lot of things and it's it's almost as it builds.

31:06 It might be nice to essentially say, hey, this is a this is an important decision that I think you need to make. This is an important decision that I need you to know that I have made.

Rigorous testing prevents agent hallucination and lies

31:15 And you just need to kind of like, look through it, check it. Out. And say, yeah, I'm fine. With it. Precisely. It then avoids that. Almost needless at times, you know. Back and. Forth and. Forth and that absolutely. Much interaction. So. Yeah. Exactly. So I run into this a lot as well. So like a lot of my conversation have started to seem like the same thing. Right. So what I have like a defaults right. So like my default technology stack is this let's just do all of that every time.

31:41 Let's not just decide it every time. Right. And also like when you say AI review versus human like why not both right. So absolutely get AI to spin all the stupid out and then do human and, and make sure that AI presents it in a way that's easy for human to follow because it's not easy right now. So in that case, is there almost like an do you feel an overload of review? Yeah absolutely. You almost like I. Don't I don't allow myself to go there. Right.

32:07 I don't let it. So so what I do is basically there's two folds. I think it's important to be able to answer a question. To me it's important does my system work right. And for that, first of all, I have to ask. I have tests on two levels. I have I have TDD tests. Right. So for the agent, this is a cross-check against reality. So this is not letting it go too far from reality. And it also is helping in first of all, it produces pretty good code, surprisingly.

32:37 So like I would love to go there a little bit, a little further about that. And then it also so it also makes sure my tests are not completely horrible because like when you have AI, right? Tests a big danger. Like I trust my tests, my tests even less than I trust with. My code. Because it absolutely cheats and lies, does horrible things there, and comments out tests, which is like a crime scene. Like it should be punished by something bad.

33:02 So okay, I have this TDD style. This is for the agent mostly, and for code quality and so on and check against reality. And then I have to say about does my system work right. And I want that high level. I don't want them like I don't even want them on cucumber level because Cucumber feels too too detailed for me. So what I do a lot is like I write BDD level tests basically, right. Like a little whiteboard domain language. So like how do I think about this problem?

33:28 Make it extremely scannable. So I can look at my tests. Are they complete like and I see things like depends on the domain. But like if I do API, I see here's my API, here's what happened. That's what API returned for. And it's all like a bunch of tests like this. Or if it's like more visual, like it's a game. So somebody made this move. This is what happened. So I want to see that. And I want to make sure that this is not something that agent can edit very easily.

33:54 But this internal stuff, he can edit as much as. It can. And your your style test is that is that kind of does it use a similar Gherkin style. Approach to. I use pytest, pretty much. Is it stable? Yeah. No, absolutely. It's a 100%. So 100%. When you read it, you can rely that. Those. Aren't going. to be a lie to me. Absolutely. That's crucial right. Yeah. That's crucial. Yeah I, I use approval test for that.

34:19 Basically this is like a good technique to know for, for kind of legacy code. If you're trying to pin the behavior, you basically can, can be like, hey, what does this thing return when I do this? And it helps us and you say, oh yeah, it looks good. This golden standard. So approval test, you approve this. There's a little bit of risk to that. You need to be careful what you're approving. But basically you can pin your whole system and then you can refactor inside as one of the ways to use this.

34:46 And I use approval testing for this kind of level test as well.

Building trust before scaling software factories

34:50 It's very easy to define the main languages there. Yeah. Amazing. So we've we've called the low level really nice practical tips there for, for engineers to, to to to listen to and follow. Let's take it one step higher. Now let's talk a little bit about patterns. And maybe you kind of like a little bit more about your environment to how you actually how you actually build. So software factories. What's your what's your thoughts on software factories. Is it a destination for you. Do you believe it's a destination?

35:16 And I guess if you do, where are you on that journey right now to to becoming a software factory worker? I'm not a big worker. I have to be like an architect. Please. Like a factory architect. When I said factory, I thought, oh, a factory worker is like a software factory. The developer is. The agents are the software factory workers. Exactly, exactly. Well, yeah, I don't know, actually, so I think so. We're building the system that builds other systems.

35:42 Does it work like a software factory like we see today? I don't think so, but I think we're trying to figure it out. What I don't like in many approaches that I see right now is they're kind of a little bit of throwaway at I. So basically like, let's create agents to create more agents to great musicians and so on, or let's create agents, let's make agent the right skills to write skill. And like all of that craziness and what I see a lot like it degrades or badly like the output is just garbage and there's a lot of it and there's a bunch of noise.

36:08 So I don't know how people are not, I'm not drowning in garbage in those. Maybe. Maybe I'm just not, like, doing it right. I haven't tried it, to be fair. I've tried a little bit of this approach, like, I, I tried like Claude Flow. Right. And in the problem that I tried many other. So I have like a default problem, like if agents like hey new model try this and I see the result. And it's nice because I have a constant comparisons against and Claude Flow in my experience, did the worst job.

36:35 Of them all. Yes, at the time maybe I wasn't holding it right, but but so. So I see the problem of like if you if you have the building blocks you're relying on that are not actually doing the thing you think they should be doing. Why do you think the whole system work? How? So there's a level of autonomy here, I guess, which is the key. Whereby do you leave the factory to kind of like a build an almost maintain itself, or do you allow the factory to, to kind of make suggestions whereby everything or the most important pieces need to be validated and verified by human to say, yeah, this is actually a good move or this is the direction I wanted to go in.

37:12 So there's probably, you know, levels of self building factory versus I guess you are connecting the factory to, to build code. I have no idea. So I'm really curious about this. I overheard somebody talking about their factory from Gastown, I think Gas City. Yeah. No no, no. There was. Somebody from Gas City and it was interesting. So I absolutely jumped into the conversation and asked like, hey, tell me in practice, how are you actually working?

37:41 Like, how you how do you deal with this? How you deal with that? It was very interesting because actually a lot of things he said was kind of matching my experience with refactoring. So I had my refactoring process that I just pointed things and, and I go in a good direction for me. And so a lot of that made sense. So I'm super curious about factories, what I'm trying to do myself. And I think I will believe in factory that I build myself. while it actually like works with my systems. But what I'm trying to do is I'm trying to get to the building block that I can trust.

38:11 So this is where my refactoring process comes in. Can I have a refactoring process? I can put points at a piece of code, several files perhaps, and it will just do a go away. It does a good job. I come back to good code and I have that now. Right. So that's nice. And I'm trying to kind of can I do this for extracting the knowledge. So right now if I ask I to save something into a file, it oftentimes does a very bad job at it. Like too much noise or over like focuses on something that's not important in broader context and all that.

38:42 How do we get it to extract knowledge. So I'm playing a lot of that stuff. So can I try to build those building blocks. And then maybe I can construct them into something automated that I can actually trust. But I need some level of verification here. And I want to automate as much stuff needs to be code as possible here, I think. So that's where I'm at. I'm not there. I'm this is the side with my process. It works pretty well for what I'm doing, but I'm constantly dissatisfied and I'm looking for. That's why. I'm the conference. That's the key though, right?

39:08 It's it's about knowing your dissatisfied where you want to improve. And then over time it's a gradual I think people who think they can just switch into a software factory. They will drown in garbage. And there. It's all that good. Dream, I think. Yeah. It's a nightmare. Sorry. They're living a nightmare. The dreaming about how fast they can, they can grow. It's about. And I think in order to gain that trust over time as well, it's about gradual increments and improvements.

39:33 That's what that's why Dex Horthy's talk Yesterday was so interesting because he was like, hey, we tried this factory thing for half a year. It doesn't really work. Now we're going back to this and I want to listen to these people. Right? Yeah. The people that have tried, people who got. You talking to factories also. Have you had a guy called Harry Kane? Say it. Again, Harry Kane. Say. The whole thing. Harry Kane. Patrick, you're in the background laughing right now. But Harry Kane is a goal factory and he's just put England two one up against Congo.

40:02 So I think we're like him. Don't we like him very much? He's he's an amazing person. I can't believe I'm. I'm so excited. I'm so happy for you. I thought this was going to be a five nil walkover. But you know everyone's everyone's a potential potential bananas game. But let's let's let's talk about your tooling and workflows. So terminals, IDEs.

Console-only workflows and event sourcing experiments

40:23 How do you like. No no no no. So so like I started very, very early on with agents and like, basically I was a month late to the party. So now like November when the started, I started in December and I was in winter. And immediately I was unhappy because I was I'm in love with IntelliJ or I used to be in love with IntelliJ refactoring tools. Unprecedented. Like there's just nothing compares in the industry, right?

40:49 And I can do crazy things with this. And then I have this Windsurf thing with agent where agent lives. Right. So now I have to switch between this and this constantly, and we're like, no, I just want to be in the project. I just want to talk to you. Why am I this locked into this thing? So, like, as soon as Claude Code which to me happened in May last year, I switched and never looked back. So console only pretty much. I just barely looked at files sometimes and I use ID basically as a.

41:17 Text editor, sadly. Did you do you feel more disconnected from the code as a result, or do you still kind of. Yeah. Look back at it. So you you have. To. Trust to trust things as well. Right. Absolutely. Because and it's trust because it's it depends on all of risk. Right. So what are you dealing with I'm right now. Not in the, the, the, you know, the gory and the bad. I'm actually very much greenfield a lot of prototyping, a lot of other things.

41:42 If I was in the Brownfield I would behave very differently, I think. How would I behave? I don't know, I think I would automate the heck out of that. I would make sure my test suite is something I really, really good. That is super fast, right? And this is where the skills come in. Like what I actually my value is there, right. Provide the like teach the agent to do the test. So that is just like this. And then can I probably then I can actually go a little bit higher level maybe.

42:10 But still I would be much closer to the code right now. I don't need to do that, so I don't. Let's come close to to wrapping up. We have we have about I'd say five minutes of injury time to to before the final whistle. So before the final whistle, let's wrap up by asking, I guess when we talk about where you are experimenting next, what are the things that, you know when you leave the conference, you're going to be super excited to, to to to try to play with and explore.

42:43 I'm super interested in so many things. And there's only one me, unfortunately. But yeah. I think. You need to use more agents to to explore and be your brain. Yeah yeah. So yeah, I'm definitely I'm the bottleneck. I'm feeling it like acutely. But yeah, one of the things I really want to understand is how people doing loops and how I'm doing, like, how do I do loops better because I'm doing loops a lot. But like, I think I'm doing it differently than people are doing.

43:08 So I'm very curious to just be exposed to a lot of different ways of doing things and trying them out. Another thing that's been on the radar for a long time is like this event sourcing and event modeling. So I took a class from Martin Dilger a little while back and then unfortunately didn't didn't didn't have time to actually like try it out. But it had a promise of something that was his process is pretty great.

43:34 So like it feels like it solves a problem of keeping. Up and the complexity management all at once. So it's very interesting. There's some trade offs on top of that. But basically what he's doing is he's doing event modeling, which means he thinks through the process and he writes like this little diagram of what's happening. And then each one is like the slices. So basically this piece of behavior does that.

43:59 And then he generates the code for them. Slices are completely isolated, which means there's a lot of duplicated code. But it doesn't matter because it never overlaps. Not ever. Right. And then what does it matter that you created like two objects person if it doesn't overlap ever? I'm not sure how it works in practice. I'm a little bit worried about some things plus like event events. So he does events sourcing which is basically don't don't don't collapse your state.

44:25 Right. You have events for everything. Then you can unwrap and you can do so many things there. So then you have views. Basically for every everything that UI needs, which is kind of constructed aggregate aggregation views and also very many of those. So this like very many of those and which looks like code duplication worries me. But I want to try. It and see because he seems to be very happy. And I think he's he's solved basically from what he says, it feels like a lot of that solves a lot of problems.

44:53 So I'm going to try it out as well. Sounds good. Great, great things for our audience to, to listen to as well and try try out loud. This has been wonderful. Thank you very much. I'm so glad we managed to do this in person. Thank you so much for sharing all your insights and your ways of working with our audience. Been a pleasure chatting very much. Thanks so much. I'm so happy that. England won. I know full time is almost upon us, upon us all, so that's great. Thank you so much. Fantastic. Thank you. Wonderful.

45:20 I hope you enjoyed that conversation. Really lovely insights and tune in to the next episode soon. Bye for now. The AI Native Dev is brought to you by Tessl, the package manager for skills and context. Your hosts are Guy Podjarny and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tessl offices in central London. Visit ainativedev.io/community to learn more and I hope to see you there.