Why AI Coding Agents Are Here To Stay | Patrick Debois

8,776 words · about 35 min to read · YouTube’s automatic captions

0:00 Over the years, there's different standards, different agreements, different terminology. It confuses the AI, much like it confuses humans.

0:07 We're talking about how many times a developer, how many X's in terms of 5x, 10x, 20x productivity gains are we expecting to see?

0:15 What they found that influences actually that X. If the task is more complex, it could be that it's helping, but that it's also requires you to tune and kind of correct the agent right now. So, it becomes a balance between how much effort do I help, how much effort do I actually know? The language on which the LLMs have been trained, is that actually a more popular language? Is it a niche language? So, definitely also kind of influences the code generation language.

0:43 The specs are bigger than code, because if you change the spec, you can generate the code from there. And that very much aligns to our spec-driven development that we often talk about as a way of working.

1:03 Hello and welcome to another episode of

AI Engineering World Fair focuses on native AI

1:05 the AI Native Dev. My name is Simon Maple, I'm your host for the day. And today we're going to be talking a lot about a small little conference which happens every now and then in San Francisco, which is called the AI Engineering World Fair. You may have heard of it. And actually one of my colleagues, Patrick Debois, who's been on the on the podcast gosh, countless times now, uh, visited the the conference a month ago or so, when it when it whenever it was. And had some had some really interesting TLDRs and actually wrote a blog, seven lessons from AI Engineering World Fair. So, we're going to go through some of those and talk in depth in about one or two in particular. But, but Patrick, welcome to the podcast once again. You're a regular here. You're almost You're almost joining as often as I am. How are you?

1:55 I'm great. You can't really tell whether it's me or any AI though, but we're all good.

2:01 Well, I'll we'll assume it's an AI unless unless you tell us otherwise. How's that?

2:05 Yeah, let's do that.

2:06 Awesome. So, Patrick So, Patrick, what what what do you do these days?

2:10 Well, I'm having fun playing around with the whole AI craze, especially now in the space of coding and how it actually is improving the SDLC. I feel like I'm like discovering new ways of working every day. And it's great like, you know, on the community side and everybody's sharing. It's It's It's a lovely time to be alive.

2:30 Yeah. And you do a lot of contributions to the AI Native Dev. One of the, you know, major contributions obviously was the landscape. A tool that allows you to effectively keep up to date, keep track of a bunch of AI Dev tools. That was one of one of your kind of like big projects into the AI Native Dev. One of the blogs that you wrote as I mentioned was the the seven lessons that you learned from the AI Engineering World Fair. Tell us a little bit about the AI Engineering World Fair first of all as a conference.

3:01 It's in San Francisco, not one of my most favorite places that I I tend to go, but let's talk about that as well. Uh What's the conference like? What's the vibe of it like? What's the Maybe I shouldn't say vibe these days, but what's the What's the the style of the conference? Who are the typical people who go? What's the type of sessions that are there?

3:18 So, these days, you know, when everybody's talking about AI, I found it really hard to find a conference that actually is on point. There's a lot of events that slap on more AI, you know, I'm a traditional developer conference and oh, we'll have some talks on AI or we are already doing in machine learning, so we'll do more AI in there. So, there's a lot of events happening out there, but I feel that it's bolted on and kind of like explaining a little bit more of what it does. Mhm.

3:51 AI uh engineer fair was actually one of the first events that took it down center like this is AI native. This is what we're doing, right? It wasn't about all the things of the past. It was all the new things. And I think they've been running this now like for 2 year 2 and 1/2 years. And every time the event gets bigger. Uh it's was organized by, you know, initially it's still the same people, but uh like from Swix. And he is also the person who kind of like coined the term a little bit the AI engineer and kind of grew from there. So, it was like a label AI engineer and they grew the conference. Now, the cool thing about this uh being um so much on the spearhead of the new technology is that all the new tools, all the new people all even CTO CEU CEOs of those new tools are still accessible because they all want to be there. They all want to learn this is where the head of the industry is

4:52 kind of going. That's why I love going to these events because it is not just a rehash of whatever we're doing. This is kind of like where emerging patterns and things learn. Obviously, there's always kind of educational for people are not that known with this. So, they have workshops, but if you go into their level of content, it is forward-thinking. It is kind of like things that like are on the edge and I think they have also no shortage of content because they stream most of this just live for free. So, they recognize everybody, you know, needs to be there because it's all about the community and the chat that you're really learning. All the learns of the videos are for free um and it's just amazing how much energy uh you have when you come in that room.

5:41 It's pretty big. I think they had like roughly 3,000 people right now. So, which is, you know, not bad for a 2-years-old conference. But, yeah, they got the support from the whole industry from big players all around and that's why I love going there. It's not because I like San Francisco. I'd gladly make an exception for this event to go.

6:03 It's interesting actually cuz a lot of events have in recent years, 5 10 years or so, actually moved a little bit away from San Francisco but these days AI is obviously, you know, when you look at a lot of the investment and a lot of the startups they do tend to focus very much at the at the core of of San Francisco for for, you know, the Silicon Valley space. So, it kind of like makes it more natural for AI engineers conference to kind of like exist there which is I guess why they why they keep it there but it will be it will be interesting to see if they if they at any point do look to to other places. I think earlier this year they had they had the AI engineers summit in New York as well, right? Which is another interesting

6:49 That one was more like uh they called it invite-only. I think it was still accessible in a way if you really wanted to you could come but it was not that you know the the ticket sales are were kind of like open to to everybody. And that's more of a regrouping of that like smaller group forward-thinking. So, the for example, where in the past they had like many tracks like you know, rag and and so on they just had that whole conference is going to be about agent.

7:16 So, they kind of like focused this really hard. So, one more invite-only, one open to the very broad public. And I I keep hearing rumors there might be one happening in Europe as well maybe in fall. So, definitely they're thinking about like doing this closer for us closer to home. So,

7:35 Yeah. Yeah, absolutely. Um Let's jump in. There are there are seven learnings that you had. Should we should we name them very very briefly? So, one is about AI coding agents being truly everywhere. The second one is about how we need to change the way we we think about using AI dev tools today. The third is about how specs turning into the new code. The fourth is about how agents can be running in the cloud. The fifth is about parallel execution and how you can effectively explore the results of AI in in in parallel. The sixth is about CICD and is that shifting left? And then finally, the seventh is about the productivity gains that people can get

AI coding agents are truly everywhere

8:18 from from using AI and how many X's will it deliver? Will it be a 2x, a 5x, a 10x? We'll see. But, let's jump in with the agents are everywhere and it does seem In fact, there's a TLDR that I loved at the top of your top of your blog, which is I'll read it out. Cat ideas, pipe it into specs, pipe it into parallel agents which are which are running a synchronously, pipe that into your review and then pipe that into three dot profit as normal which I which I think is a nice a nice TLDR. But, let's jump into that first one, the the first learning.

8:52 AI coding agents are truly everywhere and this is really about how everyone is trying to effectively have an agent mode, trying to showcase that that their tool is now supporting an agentic way of of of working. Talk us through what you what you saw there.

9:10 So, one one of the things to note on the AI engineer is that in the last couple of events they were talking about agents and rag and embeddings and fine-tuning and all that stuff. So, the very broad AI engineering. For me, this event was the first one who had a large percentage of the talks focused on coding. Where in the past, coding was like, "Huh, yeah, it's there. Like, you know, you use kind of a co-pilot and that's done." But, because they're all become so powerful, it it felt like the tracks there was like a full track on coding agents and and more of the bigger player were announcing this. So, that was truly everywhere in that sense that yes, and the explanation is often that, you know, you can look for a lot of use cases to use more AI, but coding turns out to be this amazing like valuable piece to use AI for. So, that kind of realization and it also shows in kind of like what people are looking for in that

10:11 track. So, truly everywhere coding agents is not just something on the side that we do. It's just that in center also at this event.

10:19 Yeah, and is it is it also showing that as users we're trusting AI much much more cuz we're leaning into that autonomy? What what's what's the signal?

10:28 I think that's still the the jury's out. So, all the vendors are showcasing what they can do. I think that like I hadn't seen much of um I am a company and I'm introducing all these tools and that's the value I got from. So, that's probably going to be next year the more econic and how's that? I think it did New York one that it was one interesting talk about like how booking.com kind of introduced us.

10:55 So, that was like the the biggest story that I heard. Besides that, it's all about the vendors kind of like defending and saying how they improved this.

Evolving developer workflows every six months

11:03 But, the stories are still a little bit shy on that conference. So,

11:07 Yeah, yeah. Okay, cool. Let's go let's jump into learning number two. Interesting one. Just in 6 months we're saying what what what what this person saying here is if you use AI tooling like we did 6 months ago, we're making mistake. What's the what's the major difference? Is it agentic again? What's the Is it is it the fact that we need to run in a in a agentic way? What's the what's the what's the call here?

11:33 So, um I would say that in the past when you we have a new technology, the cloud, mobile, and so

1 Highlight · Adapting to rapidly shifting AI workflows

11:39 on. So, we see that initial spark, and then we can say, "Oh, we just need like a few more cycles to kind of like polish, improve this. We can see where this is going." With the AI coding, it feels that every 6 months we'll have to reinvent the way we actually use this. So, that's that's was the one of the points that was made uh by uh beyond uh from Sourcegraph who said like, "You have a practice like, 'Oh, I can do completion.' All right. Oh, no. All of a sudden I have an agent. So, I'm I'm doing prompts now. So, I'm kind of that.

12:12 And then after the prompts, somebody said like, 'Okay, we're doing agents, but now it becomes about specs.' So, we see that kind of reframing is important. And so, one of the biggest mistake is actually that you keep using the completion and say, "I got it." Right? So, you have to kind of keep embracing like new possibilities by the new kind of technology that's being embedded in the tools. And we're still figuring out like what the ideal kind of workflow looks like, but every time every 6 months we're unlocking with a certain new set of tools uh kind of a new way of working, which is great uh in that perspective.

12:50 There were some like interesting like

12:52 smaller tidbits on for example, OpenHand saying um you know, in the past we would say, "Agent, run off. Come back to us." And and and we see the pull requests of an agent's happening. And agent A does this. Agent B did that. But they found that it wasn't about um they needed to actually to have a human that was responsible for the actions. So, they started changing the way that the PRs were actually on behalf of a user because they found out that like people respond better to user than to an agent telling us what to do, right? Right. Well, it could be the other way around as well, but they they kind of like turned this into that it was always like this agent did did that on behalf of that user. So, that was a smaller thing, obviously, to the dirt. And then there like the other thing was, "Oh, we we just got used to using it more in our IDEs and the things." And all of a sudden we switching to more terminal based CLI tools. And maybe that's the part where like more autonomous asynchronous because it's

13:53 happening somewhere it runs, we don't have to see it that much anymore, and it comes back to us when it's done. One of the newer typical features from those AI things is that they have a notification. We all know notifications, we hate them, right? Our social media, our internal chat systems, and now the agents will send us like, "Hey, we're done. Like, you need to do something." Because that asynchronous thing is also kind of changing how we interact with them.

14:19 And it's really interesting actually because yeah, you mentioned, you know, obviously things like Claude Code, you've got Codex, Gemini model now. That's that that's has a kind of like Is it TUI or Chewy? Which would you go for?

14:32 Um I terminal UI, I would say. Chewy, that could be in Belgium like a a travel agency as well. So,

14:41 Oh, yeah, yeah. We yeah, we Okay, yeah, let's not use that then. So, TUI. So, so your T UI. So, yeah, you get things like Claude Code, obviously, that it really does It's one of the It's one of the first ones that really does change the well, first first tools that that has gone gone kind of like mainstream to really change the way in which developers code because it's entirely terminal based.

15:06 A lot of the way that I see people use that though, they don't just stay in Claude Code and continue to to to, you know, prompt and only stay in that terminal. A lot of people that I know of that use Claude Code, make the changes, then they you know, see the changes not in the terminal per se, but in in their typical IDE. And so, they're bouncing between the two. So, while we look at things like Cloud Code being headless, it's not the headless experience that we're always that that you know, we stick to. It's the tool that is headless and we're actually still sitting in our IDE of choice to to really if we want to look at the code more and more to to be able to traverse the code, to understand what the code's doing, to be able to do those reviews. And then you effectively use whatever it is, whether it's Git or something like that to effectively do your rollbacks and and and uh maintain state um of your of your versioning. Is that Would you say that's fair across

16:07 Remember that we talked about you using the old paradigms from 6 months ago. And the the 6 months ago, we were chatting. So, having a chat with an agent was a very synchronous thing. So, you could definitely use a terminal to do the same thing. You just chat and you kind of have this happening. Now, people have been saying it's the end of the IDE. Um you could say maybe it's the end of the code editor, but you'll still have to do the review. So, somehow you need UI or some way of like explaining things. As you mentioned, like when it's done, you kind of like have to visualize and see whether it's it's okay.

16:44 So, it's it's it's not completely headless, but it could go off if you would say if you really embraced like the asynchronous coding way, you would give it a larger chunk of tasks and it runs for a couple of hours and then it comes back to you. So, that's the same thing. Like but that means you'll have to specify more up front than just that simple task or that one thing. Or otherwise, it's more guessing and you don't have to stay in that loop. So, that's why it is you can

Specifications are becoming the new code

17:15 use a technology in the more synchronous way, but if you embrace asynchronous way, you start doing more of those kind of specifications and bigger tasks that you need to do.

17:26 And talk about specifications as your learning number three, how specs are the new code. Um and I think this is kind of like we're seeing this being mentioned more and more and I think specifically at this event, it was Shawn Grove who um works at OpenAI and he was talking about a number of things called model specs and things like that. Um give us a kind of brief overview. We shared this video of course at Tesla behind the scenes as well cuz it was very on on on

2 Highlight · Why specifications are replacing traditional code

17:54 the mark for us. Um tell us a little bit about how OpenAI are thinking about model specs and specification.

18:01 So they talked about a spec being important to guide a model for certain generation. And internally they have a concept called a model spec. And in that model spec they say things like you should always be friendly to what you're generating. It should be always on point. If you don't know the things, you should do that. So they kind of the example he gave was more about like training a model and making the model actually respond to those things.

18:31 But the same you can actually then apply to not training the model, but asking the model what to do. So if you give it specs to do a certain task or to code it, you kind of like give it that longer or that bigger chunk and it does it. If you want to change something, it's not about like the kind of disposable prompts that you never record. You just change the spec and you generate from that again. So that kind of reasoning for model training with specs and then code training or code generation with specs kind of like started to coincide in that kind of presentation.

19:08 And what was really nice is that he said um it actually they use it for alignments of the models and the AI. But what he also said was writing specification actually aligns the humans because you have to agree what the task is at hand. So, there's a benefit from kind of both sides and human and uh I explained it sometimes as what you're describing for a model is almost like your code of conduct of your model. Like this is how I think you should behave and you kind of bake that into the model. But very similar for your code, this is how I want you to write my code.

19:45 This is my directory structure, this is my language, these are my coding conventions, this is my process that I'm doing. So, you can all put all those things in in in the specs. And so, hence he just, you know, like concluded like the specs are bigger than code because if you change the spec, you can generate the code from there. And that very much align to your old expect driven

Tests validate AI-generated specifications

20:07 development that we often talk about uh as a way of working. So, it was a surprise coming from that angle from OpenAI model training and then all of a sudden like, you know, kind of crossing the chasm to code generation and saying actually it's it's there uh valid as well. So,

20:22 Yeah. And I think you mentioned in the blog as well how tests play such an

20:26 important role in effectively being able to validate uh specifications. So, obviously a specification is we're going to need to have some implementation created from that. And it's the tests that are written that uh that need to need to validate that. I guess some would say, "Okay, the tests are part of that specification." Um and some would say, "Actually, the tests are actually more important then as a result than than the code." Uh because they they they're proof of your intent.

20:57 Um how much how much was that kind of like mentioned cuz I feel like that's a a very crucial part.

21:02 I think he said uh what isn't uh in like if it isn't specified in the specification that behavior then it's actually it shouldn't be important. So if you really need it to be written down it needs to be in the specification and it be there. But we all learn from executable documentation remember that word a couple of years ago and it's the very same thing. So if you were write like the cases that actually it should work with that are the tests and they're an integral part of kind of examples obviously in AI you would do this the multi kind of examples in your prompt but this is very similar to your tests being described.

21:46 I personally believe that this helps actually with the generation because you know the actual LLM will look at the the the tests or the examples and try to figure out whether that works or not. Now the jury's still out whether we can trust this like completely but that's a different story. But the fact that we can now also it generates the test that still relieves the human to actually verify whether the tests were generated whether they were correct and actually the specification will help again with that alignment as well because now we can see the examples in the specs so the human can also kind of verify whether the behavior was correct. in both ways.

22:28 It's interesting actually because I I I met up with Baruch who I we had on the podcast a couple of weeks ago. And one of the

3 Highlight · Bridging natural language specs with BDD tests

22:36 things that he was talking about was around using an existing framework like like you know Gerkin and Cucumber so Gerkin specs and Cucumber the tool that allows you to create that BDD style approach. His approach to something like this is to be able to create a set of test descriptions effectively in the Gerkin format that could then be validated as you know natural language and so you can kind of like look at it and say yeah I agree I agree I agree and and you're effectively effectively part of your specification then right cuz it's it it shows that intent and then using something like Cucumber to then turn that into tests there's no LLM it's it's there's no non-determinism you know so long as you get it right in your given that given this then that so long as that's right then you will get those tests that actually that actually perform that and you have that greater level of of of

23:36 confidence that those tests are going to be correct

23:39 developer actually writes that code right like you know fulfills that test so it is the business person kind of describe the business functionality thanks to a behavioral test and then that gets implemented now you can use that same kind of language to specify things in your examples for example no pun intended but the then the LLM just generates from that the actual code and something that adheres to the BDD test that you provided still it can get it wrong so we're not there that it's 100% but it actually makes that mapping even more seamless like I don't have to write that code I can like bootstrap the code and eventually have almost like perfect code to run this and

24:27 see them execute from the code I have and so in that loop that an agent goes through it wasn't that what we're looking for for a completion or the initial chat but with an agent you can go into a loop and say oh it didn't work let me look at the or error kind of let me run that test again so that kind of loop

Moving agent execution to cloud sandboxes

24:46 is now more has more context to what it actually should be achieving there

24:52 cool almost halfway now four number four is agents to the cloud. So, this is about how people aren't using agents in their kind of like local local machines anymore. This is more about how they can be pushed to the cloud and then executed somewhere further away. Talk us through I guess Is this something that you're seeing fairly fairly commonly you know, rolled out across different tooling vendors and different providers?

25:25 So, the the struggle you have with more of these async longer running threads is when you close your laptop it goes to sleep. You have like some annoyances there. So, that was one thing like you know, I have a longer running and it will not keep running. The second one is if I run these agents and they they do a lot of CPU work, it's hogging my machine. So, that's also kind of annoying. So, you know, how do you solve that problem as well. And then the third one which you start more and more appearing is if you do it parallel, then it becomes even more constrained to do this.

26:04 Now, where do you start like seeing this in a tool? Maybe the first one you could say was the amp code was doing code indexing from your embeddings into the cloud. If you have large code bases, your machine doesn't have a GPU, it takes a lot of time to do all that local indexing. So, they resorted to moving that to the cloud. Now, that's a minor thing, but it's definitely helpful for bigger code bases there as well.

26:31 Now, what then you start seeing tools like cursor you know, trying background agents. And so, they they have like a hosted version of that like execution thing. And now all all the others are kind of like following suit that like you can have the option either run the agent locally or run it on a remote machine. Now, there's a bit of a challenge here is how do we actually replicate the environment

26:57 Mhm.

26:57 of your dev environment somewhere in the cloud. So, that the first attempt was let me like almost like create a virtual like shadow environment from your coding environment and we'll code on that. So, that was one of the first tries and becomes immensely complex because like what's all the things that have been set up and you just can't say clone dev environment or so specific so kind of annoying. So, that's where people move more into hey, but we have this technology called containers and so we use that in our CI/CD. So, let's kind of run these agents in those kind of more sandbox environments and then obviously

4 Highlight · Running agents reliably in cloud containers

27:37 that became like the logical thing to do either in a local sandbox in your container or somewhere in the cloud. Now, that means they're mostly shifting towards the following pattern. I have an existing code commit and I want to do an update on that commit and then I ask the agent to do a certain task on that commit, right? So, what they do behind the scenes is they say here's the commit. Oh, let me kind of do a get pull from the repo check out that commit, run my agents in the cloud, and then I kind of like create a feature branch and present that back to you. So, that's kind of the flow that all these tools have been following to make it all more lot easier to have like a more reliable workflow than trying to clone your desktop which is kind of like insane

28:25 Mhm.

28:25 and would not work. So, it's definitely there more and more in the tools. Now, running agents is not also without any issues.

28:35 Like you have to do the permissions. And if you're still doing the babysit version of yes, no, agree that's great, but that's for a a running thread that's going to be annoying, right? So, if you can kind of control the network, which tools it has access, you can control that way better in a sandbox environment than trying to run on your laptop as well to do that. So, those are like two reasons why people are moving towards the cloud and it like the major coding tools are doing this. Now, whether your whole IDE will move to the cloud or we still have an IDE, that's a different conversation, but you start seeing uh I think um Cursor just released like a mobile uh view on their editor, so you can kind of code from mobile. So, you see kind of something is shifting again on how we deal with this and and how it reports back to you when it's done um in there as well.

29:27 I didn't see the mobile. Was that Was the mobile Cursor uh view was that um Was that also at the AI Engineer Summit or is that more recent?

29:35 No, no, it's uh it's something that released like last week or something like that.

29:38 Oh, wow. I didn't see. I missed that. Wow, it it it's such a fast-moving space, right? It's very hard to It's very hard to catch it all. Um very, very interesting. And And one of the things that that kind of like opens up as you

Parallel execution creates a review bottleneck

29:48 kind of mentioned there was was parallel. Obviously, if you're trying to run multiple agents in parallel, it's going to be a real consumption hog for your for you know, your local machine. And that kind of like leans us into your fifth learning, which is around parallel execution. And there are I guess a few reasons you might want to do something like this. And And And in your in your blog, you mentioned a few things that were kind of mentioned. One is a really around breaking a task up uh into into subtasks and having multiple different agents work on those subtasks.

30:20 And I think I guess somewhat you could actually it's nicer to almost like have different uh models, I guess, potentially working on those depending on what the right model is for that um for for that task. Um others, you might have um uh just a variety. So, let's let's have three perhaps different models or maybe the same model running something three uh times in parallel and then trying to work out, okay, which is the which is the best one that I want to go with. And it really does open up this this new world whereby because we are with the genetic coding waiting longer, we can afford to kind of like say, yeah, run a few. Let's Let's try and now get the best answer.

31:05 And I don't mind if it takes an amount of time. Just come to me when you're done and we'll we'll we'll sync up what's um you know, what what what's been created. It really opens a new world for us, right?

31:16 Yeah, yeah. So, you're right. Like two pieces, one is faster delivery or split this up a parallel. We actually do this intuitively, you know, without agents. Like let's have five developers. Let's split up the features, kind of go from there and then, you know, they can work and merge in and be more like independent of each others. Like now the the planning is actually done by the coding tools and they kind of try to split this up. Now, it is challenging which actually you know, thing overlaps with another tool because they have the same problems as humans have is in the end you need to do the merging of all those parallel things.

31:51 You could use things like stacking or like a few of those changes together. Um but yeah, that's that's the first reason

5 Highlight · Using parallel execution to explore competing solutions

31:58 why people use this and and they have some challenges. Uh the other is variations and it could be uh things like um I want to like explore three different ways of solving this problem, right? So, use this framework, use that framework, use that framework. Or come back with three hypothesis why the performance is bad. Like an explore three of them. So, it could be also be for performance or support reasons there as well.

32:25 So, so I like that whereby you can almost like say to the LM that created this, okay, why is yours better than these others? So, almost like almost like argue between each other as to which is the best. Or maybe there's some there's some uh you know spot in the middle where actually it's maybe less performant but actually there are other things that that go for it more reliable or whatever it is. Um whereby actually they can come to a decision based on their own discussion their own arguments as to as to why each are best.

32:54 It's like the the the it's bizarre how great this could actually turn in.

33:00 Yeah. But that's that's a challenge, right? So imagine uh as a human we have five people doing coding in the different tracks. There's one kind of orchestrator or something or manager who's to say well, this is the best.

33:14 Mhm.

33:15 Um Now in most of the tools you will find that they're all about good about creating all these variations. But they actually that brings the burden on us, you know, kind of like managing which one is the best. Now we could use maybe some help from an LLM who says like I think based on the input that the other agents gave me, I think this one is the best. So or it could help us also show different diffs like

33:42 Mhm.

33:43 this kind of application that like or gave measurements. Now I feel that this is the field where kind of we're moving into. Like now we have parallel execution. Now we need a way better way of doing that review or parallel review. Now some of the tools are starting to build the new UI in a way that is not showing one version of the application but it allows you to compare three versions of the application kind of more in parallel or three kind of like rendered things.

34:12 And and we'll see more of that. You know, I was once talking with one of our colleagues Tom and and we came up with an idea like we often asked to do vibe coding, you know, to kind of build an app. But what if the agents actually can vibe code an application to do the review?

34:30 Mhm.

34:31 Because then it becomes easier to kind of like say which one is the best. the They come up with the best interface to actually check whether like what's the best that we should select. So there's a lot of like exploration to be done there. But the burden of reviewing still stays on us. So anything we can do to improve that, reduce cognitive load, help us understand. That's probably the next phase in in whole this journey of going there. So

34:59 Yeah, and and the developer usage you kind of mentioned in the post as well that Solomon Hykes, the CEO of Dagger, one of the things that that they showed was how you can effectively hook up an MCP server or you can invoke through your IDE an MCP server whereby if you say I want some different variations for this, it'll go away and do that from an MCP server and actually when you pull the changes back that it's already in a branch. Each each variation's in a branch each and each you can kind of like click through, have a look at them, and then merge merge the branch that you want to accept. So from a from a user point of view it's actually very very aligned with potentially the way that developers want to consume these changes. It's not some completely different UI or anything like that. It's nicely within the their IDE which is wonderful.

35:54 Yeah, and I think that's that's also a journey that we've seen. So last year I saw quite some tweets about let's open up five different IDE windows and have them all code on the same code base. Disaster, chaos, you know, obviously from experiment. Then the next kind of phase was well, what if we clone the directory to kind of like and have them work on separate and kind of merge that back in. So that was the whole idea of get up work trees and kind of get work trees to kind of like have different like agents working on each other that the environments weren't isolated enough because you need to control more and then we put in a container and then ultimately be we put that into the cloud there as well. So that that parallelization has been going on for a while and what you see now is that when you run it in the cloud, how do you actually merge this back in?

36:48 Cursor has this behind the scenes is not very well described but they call it like a the remote kind of cursor protocol which is emerging from a remote server back to your local branch and then kind of dagger has a similar system where it actually exist in a container but you're almost as simple as a get merge get like whatever was in the container back to your local code thing thing there as well. So again another learnings and how it's progressing on like all bits and pieces they come from intent kind of break it down to task which we did like parallel and then kind of do it this parallel execution and now we're up to the reviewing as kind of the issue whether we like it or not. So that's the

CI/CD shifts left with agent sandbox testing

37:35 next frontier there. So

37:37 Yeah. Yeah, very interesting. Looking forward to seeing where that goes. Learning number six, let's move on to learning number six. Is CICD shifting left? This one baffled me. Tell me how CICD can can can shift left Patrick.

37:51 So um with all those agents kind of putting more and having specifications having examples they do a lot more testing than maybe we would continuously do. So that means that like whenever you're still on your local branch, you can spin up the same CICD and have the agents kind of almost kind of run that. So in that perspective it is shifting even more left like not all the testing has to be done on the CICD system but we can do like, you know, code analysis or other stuff. Now, this was possible in the past as well, but now that we've moved into the parallel execution and the parallel thing, we we we we get that to a bigger extent where in the past it was always like, "Yeah, in the CI/CD it has the real environment, it has the real setup." But what if you can just like take that simple system and have the agents have their own CI environment and do the same thing? So, we get way master faster feedback as a developer and don't

38:52 have to wait until the whole merge is complete until it kicks off. So, that's was the whole idea of CI/CD uh is shifting left with kind of faster feedback to the developer even before anywhere like merging this into one of the main systems or kind of the staging system. So, yeah.

39:11 Yeah, interesting. And and um you kind of also mentioned that um Josh Albrecht uh in Blue CTO uh just talking a little bit about how uh specs with strict guidelines helps catch these issues earlier in the in in the value stream. And I think this is all about um the kind of intent, making it easier, being able to really clearly describe intent makes it easier to be able to validate whether your implementations adhere to that intent or don't adhere to that intent. Is that Is that Is that fair?

39:44 And is it from a from a spec point of view, is it easier for us to kind of like generate tests and and and uh really build out the tests that are important because we're very clearly defining intent?

39:57 Yeah. I think um when we code traditionally, we take specifications either from, you know, your product manager what it's supposed to do, and we break this down into actual code. Now, imagine you would then run a security scanner or something on that, and you just run this on the code. The code is just an implementation as such. But imagine would actually know what the intent or the intended behavior was. It could do a way better job at understanding what it needs to check. So it's almost like that we throw away that information and like in a game of telephone and then like assume the next kind of agent just like our tool knows it. So if they can all refer to that same spec, they have a better chance at collecting actually understanding what goes on. So it's it's funny because it's it's actually almost like a when you're making a movie, you you you kind of remove all the cuts, you remove all kind of the the takes and

40:59 then you only have the last thing and then you don't know anymore how actually the movie was made because but when you need to do it again, you're like, what what lightning did I use or like what what like okay. So kind of very similar of throwing away metadata at the end of the pipeline.

41:14 And and going back to I guess your third learning there when you talk about specs of the new code, one of the things that Shawn was mentioning is how specifications align humans and what we're really talking about here is it's not just aligning humans, it's aligning aligning agents through the through the through the workflow to make sure they all have that access to that that core intent data and the core the core data that we're we're using throughout the creation process.

Productivity multipliers depend on codebase health

41:39 Cool. Last one Patrick. How many X's will AI deliver? And we're not talking about the the the funky I'm still going to call it new the funky new name brand for Twitter. We're talking about how many times a developer or how many you know, X's in terms of 5x, 10x, 20x productivity gains are we expecting to see? Are we are we really looking at the kind of like 5x, 10x, 20x style claims uh, that people are making or or is this overblown?

42:13 So, at the event, uh, there was a

6 Highlight · Factors influencing AI productivity multipliers

42:15 researcher, um, kind of explaining, uh, you know, what they found that influences actually that X. Uh, and it could be if you're doing a simpler task, right? Of course, the AI is better at this and so that gives you a performance boost. So, if you're doing that repetitively, you get like an X that is bigger. If the task is more complex, right? Uh, it could be that it's helping, but that it's also an it requires you to tune and kind of correct the agent right now way more.

42:50 So, it becomes a balance between how much effort do I help, how much effort do I actually know. So, that kind of influences that. The other point that he made is, uh, if you have a code base that has been long running, um, there over the years there's different standards, different agreements, different terminology. It confuses the AI, much like it confuses humans, I guess. But that means the effectiveness goes down. If I ask a question, well, it says like, "Well, where are my coding standards?" Well, you had like seven different over like five years.

43:24 Uh, so that inconsistency actually also influences that. Uh, obviously the bigger size, but that seems to be overcome by having bigger machines to index the code base. Uh, and kind of find the relevant pieces that need to be changed. So, that is a factor, but it's it's maybe a less uh, thing. And then the last one he mentioned was the language on which the the LLMs are have been trained. Uh, is that actually, you know, a more popular language? Is it a niche language? So, definitely also in in in kind of influences, uh, the code generation language. Now, all of that is always a snapshot in time and all these points they valid, but then we saw that it was not just about the LLM, just doing one code generation. It was going into a loop, and then the loop got documentation, got specs. So, there is a different like definitely a rise on this, but it it's not like a excess or something. Um but it it is

44:24 there is a certain influence that, depending on your use case, you might not be getting what you others are saying about their code base. If it's like a small, new, modern language, kind of like fresh, it's completely different than a more legacy enterprise or kind of like tool base that that you're working on. So, that was kind of his finding

44:45 uh where he says, "Most of those um are you more efficient with AI?" is very anecdotal because they they didn't kind of specify these aspects. And then it becomes like, "What are you actually comparing?" Yeah, that we're working on, so.

44:59 Yeah. Well, I think one of the big takeaways from this whole thing is really that, and I think you mentioned this in your closing thoughts as well. Uh and let me read from it. What is considered good practice now can change really fast and it's hard to keep up. And I think that's one of the That's one of the core kind of things in particular in one of your one of your learnings, which is around uh you know, using even 6 months ago an AI tool in the same way as today, you'd you'd actually be, you know, pretty you know, doing it pretty poorly or not not really using the most effective means of of developing. So, it's like you're going to change again in another 6 months, and it's really really important to to stay on track and to and to keep learning because uh you know, today we have TUIs and agents and cloud agents and things like that.

45:46 Who knows where we're going to be in 6 months. Uh any any predictions if I put you on the spot there, Patrick?

45:52 Um What predictions is the reviewing is going to get a bump, and then after that is going to be knowledge management. Because if if we got more automated, we have to kind of keep track of all the learnings that the agents are doing and kind of spend more more time on that. I I understand it's very frustrating to have to keep checking what the tool do, how it's actually improving, and a lot of people are dismissing, "Oh, I looked at tool X and it didn't work for me."

46:20 But then maybe next day they kind of bumped and they improved the whole version or they did differently. And that's kind of the annoying is that like whatever uh and I I I often refer to this as the innovation tax, right? So, you you just have to pay that tax of having to try a new tool, but it's definitely different from previous waves of technology where you say, "Oh, it's maturing." And now it's like constantly changing how we work, which doesn't really help to that kind of perspective, I guess.

46:51 Yeah. And I think as us as developers need to kind of always be open to that constant learning and never never thinking if I'm you know, I'm using this today, um I'm going to stay on this for the next couple of years. It's just not possible right now. You have to absolutely keep looking and have to keep updating and and and and taking on the new the new features, the new ways of of of doing things. Uh Patrick, once again, it's been absolutely amazing to chat with you.

47:17 Really really appreciate all your all your kind learnings, sharing your kind learnings not just on the blog, which we'll share in the show notes, but as well on our podcast. So, really really appreciate you joining and thanks thanks again for for supporting the AI Native Dev.

47:32 Yeah, my pleasure. And also kind of as a listener or viewer, if you have like a cool story you want to share or something that you know, you learned, do feel free to reach out or put in the comments or like we're always kind of like willing to see what others are doing. So, that's cool.

47:48 Absolutely. Absolutely. Thank you very much. And thanks everyone for for tuning in. Really, as Patrick mentioned, interested to hear if uh if you went to the AI engineering uh world uh fair. What did you think? What were your takeaways? And uh thanks for tuning in as well. So, till next time. Bye for now.