Why Agents Are Forcing Enterprises to Finally Fix Their Dev Process

11,769 words · about 47 min to read

0:00 now that it's agents doing the work and tokens and the cost of those, everyone's going to be really keen on making sure the software lifecycle was effective and efficient, as it possibly could be, but when it was humans, it was like, oh, well, Timmy's just lazy. And that's that's why stuff isn't getting And they didn't focus enough on process and value stream mapping and all those important things now it's when it's agents. It's like, well, that's going to cost us money and we can easily see that it's costing us money.

0:25 most people know me from the DevOps transformation and the cloud native transformation. And I love every new chaotic period in the industry because that's where the learning is happening. And right now that is AI coding. the way I think about it is I go back to gen AI, and I imagine I had all the manpower and all the time in the world, basically with that mindset we say okay now you have a you have agents. There is no excuse not to do everything you ever thought you'd want somebody to do and then automated.

0:52 So it happens by CI automatically. The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Podjarny and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

1:18 Hey everyone! Hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests, so we can have the most informative conversations about agentic development, whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content. Please do us a favor and hit that subscribe button.

1:46 It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode. Hey there, Simon Maple here. Welcome to another episode of the AI Native Dev. This week we're sharing another fascinating conversation that we had with some of our AI Native DevCon speakers, this time on the topic of AI enablement and what enterprises need to get up to speed with the genetic development.

2:13 I sat down with the godfather of DevOps, Patrick Debois, Autonomy AI's Tammuz Dubnov, and Daniel Jones from re:cinq. What I think you'll agree was a really intriguing discussion. And what I love is the way that some discussions you very often get question, answer, question, answer. But these folks were very, very passionate. They really added to each other's points and it was a true conversation. I hope you enjoy. Hello Simon Maple here and I am at the AI Native DevCon 2026 in London, and I've got a wonderful panel here to talk about AI enablement within organizations, how we can be most successful, scaling agentic development across our organizations.

2:54 I've got a wonderful panel. As I mentioned, we have Daniel Jones, DJ, head of Product at re:cinq. We have Tammuz, who is the co-founder and CEO of Autonomy AI. And we have Patrick Debois, who is the DevRel at Tessl, the DevOps overlord and godfather of all things pipeline and workflows and DevOps. Welcome everyone to the panel. How are you doing? Not too bad. There was a party on a boat yesterday. There was? Yes, and I am just about functioning. Excellent.

3:21 Good. Well, that's. Always a benefit. Even with good weather in England, right? Yeah. Yeah, yeah. Good weather. Yeah. On a boat on the Thames. And it was sunny. You knew. He knew that was a possibility. So why don't we start off with. Why don't we start off with you, Jay, and let's go left to right. And we'll tell us a little bit about more about yourself, your role, and maybe a little bit about reasoning. Sure. So I am head of product at re:cinq, which means product sizing our services. We are a AI native transformation consultancy.

3:48 We were big in the co-founders were big into cloud native transformation, wrote the O'Reilly Cloud Native Transformation book. And what we see is that organizations are repeating the same kind of mistakes when trying to adopt a disruptive technology, or just plopping the technology in, not changing anything about how they work, and being surprised that they're not going faster. So we offer various services around that. And I do a lot of the training for agentic coding.

4:14 Awesome to meet. Co-founder and CEO Tammuz. We built an operating system, top of your code base, to enable an easy platform for non-technical or semi technical individuals to actually meaningfully contribute to the code base and organizations. Something we can do their visual iterations and actually get stuff ready. Do backlog items, do new items, but everything inside of brownfield code base and it ends up in a meaningful PR that should be merged. Awesome. Patrick Debois.

4:41 I am from Belgium and I have gray hair, so I've lived through a bunch of transformations in my life, and most people know me from the DevOps transformation and the cloud native transformation. And I love every new chaotic period in the industry because that's where the learning is happening. And right now that is AI coding. And so I'm looking not just at the tools, but also we're kind of the new organizational patterns are happening as this field is forming in the industry.

5:10 So that's kind of my jam right now. And every time there's a chaotic transformation the hair gets great. Yeah, it's hard to know whether it's the cause or the effect. But it's. Like yeah I know. So round this table we're going to be changed about all things AI enablement.

Scaling AI development through platform and incubation teams

5:26 Let's kick off with an interesting question about when we want to scale AI development across our organization in a way where we can control it. It is something we can govern. It's something we can do in a in a meaningful and controlled way. Who owns that? Which team? Which individual who's responsible for making sure that happens? Well.

5:51 It's a good question, and one that I think a lot of people are grappling with, and I should imagine that we'll probably end up with new departments, new teams, new named units and movements forming. I'm working with one customer who happens to have a developer experience in, which is kind of overlaps with the platform team. And so there's an amount of kind of it makes sense for those folks to be doing things like building and distributing and assuring the quality of skills.

6:20 For example, platform teams are kind of interesting in that they're there to help developers by presenting a, you know, a raised level of abstraction, reduce cognitive load. And you can imagine that, like making sure that agentic coding goes well and, you know, context is managed and available to agents would fit into that group of people's responsibility. But I've seen other customers as well who have a DevOps team, which probably shouldn't get to deep into that.

6:48 There was an anti-pattern or not, but I've seen those folks like the people that get lumbered with like you're doing the CI/CD pipelines for people like those people. So being kind of made responsible for these things. Patrick, I'd love to go to you there. How many, how many hours of sleep did you did you miss when, when when the DevOps role came about? No no no no. It's quite easy. The big thing is that every new change in an organization usually has like a small incubation kickstarting team.

7:18 We had the agile team. You know, we don't talk about an agile team anymore, like everybody's doing agile. We had the DevOps team, and that kind of is just a way of scaling things out. Like, you need that one team that gets duplicated in multiple teams, like, hey, can I repeat this? Can I repeat this? And then I think the pattern is indeed right now is it falls back to a central team kind of nurturing all the other teams in a way.

7:44 Now I think the challenge, what you were saying, like, yeah, it might end up in the developer experience platform team as a role. Their challenge right now is they're still often focused on oh will provide the infrastructure. They're not die savvy tech coders. So that's why there's now a little bit of a vacuum in kind of the companies where like okay, you know, they're not the best fit, but they they do control the spend. They have the models, they have the gateways.

8:10 So you can see their stack kind of expanding. Now whether they will stay in one group as a platform that's uncertain. What you might see is that you're going to have the cloud platform team, the AI platform team, the data platform team, and there are all kinds of platforms supporting it. So it's not because there's like one platform group that they have to do it all. And that's the same thing with like feature teams.

8:37 They can't do it all because they have a specific focus. Small companies. Yes. Same team that aligns bigger companies. They might have like different skills that they're going for. So anyway that's kind of how I. Look at it. Tell me as a CTO, you know, you've got your engineering team there. They want to use AI as much as possible. They don't want to be slowed down, you know, through through extra governance and things like that.

9:03 Where's the balance between, you know, having a team wanting to add governance, add ways of working, showing a golden path versus at this stage of AI's, you know, maturity, immaturity, allowing people to kind of find their own paths and understand what's good through, you know, developers playing around and trying different things. Sure. So for us to speed up has been getting the whole team alliance.

9:32 Which person finds a new skill that's important in our workflow in some some manner, and then we have a way to share it with all of them and have everybody else they use it itself learns. But the mechanism that's really kind of sped up. Our R&D is getting to focus on the stuff that really matters, and we get that because we offload the stuff that doesn't need an engineer to our actually a product team and our designer team. So part of the game for us has actually been, well, we have builders that are engineers, builders that are not engineer.

9:57 How do we manage the non-technical opening the press and how do we manage the technical side, getting those press and that kind of reshuffle? Actually, that's what has really helped us get through that native speedup, because you have to do change management internally, both on the receiving end and the the new builders. And suddenly when that does click, that's where you see everything running forward and you see the whole team playing together in harmony.

10:23 Yeah. I think that's interesting because, you know, when you recall in the past, Devin Ops bridging two worlds like developers and coming together, I think the term that like comes up a lot now is the AI product engineer. They're like in both worlds trying to cross the silo and the bridge like you were working on, like how to align that better as a field. And I think that's fascinating in a way that like when the the toys or the daily things are being more and more done by I, we can start like redirecting them.

10:54 It's not about like, how do we building the thing. Right? But are we building the right thing. And that's kind of where the product focus like comes stronger again. And I think I like kind of how these roles are blending. It's not for everybody I guess, but like it is kind of a good alignment to have in an organization. So so we see that there's kind of a span of work that needs to be done. And frankly, there's the span of people that care for different parts of the work. So there's the stuff that the designers and the PMS really care for.

11:20 And honestly, that's not the stuff that developers care for. So the fact that we can say, great, those builders are now doing it a developers UK better like architecture scaling, like the sandbox environments and how they live. You can focus on that because all this let's say annoying work you used to have to do. Well now the builders that care about them, they also have the authority to those decisions. They're the ones doing it. And hey, you can focus on the stuff that you care about and making them more friendly and faster.

11:47 Developer. That's just happening in our team organically. They're already there, already trying to speed up their already figuring it out on their own. And for me, the focus of the CTO is enabling. Decided isn't pushing or wasn't pushing as much, which is a non-technical. And when you get them up to speed, so you get the full speed up and you get people working on the stuff that they care about, and then that's the stuff that they're past us on. Yeah. And, and, and speed is the most interesting thing here because I think, you know, when we think about a year ago, two years ago.

12:12 Gosh, I'm trying to work out how. Old AI in the mainstream is now. But if you think about the last couple of years we're really thinking about, let's just try and get AI usage. Let's try and get AI adoption as almost like, you know, get it in the hands of developers. Play around with it and see what, see what works and what doesn't work. It feels like more and more now going forward. We're thinking about how can we make it work effectively for us, and how do we actually get the entire organization using best practices.

12:38 So let's flip it a little bit and say what? As organizations start rolling out

Software maturity and turn counts dictate agentic speed

12:45 AI seriously, in terms. Of. You know, as a practice across every single development team, what would you say the ask you this, what would you say the greatest bottlenecks today, which really maybe it's platform teams, maybe there's other teams that they're struggling with as part of that scale and that AI enablement across all. The teams. The bottlenecks that become apparent are often deficiencies in just doing software development.

13:13 Well, I think that we may probably discover in the next couple of years that we spend a lot of our time repeating ourselves. And the things we've been saying for the last 10 or 15 years. Is your CI/CD any good? Do you have tests? Do you have coding standards that people actually agree on? So all of those things kind of get exposed. And the the Dora report showed that if you are not very mature in your software development practices, then agentic coding is likely to make you go slower.

13:41 If you're doing well in development maturity, then you'll go faster. The problem is, we don't know where that tipping point is. So there are all these things inside the development process. But even with successful agentic coding adoption. I've seen in a couple of customers and people that have spoken to on the podcast probably shouldn't plug. Or they're. In the comments. You're on the native dev, you're talking about that.

14:07 One because you do. Have. Your own catharsis. Indeed. But we've seen people there of you suddenly speed up the act of creating software. And then products are called flat footed and they're like, Crikey, we didn't have enough stuff in the backlog. And that's hard to speed up because it's strategic and you need deep thinking and you need to be talking to customers, understanding their needs. So we definitely seen cases where software development has started to go much more quickly.

14:34 And then in people that do QA as an after the fact thing, that's been a bottleneck. I'm not sure I'd recommend that pattern generally, but yeah, products tends to become a bottleneck. And then we need to empower those people to to have more free time and to do the strategic thinking and the discovery work that presumably they really enjoy and they would prefer to be doing rather than writing JIRA tickets that developers aren't going to complain about. Yeah, yeah. How about you, Patrick?

15:01 I think if you look at the door, first thing is the adoption and kind of is it like cranking out new code that was almost like the developer productivity lines of code for a long time. Then we learned, okay, it needs to work in production. So we brought in the metric actually, like how many times do we need to rework it if it isn't good? Like that kind of is your defect rate or almost like that. You kind of need to be in control, both in generating and kind of making sure the defects are not like our imbalance there as well.

15:30 And I think what I like about as an almost like new proxy metric that like when you look at like not just the users coding with AI, but if the agents are doing the coding, you look at the things that like how many, how many, almost like turns does an agent need to do to do their job effectively? And you kind of can get that number down by you can select another model, you can kind of give it the right tools and you can provide it better context.

16:02 So in that way it is not any more like can I do better code in production, but can my agents do better code towards production. So it becomes like delegate. So you need to think out how effective they are. And that's kind of a flip that I see. It's now slowly moving from the def, working with AI to the Def, instructing the agents to use AI to do it. And that's kind of like an interesting proxy metric, I think, to to track.

16:30 And I think there's something both amusing and mildly depressing about this, in that when we have software factories and that idea matures and becomes more commonplace, we're going to have this infinitely tunable and measurable way of developing software, and we see exactly how effective it is in that way that you mentioned, of looking at how many turns you use, how many tokens are used. And unlike with human teams, we're going to be able to a B test. We could like have prompts and software factory setups that use one approach versus another, and we can reset their memories and try it a second time and see, did that work better?

17:02 Did it not? And you can't really do that with software engineers and real teams because you can't do it.

1 Highlight · The financial realization of software efficiency

17:06 Men in black and memory wipe them. But the thing that's like amusing and depressing is that now that it's agents doing the work and tokens and the cost of those, everyone's going to be really like, keen on making sure the software development lifecycle is as effective and efficient as it possibly could be. But when it was humans, it was like, oh well, Timmy, you just lazy. And that's why stuff isn't getting delivered. And they didn't focus enough on process and the value stream mapping and all those important things to to tune in.

17:37 It was just, oh, the humans can suffer, but now it's when it's agents. It's like, well, that's going to cost us money and we can easily see that it's costing us money. So now all of a sudden we care. Yeah, yeah. It's funny, we face this challenge today because we have agents working in 160

17:50 plus organizations which wildly different codebases. And we see the ones that are that are called quality. The agent does a better job faster, the ones that are lower quality. And we have an onboarding flow that the agents get better after a few tasks, they onboard themselves to your repo, and we see that in the difficult projects. It takes them longer to onboard to good quality, but we cannot go to we focus on onboarding quickly to an organization and we go top down. So that means that the business leaders we talk to, we cannot say, hey, we like a sprint to refactor so the agents can have a better time.

18:22 So everything will go smoother because they don't care about that. They just care about speed like you mentioned. So for us, it's been all about how do you mix in doing the user facing work quickly while writing the natural code term? That organization has to slowly refactor. But we have like a we call it the Pact framework. And as you do work, you slowly refactor a bit like bit so it's easier for the next agent comes along in that part of the code. But you do not come and say, hey dev team, you have this task.

18:48 Make it easy for our agents to work there you have the agents doing the work in bite size, going back to the PR fatigue in a way that doesn't affect the PR because then people will go over them. So it's a really it's a balancing game getting the the leadership happy because features are being delivered, getting the developers happy because PR are not too big, and getting the non-technical, the PM's, the designers happy because they can do the work, all while slowly pushing your agenda better and better coding standards.

Bridging the gap between technical and non-technical builders

19:15 Yeah, so everything needs to happen. Kind of weave together. And I'd love to kind of talk about that. You previously you mentioned about the kind of like. The. Technical devs. The traditional devs and the non-technical, and maybe it's a PM or something like that. When we talk about rolling out and enablement here, do you see different Tennessee, different bottlenecks between a technical dev and a non-technical dev when adopting like a more of an a process there?

19:41 Yes. Okay. So we'll go with the technical side and the technical side. There's an ego that we all know we are working through that whether it's with clients or whether it's internally with the devs. And so for us, they interface with us. They can use our our platform, not not to show off, but like the, the PR that come out of them versus out of the platform versus developers after our platforms, they do a better job because they map your recording practices much more in-depth than developers, and they keep much better tracks of existing components.

20:11 Like we have clients with 2700 components, no developer will know which component. To use. We do know we do data science on it, and we can actually rank and pick between them in a much more intelligent way. But you have to come over that that ego of the developer getting the PR. So we tuned it. We're a little gentle when it comes to them and then very fragile. All right. I'm not gonna. Say that right. Go ahead, go ahead. I don't remember.

20:37 And then on the side it's a confidence thing. It's like, hey, I'm. I'm okay opening a PR. It's getting the the dev team to play along with. We have some experiences where PMS open PRS and the dev team pushed back. And then we asked like a team lead, look at the code. Is it good? It's like, oh, this is solid code. Why did the developer push back? Okay, because it didn't come from a developer. Yeah. And so when that's worked out, we basically work on a conference with the PM saying, hey, you don't need to write a ticket. That's not the deliverable.

21:07 You don't need to make an HTML prototype that is disconnected. That is not little verbal. That's effectively a ticket in a figma. Yeah. You need to actually prototype inside the code base. You need to edit existing stuff in the code base. Hey it's possible. Hey. It's feasible. Hey, it's really easy. And when you work with the agent, it already knows it. Constraints from the code base built in when you interact with it. And yes, when you click Send to Dev and it goes to the developers as a PR, be proud and won't be scared.

21:33 And that's like a big enabler. Of imposter syndrome. I guess for non-technical folks. To. To feel like maybe they're resistant to to do things with confidence because they feel like they're not a dev and a dev's going to scrutinize it more almost. They they're. Scared, they don't know how to answer technical questions. And that's a part of the process where we're going through, like, you don't need to. That's the beauty of the agents. They know the code base. They manage it for you. You need to give us the product content and that's what matters.

21:59 And you, the agent, delivers something visual, and it's a part of their app that is rendering and running, and they can react with it. If you do the product review or do this interview in your persona, you should be confident the code works the way you wanted it to. The code quality is there, you don't worry about it. And hey, if there's like judgment calls on the back end side, the developers will take it. So let's move in a different direction. Now we're going to talk about ways in which a development, or rather patterns of agentic development,

Applying good software engineering practices to the machine

22:30 how people are doing this in the real world, and what we see from people externally in what they're saying they're doing and how they're approaching it. Patrick, you recently created a really interesting tool that's called the Tessl Patterns site, tessl.io/patterns. And it's a ton of research in the background. I think it uses the Karpathy. Wiki approach in the background, and it provides a bunch of patterns that people can look at and say, oh, okay, I can see that people are thinking and doing things in this way, this approach and, you know, sites, various social, you know.

23:10 I guess, ways in which people are going into more depth to say that's how I did it like this on in this infrastructure, in this way. Talk us through a little bit about that and maybe highlight a couple of the more interesting. Patterns that you found. Yeah. So I think one of the reasons is that when you do surveys right now, you get like an enormous spread like person A savvy. The other ones are not savvy. And then the effectiveness and yes and no, that's hard.

23:35 So I figured that what if we look at the socials on how they do things and kind of capture that signal that has a little bit of a problem. It could be vendors saying how they do great things and practitioners, but kind of the old ideas that we filter out, like how some of the people are using this and if there's many people doing similar things, that boils down to kind of a pattern. Now, one of the surprising and unsurprising things is almost like what is good for eye is actually good for a human and the other way around as well.

24:08 Like given that you're working with an agent, if you actually give your agent good documentation, lo and behold, it performs better. If you give them tests, then you actually know whether that still working yes or no. So if you have observability, you can expect itself. So that's kind of like, you know, when you start thinking about like this analogies on like what is actually good and what is not.

24:34 You can go up to rituals, for example, on the team we have a team lead and they're used to doing scrum and retros and kind of those things. But what if you would have an agent review kind of at the end of the sprint? What were the missing, how effective where they are. But it's not. Also, what do you do as a team leader? You set a goal for a person, so you go all set the measurement.

25:02 So that is another account. So so you find like people doing like those little ceremonies or for example, they do writing almost like context together as a pair. So you get nuances. So that's the pair programing but now with context. So that's kind of interesting that those signals are bubbling up in a certain way. Yeah. Goes back to what you said earlier about it's good software development practices.

25:29 Yeah. Everything that you mentioned, there. Should. Be things that we are doing today manually, but it's about almost like, can I say the word agent in a gentrification, you know, yeah, it's actually making an agent take care of all. Of that. Yeah. Yeah. But I think it's and you already mentioned it if you were.

2 Highlight · Shift from collaboration to clear instruction

25:48 Yes. It gets used, gets a little bit getting used to as what's the new world as a PPM and an engineer and kind of I feel okay contributing and working on that. But for example, if there's one advice that I would give a developer right now moving to agentic, it's just the most important thing would be don't repeat yourself. Like don't keep telling the agent what to do, but write something down.

26:14 If you're using a tool to verify whether it's doing, give it a tool so you're not. So kind of that is a very strong like almost like mindset. And it's different from oh, let's collaborate like not not collaborate like just having do all the work. And and that pressures you in all the correct engineering points. And I think one AI coding came up. Oh you know the typical is like this is the end of coding and there's no more engineering.

26:43 And lo and behold, you know, if you really want to get all the value of that, it is good engineering that we bring in there. So and the engineering becomes creating

26:54 the software factory or the assembly line or whatever you want to call it. I can imagine that for a lot of like, I can see a bifurcation for developers of the more product minded ones who like achieving outcomes and getting motivated by delivering a feature to users they find valuable, they're going to kind of become more product focused. And the ones who are like, I like making nice, tidy, neat code they're going to be more motivated by.

27:19 I'm going to make the best agentic software process that I can. I'm going to tweak the dials and the knobs on how the software gets generated. So that kind of engineering mindset doesn't go away. It's just you're not applying it to the code. You're applying it to the machine. That makes the code. Exactly. Yeah. Yeah. And I think that is interesting about we had that when we went to the cloud. The cloud. Everybody wants to like build their rebuild their own kernel.

27:46 Like why like it's good enough. We don't need everybody to build their own kernel. So if there was one advice that, you know, people come up to me over all the years of DevOps and I say, but yeah, you know what? We're special. It doesn't work for us, blah, blah, blah. Guess what? Like, you know, the industry kind of said, like that loop is actually universal. It can work everywhere within software development. And so my advice then to platform team would be saying like, okay, your number one focus is telling people that, like they're not so special in what they're building for code, but it's their mindset that is the difference in how they approach the problems and what they bring into that mix.

28:27 And it's really weird because everybody is repeating this, which is great. Like, every new technology has a learning phase, and we all have to go through the motion and kind of say, hey, oh yeah. Now I understand. And and yes, all the stories about like I vibe coded this app like all great, but how do we bring this into the maturity of an organization. And that's only by saying like, well, we're going to probably don't need to. Every developer needs to build their own pipeline. That's probably not very effective.

28:52 And it's actually the same thing over and over again with two variables or something. And that's kind of the mindset of the platform, the reusability across different teams that goes in. I can share a little bit about our process internally, and I guess like the R&D leadership perspective and trying to push, the way I think about it is I go back to pre gen AI, and I imagine I had all the manpower and all the time in the world, somebody opens a PR I would want to map out is it high risk, is it low risk?

29:18 PRs merge. I'd wait two weeks. I want to go over the logs and see did what they do. Was it successful? Was it not? Is there some edge case they can cover? And basically with that mindset we say, okay, now you have Gen AI, you have agents. There is no excuse not to do everything you ever thought you'd want somebody to do and then automate it so it happens by CI automatically. So for example here all of our peers get labeled by risk automatically by an agent. Cool.

29:44 And then they go to staging. They live in staging doing the QA process. Before we release we have another agent that goes for that is sorted by the risk level and goes through our logs. Our logs are variable for agents and they actually go and check every single PR. Hey, what was the intent? Hey, what are the logs look like? Did it cover what it was supposed to change or fix? Was there some edge case we missed? And it's been amazing that it's caught more and more stuff like the high risk PR is often do miss something, even though our engineers are agitated in how they work and everything, and we have QA and we have testing and we put in all this effort.

30:16 But hey, when you retrofit a week after it's in staging, you find more stuff. And often it's like easy stuff, like the agent knocks it out, it's like, oh, this edge case pops up once and never because we're not deterministic product because of AI. And but hey, now we see that and let me capture it too. So that sort of mindset of if I had all the manpower in the world, what would I tell them to do? People start thinking, agents do it once, twice, automate it for us. We put it into our release cadence and and suddenly you just get better and better quality and you move stuff that you would need to think about or your developers would need to think of.

30:49 Just move it, just automatically. I love that you bring up the risk thing, because there's the belief that the digital factories kind of will turn out and all the things, but because when you know what the risk is, you know, again, translating it to a management world, different risk levels require different management techniques, like are you micromanaging? Yes, probably the risk is very high. And then you need to do that. If you have guardrails or something that is not that important, or you have some way of mitigating this in the product, by all means, and we'll get the feedback later.

31:23 So that is the the levels and that you can only do that if you start thinking about like risk levels of like things pushing through. We even couple it to our PR fatigue. So high risk get more human resources to go over it. And we even couple it to our token budget.

Transforming CI and merging quickly with feature flags

31:38 So low risk we're not going to spend a lot of effort to check it from agentic LLM spending. So just a way to prioritize everything. It sounds a lot like running a business now, right? Yeah. I'm going to talk about adding to like. You know. Instruments is through your CI. Is CI still fit for purpose in this in this flow or are they all or changes we need to make to our CI style process to make it better for agentic worthless?

32:03 Oh, there's a key transformations you have to do. So. One key transformation is whatever your platform is, you need to make sure it's easy to query for an agent so you can verify both while it's doing the task and also retroactively on entirely, I don't know Deb environments teaching environment problem. You need to make it really easy for the agent. And we have skills that also tune themselves. So we continue to get better, especially as our product evolves and our law exchange and the way you query a change. So that's a key part.

32:29 RCI is very agentic heavy. And even the way you push an open a PR is extremely genetic heavy. And all the actions that we put in from a DevOps perspective are also like really agentic heavy. And that's just the way not to have like you want it to end, you want you to test. You want those mechanisms for feedback to the agent. So the agent works in higher confidence.

32:55 Yeah. But you got to think of how is the agent stepping in and how are they doing. More than just writing code. How are they your confidence layer that there is quality, that the risk is managed. The stuff is moving faster. We even have agents help us prioritize. Like I don't know, you have 20 PR open. Which one is important? The agents will tell us and push the reviewer to act on it. And you see it almost like, is this this series of actions that need to happen, or do you feel like it's more fluid than that where you have you have different, different agents providing you with different data to make decisions more, more integrated, more collaboratively.

33:31 Hey, so I'm going to go with your answer. Which is. It's fluid until I figure. It out. And then it's hard to audit cause I don't want to repeat it. I want to get that mental load off. I've got a strong hunch that or strong opinions really on this in that when you are a software developer and you're trying to work out like you're implementing a feature like one, just make it work, then make it work maintainable, and then make it work, readable, performant, securely.

33:58 You've got all these different lenses through which you have to look at a code change. I think that what we need with the genetic development is to apply all of those lenses deterministically, instead of hoping that an agent is going to figure out all of this in one pass by sticking all the things in agents MD instead, we can have some kind of CI like process. I built a little tool called Assembly Line that does this, where you make a change with your agent, or the agent makes a change by itself, and then another agent gets spun up with one deterministically defined prompter, check for dead code, check for missing test coverage, check for this, check for that, check for the other.

34:34 And to your point, you've got infinite engineering resource. So why wouldn't you look at a change through all of those lenses and then try and provide feedback as fast as possible to the agent that was doing the main kind of set of changes, so it doesn't deviate too far and can fold that back in. So I think the like we need to as an industry or, you know, as practitioners, figuring out that the flexibility of power of LLMs to work in a non deterministic fashion is really powerful.

35:02 But we don't want that all the time. I know of some people that are like using agent skills to figure out how to deploy stuff into prod, and I'm like just build a platform. Like we have platforms for ten years. They're nice and term stick and straightforward. Maybe use agents to build your platform, but don't replace your platform with agents. Burning tokens, reinventing the wheel. There's a place for the determinism, and I think it is in that kind of figuring out what you explore, and then you figure out, okay, right, that was the right place where we should have run that kind of check or a pass, a sweep over the code base through this lens.

35:36 Yeah. I have a ton of time to say, okay. Second. Excuse me. So we have two harnesses in my rendition. We have the harness in the product and the harness for the R&D team internally. So it's beautiful because what you said is stuff we have in both harnesses, meaning we do not want the agent to go through this route to exactly what we want. We want the agent to do a big step this way, and then we want that step makes sense for its statistically. And then a big step this way. Again, it makes much more sense to it statistically. You see that you're no longer fighting with the LM.

36:03 It does not want to do 1 to 3 steps in one. It wants to do one, two, one, two, three. You should get a better quality that way. And it's funny, you call it a semi line. We call it nudging. But in that sense you get a very clear workflow and you no longer fight the the agent and how it wants to work. What we do internally for our R&D team, the R&D harness, not in the product is sadly, we have to factor in the human element so we can just because we do.

36:33 So we develop humans. Yeah. It's like the slowest. Part of the whole thing is it is. So we develop kind of pump is what I call it.

3 Highlight · Why traditional pull requests fail at agent speed

36:41 Which is plan, merge polish. The code is moving so fast you can't have an open PR, meaning if it's an open PRT and I want to do QA on it, and I want to do like a product review and I want to do a design review. That is all lovely. By the time those are done, the code is going to have so much conflict. Everything looks different. I have to restart. I am going to be so, so happy. If we manage to go back to base development. As if it's. Like pull request based workflows inside enterprises are a damn silly idea.

37:09 They make great sense in open source repositories with people who are not strategically aligned and who don't trust, you know, it makes sense there. But like that, because of that speed change that you mentioned. Like you've got to get the feedback through the agents working on the code as quickly as possible.

37:24 And that kind of goes back to a point that I think is running through all of this, that the fundamentals of like what made good software delivery, what made good methodology, those haven't changed. Things like fast feedback are important. They always will be having feedback, being able to tell the point about your ability that the agent can tell whether it's broken something in production. Having all those feedback signals is was important before for humans and continues to be important for agents, if not being even more important because of the rate of change.

37:54 Yeah. So I'll spend 100% agree the issue there is also that as soon as preserved, the landscape changed, the code changed. So the next agent that starts will do a different job. Like everything impacts it, the specifics of the path that then goes through. And so for us we called it like the M in the pump, the merge. As soon as the PR is open, we want to get it merged as fast as possible. So hey human reviewer, look at it right away. Get it in. We need to pass our CI. We do not need to, pesca.

38:19 We do not need to pass the product or the designer. Everything is feature flagged. Then we go into the last week, which is Polish. Stuff is in the code. Every new feature that gets developed is off of the same code. And our product manager can go and review it, and they can polish it using our platform and get the UI and UX exactly right, and see all the bad decisions from the user facing perspective that the developer made. Well, the developer got the feature working. It did not get the feature polished, but that's fine.

38:45 We have other personas for it, and now they can do their work in a way that is smooth within the evolving code base, like developing really quickly. And so that kind of feature flagging, merging quickly with feature flags enables us to do it kind of two step. There's no one PR. A feature should take 3 or 4 hours, one PR from the developer, one PR from the product manager that changes functionality, one PR from the designer that tunes the UI weeks, and then maybe one last PR from the QA.

39:13 That's some sort of education and that's. A feature.

Automating code reviews using risk-based guardrails

39:16 And in terms of the the human reviewer being the slowdown there, I had a really interesting chat with Ryan Bobbili, who's a member of technical staff OpenAI. And one of the things that they're doing, some of their projects, is to entirely remove a human review of how much does that make you go, oh, of course that's horrible. Or how much does that make you think, oh, wow, there's a real opportunity here in certain cases for us to actually allow either an AI or tooling to.

39:43 To perform those reviews and give us that feedback, maybe for different levels of criticality, of change. But is that something that you look at much. Oh. All the time. So again, goes back to the risk conversation. High risk that way. Yeah I want a human person. And I also define that risk is also by what part of the codebase start touching how sensitive it is. But beyond it I'm trying to automate the PR, commenting and comment resolution process, which means I've agents have modeled every single developer on my team and their comments for the last few hundred dollars, and I know each and every persona.

40:16 So as soon as the PR is open, we have a whole prepared PR, which is a bunch of agents that check it in all the ways that we've modeled that we need to check it. Once it bars open, we launch different agents that model different people from our team and comment as if they were those people and doesn't surprise a good job. Then we trigger agents again to resolve those comments. Then we trigger the retro to see are there any error logs stuff actually behaved like you wanted to? And again, if the risk is not high from the PR nature, I'm moving towards being able to like skip the human reviewer at all.

40:48 Again, I have developers on my team, I have some developers on my team that are absolutely against this because they want the developer to own. So I'm going through the motion, massaging them and getting them comfortable with the idea. But I'm definitely. Pushing for that. And it depends how reversible is. As well. In terms of yeah. It's the guardrails. Right. And it was the same thing like when everything was being automated during the DevOps transformation. People said like, but if I did stage one thing, I can delete everything now.

41:13 Automated. Right. But it was the harness of test the CI/CD that kind of was like the fail system of and the narrative was you can have a junior come in, push to your, you know, kind of your main and like your harness your test harness will catch it like if that's a good one. And but what you saw is that it was still a difference between I'm going to do changes in the code of the compute to I'm going to do changes on the data scheme model because people felt the risk was higher.

41:46 So you can see kind of similar patterns on like the choices that you make. But it's a mindset like how far do you go? There's also a cost to kind of doing the automation. And you mentioned a lot of things like, hey, I have a nation for this, for that. You know, putting everything in place is, is, is currently like a lot of work, right? Like, and yes, you can buy your product or your product or like everybody's product, but that kind of is, you know, funnily enough, we hope everything was be more simple if I could and everything will work and what we end up and that's usual for any technology maturing, is that when it's matured, it's even more complex when you want to reach the next level of kind of like full kind of power of the new technology.

42:32 So one of our customers, they're doing some really great work where they've got a team that has gone fully. They changed their development process as to embrace AI in many different ways requirements gathering, transcription, story writing, implementation, all of those kind of things. And they are now going 72 times faster. So they have done eight years worth of work, and they're on track to complete that in 11 months with a third of the people.

42:59 So they've done great things there. They've given up on human code review. But each individual developer, they take an epic and then they some of them use beam add something. You spec it, some of them vibe before they self merge their PR, they run about 7 or 8 code reviews, different tools, looking at different things, and their confidence is they've been doing that. They're now confident enough that they don't need humans to look at it.

43:25 And I think there's something interesting. I don't know how true this is. I mean, data is definitely like when you screw up data, your data is gone. Do we need to move to event sourcing and having a, you know, a replayable history of data transformations? That's one thing. But like generally as a trend, we've always seen that trying to prevent bad things happen, like a check in place is the wrong answer. Like with data and databases, we went from like having transactions to prevent inconsistencies happening.

43:55 That was slow. It doesn't scale. So what was the answer? Eventual consistency and were reconcile after the fact with the cloud native transformation. We found out that, you know, you can't have 1000 microservices and then integration test them all. You just have to deploy them and find out what happens. So you need great observability and you need to be able to fix forward very quickly. So again you let the bad thing happen, but you react to it more quickly. I imagine that where we might end up with agentic coding is something like outcome driven development, where you've got observability of the business impact you wanted a feature to have, like is the application doing the things the application is supposed to do, or users able to do what they should be able to do.

44:37 And then we have software factories chucking features out, and then you look for regressions in like, okay, the number of checkouts on the shopping cart was dropped and reacting and fixing forward at that point. Is. That something you feel users are going to accept? It would be interesting to see in that. Like, do we all end up getting used to. Like. Buttons, moving locations and software being less kind of, oh.

45:05 Who is it that coined the term? I think it might have been. Steve might be using that, but of non-deterministic impotence of this idea of you have the same specs, same product requirements. You run it through an agent twice and you get two slightly different things, but you enable the same user behavior, just as we've got used to waking up and finding out that, you know, the Apple have decided to change, like how the photo album works and our elderly parents are now very annoyed.

45:33 Do we end up with something like that on a more daily basis, like this rapid churn of features changing and being slightly different, but like I can still achieve the outcome, I. Don't know. Yeah. There was. A testing Facebook for example. Like it has so many rules, so many features writing that kind of in a consistent test. And the way that they did it was they they added agents that were using Facebook and they had their own rules, like for example, like that agent should never get a friend, like whatever.

46:07 Like it should not be listed visible in anything. So they had these kind of like flags out and by all these kind of things, but they couldn't only make it work when they hooked them in on the production system. And that was a novel way of thinking of instead of the rigid test flow, almost like you track whatever behavior is going. If there's bad behavior, it's not like it's a different way of looking at observable instead of your logs.

46:32 But that was an interesting way. Like a digital twin is also something I think will have a more future is kind of predicting things. What is going to happen? And then the risk comes back like, okay, I have a model, I can run it through, I can see what's happening kind of those things. But yeah. It's funny, I thought about Facebook too, but from a different perspective. They have a huge infra set up for AB testing at scale, so any developer can do whatever they want and it'll go and get tested a b testing a facilitated again on scale.

47:02 And I think that's kind of what you were going for. How can you do them scale and just see what wins and which sounds great on paper. A lot of the organizations we talked to don't have the A setup for it. And then we're we get some organizations that come to us and hundreds of organizations that are I think the one I talked to last week was like just shy of a thousand people. And they said that their product is just downward spiral because everybody sit up there.

47:29 I think development is moving so fast. There is so much PR fatigue. The stakeholders like the PMS and the designers step up and go through them anymore because we think moving so fast and then like on the call, was like the head of the at the front end in front. And he says, yeah, I'm on calls and I discover new features on the call that don't look anything like their other features that don't even match our design system. That, functionality wise, don't make sense to me. And I find them out with the client live.

47:55 And then I ask the developer after the colleague, what is this? He's like, oh, this got merged two weeks ago. And then I asked, did anybody look at it? I did it was a part of your view. If there was like a B testing, that at least would be guarded in some sense and not just immediately impacting all of that, all of the users and again, huge organizations that's been around for over a decade, but they have adopted agentic coding really quickly without amazing guardrails.

48:24 And now they're kind of spiraling back to like, wait, wait, we got to control it. We have a huge brand. We have users that depend on us for years. I can't move the button and make it hidden because this is a core workflow. They need to work through quickly as a part of their business needs. But it's almost like it's the you know, it. Is the UI really going to be the way we, you know, interact with software these days? An agent will be absolutely fine with it.

48:50 It's us again, pesky humans, which is the problem. And how much of this will actually change? And you actually think, no, I don't I don't need a UI on screen. I need to ask certain things, or I just need my agent to go ahead and interact in a specific way. So I don't know. I wonder if in years buttons and things like that will just be replaced with completely different ways of us interacting with apps through. But we'll still be a visual oriented creature as well, right?

49:19 We can't just say we're just typing in the words like there was a talk here as well from Tell Draw. Kind of interesting that like, you know, having your agent giving it a whiteboard because sometimes you can express things easily in a whiteboard and like typing in and that kind of fluidity is also, I think in the UX, like sometimes it is going to be I just want to ask a question. You reply back to me sometimes, just show me like explain it to me.

49:45 But it might be interactive and kind of going through that. I think that's also the pendulum, which was funny, like I'd cli and now we're back at orchestrator review tools because we need something like we need to see what's happening and we can't just like do that. But the agent might crack along in their own communication and find the best way of kind of dealing internally. But as long as we need to make call shots on kind of those kind of risk, we need to be informed.

50:15 And somebody explains it to me like, what are we doing here? And that's a challenge if we don't do it ourselves anymore. So, right. we've talked a lot about good patterns, about ways in which people should, should change the way they develop software and to essentially, directing people and groups as to how they should change. But what is it that organizations, enterprises are doing today as a part of the migration, part of their adoption?

50:46 That is actually just a mistake.

Avoiding common pitfalls in enterprise AI adoption

50:47 It's wrong. It's a it's a it's been proven that, you know, dragons live here. What should they what should they not do that they're doing today? I think I'd give them the advice is that there's the still the tendency of like, if we're doing a constant conversation with our agents were in control. I think there needs to be for them to get into the next mile.

51:15 They have to let go. But it's almost like you have to mentor the other one instead of doing the work. And that's kind of the mental challenge that I think a lot right have right now. Like stop doing the work yourself. And it's really weird if that's what you've been doing for so many years. So actually hold. True to their identity. Yeah, maybe it's what they feel sometimes actually makes them the expert, right? Yeah, correct.

51:41 It's interesting. I have a few different answers to this. The first one is you want you want to control your AI spend. People are releasing crazy budgets, and everybody's trying to token max every single developer. And if you think the right way is give the developers attested interest, then you will see their way more engaged with the agent in the session where they're building it, versus give them the task that the developers not interested, and then they will off source all that mental load to the agent one.

52:08 You get poor quality PRS, but two, you get way bigger token spin. Like imagine that something's out the the disinterested developer will just say agent test it instead of testing themselves and seeing exactly what's wrong and saying, oh, this is the issue, let's fix it. Hey, we'll let the agent do seven sessions of QA, and the token count for the session goes crazy because the developer is not interested. And then so that's one part like, yes, people can fester.

4 Highlight · The danger of giving developer tools to non-devs

52:33 Giving them paths to don't care about is a great way to spend your AI budget really quickly. And that's when the other one that we've seen a lot of organizations do is they want to jump in headfirst to AI native A, and they do that by giving Claude Code to everybody. And Claude Code has code in the name. It is great for developers, I love it. It is most definitely not good for non developers. And if you give them that you will get PRS one.

53:00 They don't know what they're doing, they don't know to set up an environment and they're going to drive your dev team crazy asking questions. Even if they feel comfortable with asking questions or they just be not doing any meaningful work. But if they do open PRS, we see sorry, garbage in, garbage out. Pass that the VP we're on calls with like CTO and VP product, and the VP product shows off to us saying, hey, I open a PR yesterday and then the CTO sitting next to him says, yes, it was great.

53:25 And that also a great way to not go about it. So when you do enable, then you want to get the tool that is really optimized for that user. Again, we optimize for PMS and non developers, but giving the people in the tool, that's not meant for them, it's not the answer.

53:43 Again, you use your your AI budget really quickly and you get frustrated. People all around the PR fatigue is there in the developer side and VMs had open. Really poor PRS are increasing the PR fatigue. They're getting more frustrated themselves. And you get Uber scenarios where everybody is just disappointed. Yeah, I think in terms of what people should not do or stop doing, like the whole piecemeal approach to last year, 2025 was very much the year of CTOs going well.

54:14 I let people use their own tools and whatever they're comfortable with, like that is going to end up being inconsistent, and you're not going to get the organizational learning that you need. So a firm mandate that like the AI bus is leaving, please get on board. That's been shown in research by multitudes, I think, to be a strong indicator of success over genetic coding adoption. The the counterpoints to the token maxing is also don't go to the other extreme and set a really low limit.

54:46 You know, I've seen places where there's a €100 a month token limit and like €100 you can't really do very much. So it ends up adding cognitive load of people like, oh, should I use Cursor for this or should I? Is this the most important thing I'm going to do this month? So having something in between words would also be sensible and then maybe rethinking what a test is.

55:12 But you know, to invoke Beyonce, if you liked it, then you should have put a test on it. If you care about the thing, then you need to have something in place to assert that. Whether that's functional tests, whether it's a sweep of an agent with a particular lens looking at a concern, whether it is connecting up for MCP, your agent to observability so it can figure out the thing that it built and got deployed is now breaking, making sure that there are those feedback loops in place.

55:44 If you just have an agent on its own and you expect it to get everything right first time without being able to even perceive the mistakes it might make, then you're going to get bad results. So making sure that you think about what you value in your code, that it's architecturally sound and the code quality is high and the test coverage is good, you need to work out what you value, and then make sure that there is something in place to prove that that is there.

56:14 So it's not quite testing, but it's like thinking about the nonfunctional aspects of your code as if they were tests, if that makes sense. Yeah. Expand slightly. Love that point. And I think something that's really important is also that the the temporal mindset and the context hinting mindset, which means every test needs to stand the test of time needs to be evergreen, but you also need to think of it as an opportunity for context. Say, meaning when something fails, the test has enough context in whatever the assertion is, the comment, the error response, whatever it may be, so that the agent that gets it well know how to act so lazy.

56:51 Developers are just like, oh, I covered it with test no, but you need to cover it with test, and you need the test to have the context hints that the future agent that will see them will know what to do with it. Like that's been really impactful. On getting the agents actually managed in sophisticated codebases. Yeah, and it's exactly the same as we were saying earlier, if like that's always been good development practice. Like it's nothing worse than as a developer, you run the tests and you just get like assert true failed and you're like, great error.

57:17 Thirteen-seventeen, yeah. What? Like what do I do with this now? Yeah yeah yeah. It's important stuff. And thank you for quoting Beyonce because I'm trying to work out how I increase that. But finally someone has to say thank you very much. Well, we've talked about a lot and it's been super, super valuable to have you all kind of like discussing on these points. I appreciate appreciate all your input on that. Thank you very much for joining us at our native Dev Con.

57:43 Why don't you very briefly talk a little bit about some of the the sessions that you're each giving? Patrick, won't you start? Yeah, I'm talking about the different layers from up to a depth to a team to a platform team and a VP. What's the mindset that you need to have to go into this new agent world? I'm not talking about generic transformation like do a hackathon and so on, but what it's really like, how do you embrace the new thing? So that's my talk.

58:11 So my talk is about what it's like to become more of an AI native organization where Europeans, your non developers are actually opening PRS. And how really the the metric you should be actually should be checking is a merging so that it appears that they open. Don't say how many pairs did you open. Look at what the merger it is. That's how you will know. Are they open quality stuff or are they just increasing? Yeah. I was co-presenting with Tomasz my from Adam, and we were talking about the story of how we upscaled 120 of their developers in agentic coding.

58:43 And the most important thing about that tool is you get to see me in a dayglo fluorescent tracking. So you should definitely. Be answer. And this really. Is Beyonce in that way. I don't think there is a major. Sadly not. Okay. So why don't we if well, all of these talks are recorded. So you're very very welcome to go. Will add some have some links there in the show notes. If I or our listeners can only go and watch one talk, which one would you agree that they should know?

59:13 Now let's not forget that all of them. All of them. Thank you very, very much. Really appreciate you being here and sharing your time with us. Thank you very much. My pleasure. Thank you all. I hope you enjoyed that. That was really great from my point of view. We'd love to hear your feedback on that. Please let us know. Podcast at IO and tune in to the next episode by For now. Hope you had as much fun listening to this episode as I had recording it. And if you want to hear more from the guests featured in this episode, we have their full talks linked in the description below.

59:44 So that's it for this episode. See you next time. The AI Native Dev is brought to you by the package manager for skills and context. Your hosts are Guy Podjarny and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tessl offices in central London. Visit Tessl.io forward slash community to learn more and I hope to see you there.