Is Your Team Ready for AI-Driven Modernization? | Birgitta Böckeler

10,534 words · about 42 min to read · YouTube’s automatic captions

Reverse engineering precedes forward engineering

0:00 One big challenge that will be when we don't understand what's going on in our code base anymore, will that be a problem or not? Some people are saying, "Oh, that's not a problem because AI will also maintain it." But I've also run into situations where I've used a prompt and because of my lack of knowledge of what was going on in the code base, AI actually deleted important code.

0:18 I always say that code that was written a day ago is legacy code. When we think about true legacy code, we might want to migrate frameworks. These are massively different challenges that AI has to kind of tackle.

0:31 In areas where you do legacy migration, is that usually you don't feed the large language model with the legacy code and then turn it into new code in one step. But usually what you do is you look at how do we usually get from A to Z, from the COBOL to the Java, and how can we use AI on each of those steps on the way there? And so we took it the same way here. So we looked at first, how can we reverse engineer in the sense of like how can we create a comprehensive description of what this application does right now?

1:00 And then when we have that description, how can we use that to do the forward engineering?

1:12 Before we jump into this episode, I wanted to let you know that this podcast is for developers building with AI at the core. So whether that's exploring the latest tools, the workflows, or the best practices, this podcast's for you. Now, we've had amazing guests like Olivier Pomel, the CEO of Data Dog, Matija Šoškić, the co-founder of Eleven Labs, Victor Riparbelli, the CEO and co-founder at Synthesia, and of course, Tessels very own Patrick Dubois. And they're all sharing lessons right from the frontier of our industry. A really quick ask, 90% of people who are listening to this haven't yet subscribed. So if this content has helped you build smarter, hit that subscribe button and maybe a like. It really does help us keep bringing the

Generative AI dominates the technology radar

1:53 folks who are actually pushing this space forward and helping build that AI native future. All right, back to the episode. Hello and welcome to another episode of the AI native dev. My name is Simon Maple. I'm your host for the day and joining me today we have Birgitta Böckeler who is a distinguished engineer at ThoughtWorks and today we're going to be talking all about how AI can be used not just with you know new features that we're building in our applications, but as well as that looking more into the legacy applications and how we can unlock forward engineering with code that is potentially lost the source code, lost the engineers that actually originally worked on that. All these classic problems that we that we have with legacy code. Birgitta, very welcome to the AI native dev. How are you?

2:45 Hi Simon. Thanks for having me and I'm well, thank you.

2:47 Absolute pleasure and so you're a distinguished engineer at at ThoughtWorks and you've been an expert over the last few years on how teams use AI or essentially exploiting AI in software teams, how they can get the most of it. You also from what I understand did some work on the on the tech radar. You're one of the people who work on the tech radar which is a super cool way that people can really look at tools and understand which levels of maturity they're at and and whether or not teams should you know assess them, be adopting them, those kind of things.

3:21 Talk us through a little bit about about kind of like your role and how much fun it is to or not fun it is to to put that that tech radar together.

3:30 Yeah, so ThoughtWorks is a global software consultancy, right? And also in my professional like my full-time 20 plus years professional career I've only worked in software consulting. So I've seen like lots of different organizations build software and was part of a lot of organizations building software. Always like hands-on, you know, I'm a PowerPoint consultant. Um and um so at ThoughtWorks, I've been there now for almost 13 years, but the last 2 years I've had a full-time role to just like, you know, dive into the firehose of generative AI and large language models and how to use them for software teams.

4:07 And it's been uh a lot of fun, but also quite intense. And, you know, it's very interesting to be in a pressure cooker like this in the hype, right, and try to stay up to date. And I mean, yeah, one thing that can keep me a little bit grounded is the work on the technology radar as well, which is like we publish it twice a year in ThoughtWorks. Just a snapshot of what we see at our clients in the moment, right? Like as kind of like yeah, compared to things like, you know, big market reports, it's just a snapshot of what we see, which can be useful, you know? And uh actually in the last edition, we had over 50% of the entries, we usually put around 100 110 entries of technologies and techniques that we're seeing on it, over 50% of the entries were generative AI related. So.

4:51 Did one of them did one of them make it into the adopts this year?

4:55 I don't think so. I think one was in the assess. Yeah, no no, but you know what we have in adopt is actually retrieval augmented generation.

5:02 Ah, okay.

5:03 Because, you know, when generative AI first started, it started in assess because it was like a new uh technique, right? But it's it's also like high-level and abstract enough that even at this point, like still relatively early in the in the technology, right? It's something that can go into adopt because it's just like a a general technique that everybody's using right now.

Agentic tools shift developers into managers

5:22 Mhm. Absolutely.

5:24 But that's the only thing, no tool.

5:26 Yeah, yeah, absolutely. Let's let's talk a little bit about uh I guess how we have today um worked with AI agents in our existing workflows. And I'm I guess I'm here I'm talking about active projects and how we build uh with agents in in that in that kind of like uh typical software development flow. Then we'll lean into more about legacy uh applications, legacy modernization. How do you see, you know, you've got obviously got a lot of experience from from many different customers that you work with?

5:58 What what how do you see people using AI effectively in their day-to-day workflows today?

6:06 Yeah. Yeah, I think before I answer the question, maybe I should also emphasize once more because it maybe didn't get across quite yet that my focus is really on using AI for building software to improve how you build software, right? Like not using AI in products, right? Like putting building it into your products. And it's always a little bit clunky to describe that difference, but I think it's a very important difference to make because using AI for how we build software affects everybody in software development, everybody involved, right? Whereas like putting AI into your products only affects the subset of people that actually work on a product like that, right? So I see myself as a domain expert in effective software delivery, and I try to apply AI to this domain, right? So that's kind of just to focus it in on a little bit, right?

6:51 It's such a subtle difference, but really important. So at the AI native dev, that's what we can that's what we care about. We love we we we always think about how you use AI in your in your workflow. If it goes to production, then it's then it's not the workflow. It's actually part is the ingredients of your app. If it if it if it does go to production, but it's actually helping the way you actually build and deliver to production, then it's all about the tools that you use in the workflow. It's a subtle difference, but very important.

7:17 Yeah. And there's of course overlaps, right? If you think of an big organization that has products and they want to build AI into it and like for whatever their context is that it's actually worth for them to self-host models, have guardrails, have evals, right? You might also need those to build your tools for, you know, using it in the software process. So there's overlap, but I think it's a it's a good distinction to understand because it really helps focus sometimes. And yeah, so in that space when we talk about agents, I would say really that we have really agentic tools, agentic agents, big words that often mean very different things, right? But it only really started, let's say, exploding around February, March maybe, I would say, right? So, a lot of the coding assistant started releasing more agentic tooling support in autumn last year, I would say, October, November.

8:13 There were some open source tools that already were doing it quite early, like earlier last year, but you know, the cursor, windsurf, GitHub copilot kind of started end of last year. And then it started getting into everybody's consciousness with the vibe coding meme, right? Like the the Karpathy tweet was actually first week of February, can you imagine? That was 6 months ago.

8:34 Which year? It feels like about 5 years ago.

8:38 Um so, that's when like this agentic idea, right? Using that more in in coding became more um of the the step change, I would say, right? And what it means in coding assistance is basically that the the model gets access to even more tools than it had before, right? So, previously it was basically in the coding assistance it could edit files, um yeah, I think it was mostly editing files and often just one file at the at the time.

9:06 Mhm.

9:06 Uh and then now what we have is it can edit multiple files, it can access the terminal most importantly quite autonomously and execute commands there, execute your tests, and so on, and immediately react to the responses that you get from it, right? So, it really expanded the size of the task that a coding assistant can work on. And so, either in like a session with a developer going kind of like back and forth, or then also a little bit later, more of these products started releasing autonomous agents that you can run in the background, right? So that was open AI codex or cursor started introducing these background agents.

9:46 As you know, very early Devon of course was one of the first ones that did that. So and it's becoming more and more common in more and more products now. So where you actually send it off by itself, right? Without intervening in between. So I think those are also two that's also very important distinction the autonomous ones and like working with it myself in the in the workflow. And like as a developer working with it, I think that's the much more common one as well at the moment. So and then when you combine that with MCP servers, they become even more powerful. They have even more tools available. They can access my test database or they can browse the website in the browser and so on and so on, right?

10:22 So and with this expansion in the size of the task that we're working on um everything kind of get becomes different again. Like previously I was much more thinking of it like I kind of go maybe function by function, line by line and AI is there to help me be faster. But now AI is doing so much stuff that I actually often fall behind, right? So now there's like all of these much more sophisticated new processes and ways of working that we have to come up with as developers to even come up and make sure that something sensible comes out of this.

10:56 Yeah, absolutely. And we have actually essentially become the managers really of those of those processes. Patrick Patrick De Bois wrote really interesting piece in around parallelization of these agentic processes as well. Particularly those that are kind of like the more asynchronous that you you fire five of these asynchronous you know, agentic flows off and then then they come back and you can effectively review. Maybe they're doing different things. Maybe they're doing the same thing and you want to you want to pick through which one which one you actually want to take on.

11:27 But yeah, it's really interesting the the new role that we're taking on which we'll talk about a little bit later. Um so I guess in in terms of, you know, when when we when we have agents

Forensic data capture for legacy codebases

11:41 um writing a bunch of code for us, um it's very hard for us, I guess, to feel almost like connected with that code in terms of intimately knowing how it's been built, because we tend to click accept, accept, accept, and and we and we don't tend to necessarily give it the the the level of granularity of of of checking as we as we want to. Um it kind of almost immediately turns into legacy code, right? Because when we think about I you know, I always say code that was written a day ago is legacy code. It it's any any code, really, that you don't fundamen- that you aren't fundamentally actively changing, in my opinion, can kind of be considered legacy because you have There are There are tasks you need to do to actually truly understand what is this code doing, particularly if someone else wrote it. What it What is this code doing? What is it, you know, what what what was in the developer's head when when we first when we first kind of look at this. And when we think about true legacy code, where, you know, you may not even have access to the source still, but you have this, you know, you

12:41 you you have some application that's kind of like running. We might want to migrate frameworks. We might want to change the platform it's running on. How do we These are massively different challenges that AI has to has to kind of tackle. How would you say Do we use the same Should we use the same tools to to to to make changes or to even understand what legacy um code is doing compared to forward development or or should we look at a completely different set of agentic processes as well as tools?

13:14 Yeah. Yeah, legacy migration or modernization, I think, is a great example of um like uh almost like uh initiatives and larger projects that we can run where we can really think through how do we use agents or how do we use AI in general, right? Uh uh in comparison to, you know, just the day-to-day usage of a coding agent, right? Where we have all of these typical practices now. Oh, we plan, we break down, and so on, right? Um but there are like opportunities when we have larger initiatives on our teams like a legacy migration, which is I think the best example. It can also be other workflows, right? Where we can actually uh more sit down and think about, okay, what is a workflow here? Like what uh do we want to write prompts that we can reuse again and again because we will have to do this for 50 components or um so I think that's where those two areas we talked about before start overlapping again as well, right? So now you actually want to build an agentic system that helps you with your initiative, right? So now you have to know more about like how do you build build an

14:15 agent yourself or what tools do you have available, right? And so I've recently worked um on a uh on a few things related to uh this legacy migration. Like the the the first thing was that um so we we did indeed have a client, like you were saying, who said, "Oh, we actually don't have the source code for this application anymore."

14:35 Mhm.

14:36 So, you know, imagine like a bad vendor breakup or what it and it's apparently not that uncommon, I've since found out. And they said, "Okay, we haven't been able to upgrade this application in quite a while. We need to do something. Can AI help us, right?"

14:50 Even if it's a legacy Even if it's a security concern like

14:53 Yeah, absolutely.

14:54 there are vulnerabilities that are you know, all over the place that have been found. It could be just general maintenance, yeah.

15:00 Huge operational risk. Also, they couldn't upgrade the database because like the newer version of the database

1 Highlight · Two-step legacy migration process

15:05 had some like major changes in dialect and so on. They couldn't change the SQL queries that were in the code, right?

15:12 Wow.

15:13 Um and so um what we have seen now time and again uh in areas where you do legacy migration, also with those of our clients who have large COBOL code bases still, is that usually you don't go from you don't feed the large language model with the code just the legacy code and then turn it into new code in one step. You don't do that anyway, right? It's because just too far away from each other, right? So, but usually what you do is you look at how do we usually get from A to Z, right? From the COBOL to the Java. And how can we use AI on each of those steps on the way there, right?

15:52 And so, we took it the same way here. So, we looked at first how can we reverse engineer in the sense of like how can we create a comprehensive description of what this application does right now? And then when we have that description, how can we use that to do the forward engineering, right? So, there's some things that have been around for quite a long time that are typical data capture approaches that we do in legacy migrations also before AI, like a change data capture, right? So, in one of our experiments we put just like triggers on the database tables, had, you know, every update and insert was kind of recorded into an audit table.

16:33 And so that we could just see like in the running application what happens. Or maybe if you have an event-based system, you could like capture events in the running application. Or what you can now also do with AI is you can have AI browse through the application, especially when it's a web application. Doesn't always work as easily as with a web application, right? And can have AI describe what it sees as it navigates through the application, right? And have AI explore all the different click paths.

17:03 And then you can tell it every time you click after that, please check our audit log in the database and see what changed in the database when you click that button, right? So, you can try and create all of these uh uh descriptions of what is happening.

17:18 It's It's almost like it's It sounds like it's kind of like a

Custom MCP servers enable flexible legacy analysis

17:22 like a forensic exercise really, isn't it?

17:25 Exactly. Yeah, yeah, yeah.

17:26 It's It's incredible. Like when when you I guess first of all I got I got a question in terms of the tools. Like are there tools specific tools that you're using for this or is this something that you would expect needs to be needs to be handcrafted? Um and I and I guess my my second question from that is when you when you gather all this data how do you how do you even think about which context is most relevant? Uh so that when you actually do the analysis, you're not overwhelming the LLM with that huge amount of context, but you're providing it with what it needs to make a good decision.

18:00 But I'd love to start with the tooling one. How How would someone even think about doing this?

18:05 Yeah. Yeah, so I did uh we found that actually you don't need like a fancy gigantic platform to get started with this and do something there. And my suspicion is also that because of the nature of the types of system where you have to do all this forensics and that are really old, right? They it's probably going to be different every time, right? Just if you think about I don't know SQL or even COBOL or stuff like that. There's so many dialects there as well, right? When it comes to COBOL, for example, we have a an accelerator that we bring to our clients in in ThoughtWorks that we customize uh every single time we use this for our client. So where we say, "Okay, do you need Do you have COBOL code? Do you have C code? Do you have uh you know, a particular dialect of some other language, right?" And we load that into a knowledge graph to get like a much richer uh yeah, data set about what's happening. And the um advantage of this customization is not just that you might have different languages, different dialects, and so on, but also you can

19:06 decide based on the goal you have what else what do you want to load into the knowledge graph, right? So, just imagine if you enrich the knowledge graph with functional descriptions of the code uh by an LLM, right? Then, you can actually, when you ask the knowledge graph about things like what is happening in the code base, what validations are we using in XYZ, if you get more functional answers back, you can also provide this to a lot more users, right? There's there are users who don't want to know if the name of the function or the module or but they just want to know what's happening in the functionality, right? Or you can find ways to maybe explore what are the different capabilities here to find the seams to break things apart, right? So, that would be a more customized approach where maybe you, you know,

2 Highlight · Building throwaway MCP servers on the fly

19:55 you have really large code bases or you have a specific goal or something.

19:58 Yeah.

19:58 But to get back to that idea to that uh example I was talking about before where you have lost the back-end code, but maybe there's some other stuff, right? So, what we did there is just like work with an existing coding assistant, right? So, uh and just then point the coding assistant at the things that we have available, um and um also use a few MCP servers. So, for the web browsing, you can use something like a Playwright MCP server where that also like knows the DOM structure and doesn't just go off of visuals, right? And then, I also usually like to just create small MCP servers on the fly because they're really easy to create actually, especially if it's like it's like a little tool that you might throw away later and you don't need it anymore, right? So, that's what we did for example for this data capture thing. So, where the MCP server was just providing a way to get a current timestamp and to query that audit table.

20:56 And so then the agent could kind of go, "Okay." Could get the current timestamp, click on a button, and then ask the audit table, "What changed since that timestamp?"

21:05 Mhm.

21:06 So, that was the MCP server, right? So, um and I think that's really great because

21:11 the uh in this combination of like coding assistance, MCP servers, you can actually then build your workflow that you need, right? And because that is so flexible, um that helps us in the situation where every legacy situation is slightly different.

21:27 Yeah.

21:27 Right?

21:28 Yeah. Inter- very interesting. And I guess the question is with all of this, is the goal then to create the specifications or just to get the AI to enough information to be able to develop new new new features, new new make changes on top of that?

21:47 Yeah.

Verifying AI specifications against legacy risks

3 Highlight · Validating AI-generated specs with domain experts

21:48 So, um how we approached it is to in the first step, which we call this like reverse engineering step that just creates a description, just create uh very very comprehensive description.

22:00 Mhm.

22:00 And the reason for that is that then when you have that description, you can actually do more things with that description, right? For example, you can go to subject matter expert and experts and sanity check if it's actually correct, right? Cuz you might not even have all of the information. In that case with the client that didn't have the back-end code anymore, we didn't know what else is hidden in that back-end code. Is it calling any other services? We also tried our hand at network data capture a little bit, but not between services. So, um we actually when we decompiled the code, we found some uh some signs of like mainframe frame calls, right? So, we had all of this forensics, but now we needed uh I don't know, somebody who lived in the times back in the day when all of this happened, who can like kind of uh uh double-check with us if it's actually correct.

22:49 Yeah.

22:50 Or what you can also do then with that specification is you can enhance it with other things that you want to modernize or change, right?

22:57 Yeah.

22:57 So, um we have a team in ThoughtWorks that's working on an open-source medical record system applica- application that is open-source and has been around for a long time. And so there's a front end that's still Angular 1. So they really urgently needed to upgrade the tech stack, but also they wanted to include more compliance with a standard in the medical records domain, and they wanted to improve the what how the front end looks, right? So now they could they would go through the front end components, create descriptions of what they do today, and then enrich them with those standard standard requirements, and you know, how they maybe wanted to change the UI, and then

23:39 use that to generate the new Again, under supervision of a developer, but you know, the developer would be very accelerated.

23:47 Yeah, no, absolutely. I guess there's an interesting piece there that that that I think sometimes is is easy to easy to to, you know, pull someone in and and just say,

Overcoming the lack of tests in legacy systems

23:58 "Can you look at this?" and they'll understand it, they'll know what's right and wrong. Uh sometimes we just don't have that luxury like you say, you know, we have to go back into the day and hope there's a developer or someone who understands that. Of course, with with legacy code, the the as we say, the what and the how are combined together in the code, and it's really hard to be able to identify, was this written like this because for you know, for for a reason? Was this part of the the intended behavior? Or is this just the implementation? And I guess with One thing that really uh to some extent allows us to understand that expected behavior is tests. And tests are ultimately the a great way of being able to say, "Right, my change hasn't regressed anything."

24:44 Um now, with legacy code, of course, we're very very lucky if we have the source code, let alone tests that come along with it. What what in your opinion or in your experience, like how common is it to get you know, a well-tested legacy or or a legacy application with a good set of tests that actually allow you to perform a reliable change and get that validation.

25:10 Mhm.

25:10 Um and and I guess lacking that, how do you create that that safety net essentially? Whatever changes we make aren't going to adversely affect parts of the application we didn't even think about.

25:23 Yeah. Yeah. I mean, this real case that I was talking about, there were no tests.

25:28 No tests at all.

25:28 And also, even if for the back end, let's say there were unit tests somewhere. I mean, that doesn't help us, right? Because it's unit level, right? Like, the main thing that would have helped us is if there had been like a big end-to-end test suite or something like that, right? And in fact, like what we also looked into, but you know, it hasn't progressed to the point that I can say it was really successful or not,

4 Highlight · AI-assisted end-to-end testing for legacy migration

25:50 is we also looked into, you know, does AI can AI help us accelerate building almost like um uh fidelity fitness function. Using fancy words, right? Like an end-to-end test suite that can help us test parity that we can actually point at both applications, right? Um and I think on the one hand, uh frameworks like Playwrights or kind of like web browser testing frameworks have become a lot more tolerant to different selectors in the DOM, right? But with AI, you can make in theory make that even more flexible, right? Like, so if you have end-to-end tests that actually run on AI, you can say um find the plus button in the column.

26:32 Mhm.

26:32 Right? And it will try to look for that, right? Whereas, so you don't even have to use a selector, right? So, it gives you a little bit extra um flexibility. And also, our idea was maybe we can even use that specification that we generated with AI and use that as an input to write that end-to-end test suite faster, right? All with the goal of ultimately having a test suite that we can point at the old application and at the new application. Right? I mean, it turns out browser tests are already notoriously flaky. It doesn't necessarily help when you introduce more non-determinism into it.

27:06 So, we didn't like we didn't yet like push it far enough to say is this really viable or not. I have a little bit of doubts, but I think there's there's some potential there. But, that's one way to to do that. And then overall, it's like a constant risk assessment, right?

27:23 Constantly when you do this, you think about okay, what's the probability AI got this wrong? How can I mitigate that? How can I find more information? Like, you know, we decompile the binary to see if we missed anything. How complex is the application? What's the business impact when something is wrong? Right? How what else can you mitigate? Like, again, classic legacy migration roll out strategies. Like, can you do a staggered roll out and start with some like or maybe do like compare when the new application is running for users, compare results of the new one with results of the old one. This is all classic legacy migration stuff, right?

28:00 Yeah.

28:01 So, it is a constant risk assessment. This particular application wasn't particularly complex. So, that's like a that's like the good news, right? It was a bunch of forms. If you have a more complex application, then yeah, you have to also put a lot of work into this risk assessment and mitigation and like how do we double-check and

Reasoning about decompiled assembly code with AI

28:21 what else can we do to test? Yeah.

28:23 And what And what is your What would you say like from a success rate point of view in terms of obviously simple simple applications, you can kind of really get an understanding of of what it should do and and you know, I I would expect some one once you have all that forensic information, an AI bot or agent will be able to come in and and and fairly easily make certain changes that are a low risk.

28:50 With a more complex application where there are impacts if you make certain changes in certain places,

28:56 Mhm.

28:56 what kind of like what's your success uh expectation um for you know, not obviously Yolo one prompting it but

29:05 Mhm.

29:05 it you know, does it does it take a lot of iteration to to get there? Do you get to a state where you know what, you're actually really confident that there are no regressions this can be fully

29:16 Mhm.

29:17 pushed out?

29:18 Yeah.

29:18 What's the end game?

29:20 So, with the simple application uh where we only went through the kind of like development and evaluation of an approach, we didn't actually start um um building it but uh I was really impressed like how with the results, right? So, like I said it was a bunch of forms. We had front-end code, schema, stored procedures, and the binary, right? And screenshots.

29:40 Mhm.

29:41 And uh it was really impressive and satisfying how quickly you could get a really comprehensive description that also made sense,

29:49 Mhm.

29:50 And even um it was also really cool to see uh like two of my colleagues went deep into the decompiled binary and like fed the assembly code to AI and navigated the assembly code together and tried to find the function that, you know, when you click this button, what is the function that is being called, right? Um and

30:10 on.

30:11 Did AI navigate the assembly code there? So, uh there was a lot of manual preparation work at first that they had to do. So, to kind of like split everything in more into multiple pieces because you couldn't feed it like everything at once. Assembly is also super verbose, right?

30:25 Yeah.

30:26 Um and then you know, there were some like uh hints of where things were because you could like see string names of stored procedures or something, you know, so you could have some hints. And then my colleagues basically paired with AI to try and like reason about the assembly code and come up with hypotheses and theories where this is and then to actually find the code where they were pretty confident that's that's where it is. And so ultimately it was more like a confidence booster for the results that we already had from before where we had AI infer what is happening, right? And we would have later also been able to do some change data capture which would have given us even more confidence. But like I said, they found things like, "Oh, there seems to be a mainframe call here. We should find out what what's happening there, right?"

31:12 Yeah, and it was just super interesting to see how well a large language model could reason about what like could turn the assembly code into pseudo code so that we could read that as humans, right? So

31:24 Yeah. It's It's amazing. It's kind of scary at the same time when you say, "Oh, look, there's a mainframe call there." That's like It frightens the heck out of me thinking, "Oh, gosh." All these things that you wouldn't expect or just just like find out. It's incredible.

31:37 And also something in terms of the confidence, something that we saw we sometimes saw the large language model that we were using often like wasn't following our instructions very well.

31:47 Right.

31:47 So then what you would do is like you kind of prepare your problems, what you want to do, you know, so problems that say, "Here's the front-end code. Here's the screenshot. You know, here's the schema. Try to infer what's happening or like what are the validations." You know, that's kind of the questions the prompts would ask. And then we would run those multiple times for the same form, right? To see like how different are these results so that we could kind of see if we get like basically the same result every time, then we were relatively confident that it was okay.

32:22 And together with the human sanity check, right? But so that's how you can kind of see like the fidelity of your tool, right? Is it like totally off the rails

Combining LLMs with deterministic code mods

32:31 or does it give you something that seems to make sense?

32:34 Yeah. And what are examples of like I guess, you know, some of the some of the more major changes that you've made in terms of the legacy modernization? You mentioned from COBOL to Java, obviously, kind of pretty pretty major pretty major, but also, I guess it's it's a it's a fairly common but when you think about modernization, it's it you know, a COBOL to Java isn't the the most unusual um path, I suspect. But is it like, you know, would you also do things like framework updates and things like that? Do you have complete changes of architectures?

33:07 Anything anything kind of like more major in terms of your forward engineering?

33:12 Yeah. I mean, the framework upgrade was kind of the example I talked about before with Angular 1 to React and then but then there was also additional modernization that needed to happen, right? Um I think there's probably a lot of different things that you can do when you think about this as like, what are my data sources that give me a description? Right? Which can be like a lot of different things, right? Um But maybe like when you talk about framework upgrades, like there's a slightly different category, I think as well. Like when you just really think like upgrade in the sense of just a technical upgrade, right? Like Java language upgrade or framework version upgrade that's maybe not quite as uh uh jarring as Angular 1 to the latest Angular version, right? There's actually uh tools that help you do that as well, right? Deterministic software, right?

5 Highlight · Combining LLMs with deterministic code mod tools

34:03 So, code mods like uh Open Rewrite or uh uh Source Graph can do a lot of that stuff. So, there's like lots of tools that can help you write recipes and advanced patterns to do this, right?

34:15 Mhm.

34:15 And um there's also really interesting potential in combining large language models with those code mod tools.

34:22 Mhm.

34:23 So, kind of either letting them help you write the patterns that you need to do the to do the upgrade. Um Yeah, or like like do that, right? Kind of or and then where the where the where the deterministic software that the pattern patterns just don't are not enough, then maybe you try to fill that in with LLMs, right? So, I don't know if you remember a lot sometime last year there was big headline by Amazon that they had saved thousands of developer years by using AI to do a Java upgrade.

34:53 But under the hood they were using OpenRewrite.

34:55 Yeah.

34:56 Which is like one of those code mod frameworks.

34:58 Yeah.

34:58 And my suspicion is that OpenRewrite was doing a lot of the heavy lifting, right?

35:02 Fair enough, probably together with AI. I'm not saying that whole setup, that whole tool by Amazon was a bad tool, right? Um but I just found it a bit misleading to say that AI has saved all of those developer years, right?

35:14 Yeah, yeah, yeah.

35:15 The the pairing up of AI and like good old software tools that we have,

35:19 that's where the magic happens, right?

35:21 And and I think that's where you kind of like mix the creativity alongside the determinism, which is really important. And I think this is In fact, we had the we had the the the CEO of Modern, who they do a lot of uh on on OpenRewrite on just the other just the other day on the podcast. And um yeah, he was talking about using how they essentially use the AST, the abstract syntax tree of the of the of the application. And they're able to kind of like truly understand the deterministic flows through the app. And I think particularly when we look at something like this, when you think, "Okay, there's a big there's a big change that needs to be made. I want to make sure the flows are going to be the same uh or or or you know, perhaps in a slightly different more abstract way, but it it needs to effectively be a deterministic yes, this is working the same or no, it's not. And AI can't give you that, but mixing I love I love that you kind of like mentioned that. Mixing this like deterministic AST or existing tools that

36:21 we've been using for so long with with the creativity of an LLM and being able to kind of like, you know, make those suggested fixes and and and and then we test against more deterministic methods I think is a is a really strong a really strong plan. We've seen it time and time again. Sneak does that as well and and and Sneak code and many other tools have that combination of determinism in in in their AI solutions.

36:44 Yeah. And we have to think a lot more in probabilities now in like on the meter level as software developers, right? Because of this. So if you think about that situation where you cannot upgrade an application anymore, which is a huge risk, right? So it's better to replace it and risk some bugs than not do anything, right? But now you but you can employ different things to accelerate that replacement and to increase the probability that you're getting it right. Yeah? And then you constantly have to kind of think about those trade-offs and and risks and how you fill the gaps and so on.

37:16 That's a really good point. So like obviously AI can kind of like will you know more and more generate more reasonable applications at scale. Will we do you feel like we'll get to a stage where a legacy app won't won't ever exist anymore? It's just an app that at some point we can just kind of like it particularly for the you know some of the smaller ones. Why would I try and do all of this? Why wouldn't I just why wouldn't I just rebuild this from scratch? Do do you think like that's in the in the near term going to be a realistic approach again you know you can build it on a more modern app much much or a more modern stack much much more quickly and and the risk of what you know why pull something from a legacy an old code base? Why not just rebuild it from scratch and understand exactly what's happening under the covers?

38:08 Yeah. Yeah, I think that that might happen, right? Like it's it's this effect that we can already see with like organizations that have like hundreds or thousands of microservices, right?

38:20 Yeah. Yeah.

38:20 What I often see happening with developers is they're working on a service and when it gets to a certain size, often that's not even that big. They're like, "Ugh, this is too big. We need to split this up. We need to create a new one." "Ugh, we don't understand it anymore. We can't reason about it anymore." And then you end up with like two entities in every service, right? Which is also not the right way, right? Like maybe we should just get a little bit better at modularizing inside of our services, right? But this psychological uh how do you say uh reflex, this reflex shows you that this might actually now happen that, you know, people's reflex will be to like, "Ugh, we don't understand this anymore. Let's just quickly rewrite it with AI, right?"

39:00 Yeah.

39:00 Which maybe sometimes the, you know, sitting down and actually seeing how can we make

Evolving developer skills for AI workflows

39:07 this better might be the more effective way to do it, right? Let's see.

39:12 Yeah. Well, let's talk about humans and and talk about how we as developers in the loop, humans in the loop as we call it. Um how we need to change to be better in this process, to actually continue to, you know, run this process. There's a ton of stuff that we need to be good at in order for us to actually uh own the process and and and run the process. Uh you mentioned a ton there in terms of like getting a bunch of those metrics and actually working with AI, almost like pairing with AI to get all that correct information. I guess it's it's still an understanding of knowing what information we need, what information is important, uh and really I guess understanding the application.

39:55 But um what else do we need to be better at in order to um or or learn new skills we need to learn to be efficient and effective in this type of in this new loop.

40:06 Mhm. Yeah, that's a big topic. Also, everybody's worried about the new people coming into the profession and you know, who who don't have the scars from like from the past and you know how do we build up new skills and I also don't have the answers to all of those things. It will be very interesting definitely and I think for experienced senior developers and software engineers it's now our responsibility to figure out how we can do that for the next generation, right? Because like experienced developers don't just grow on trees like we all develop by running into problems and you know have being called up at 2:00 a.m. in the morning and then thinking I'm not going to do that again, you know.

40:50 Um So yeah, like what are the skills? I mean um One big challenge that we were talking about this before will be this like when we don't understand what's going on in our code base anymore, will that be a problem or not, right? Some people are saying oh that's not a problem because AI will also maintain it, right? But I've also run into situations where I've used a prompt and because of my lack of knowledge of what was going on in the code base I it was a let's say subpar prompt and AI actually deleted important code because I wasn't aware of that impact of that you know that oh I should tell it but don't touch that, right?

41:33 Because so um So that's going to be a challenge, right? I think we should still roughly understand what are the components in our code, what is the dependencies, what is when I change this, how many other places are calling that, so how risky is it to change that and so on. Um And then uh yeah, we I hope we will get better tools to help us review large change sets, right? So we're used to just reviewing text diffs.

42:05 Um but maybe that's not scalable anymore. And maybe there's even like new ways with AI to help us visualize more, to help us pull together lots of like automated information that before we didn't put together because we thought, oh, I I went through all this code, I wrote it myself. I don't need that static code analysis, that coverage information. I know what I did. But now if I go for lunch uh and I send off an agent and I come back and it has changed 50 files, you know, maybe I want a dashboard that shows me everything like uh you know, like an impact analysis or uh you know, um yeah, so I'm wondering about that. I think we need more tools like that and not just rely on this like text diff.

42:49 Um it's really interesting cuz I guess it's it's the question of what we're reviewing. Are we are we trying to review code, which actually testing would be a better validator for or are we trying to test decisions? Are we trying to review decisions? Um uh and we almost like want to level up uh uh code uh uh reviewing to that decision level and have almost like your your code more tested through or validated through tests. I think there's probably a a a journey that we need to go on for that for that to be true, but

43:21 You have to review the tests.

43:22 Yeah. Yeah, 100%.

43:24 generates the test and I've seen AI do make so many mistakes when building tests like tests that, you know, don't do anything, too much mocking.

43:33 Yeah.

43:34 Uh maybe making the wrong assumptions in the test data and then you think I have a uh test that is green,

43:41 Yeah.

43:42 but the tests are actually testing something that is wrong.

43:44 Yeah.

43:44 Right? Um so, that's on the next level. Yeah, okay, generate all these tests, but how do I actually know know know that it did what I wanted, right? And especially when it changed a lot of things, how do I find out quickly?

43:57 Mhm.

43:58 And well

44:00 I was going to say biases is another area in and around there, right? Where where you know, there's going to be if we're if we're allowing AI to could generate tests and do all these things, generate code, and make decisions, we need to be very aware of our of the biases that in the decisions that are being made within AI. Is that something you know, or rather, it is something that we should absolutely be aware of.

44:28 How can we though? How how can we train ourselves to to really look out for when biases within the AI having an adverse effect of you know, of of what is being generated?

44:40 Yeah. Yeah, there's the biases in the AI and our biases as well, right?

44:44 Very true.

44:45 Like in the in the AI when it comes to coding, I don't know if I would call it biases, but I've seen it do things like fill in gaps in what I prompted it with, fill those gaps with assumptions, right? Like that's a thing or as we all know, they're very sycophantic, they really want to like please us and fulfill you know, our task what we ask them to do. So yeah, like I don't I don't know if there's something about biases as well when it comes to domain logic. I haven't really thought about that yet, right? Like might it lead us to implement something biased in the application, I hadn't thought about that yet.

6 Highlight · Overcoming AI sycophancy and sunk cost fallacy

45:22 But then on the other side is our biases, right? So for example, because AI always frames everything so confidently, right? That even though we know it's a machine and it's a stochastic parrot and all of that, we still it influences us, right? Oh, it's so confident confidently saying that's a best practice. So it must be a best practice, right? And often we also just like skim over these words and like our eyes see best practice, blah blah, and then we're like, oh yeah, sounds good, right? Or you know, once we get a solution from AI, we might be anchored to it, right?

45:57 And then it makes it harder for us ourselves to imagine other solutions when things go wrong, right? Or there's a strange sunk cost fallacy at play here as well, right? Like AI generates a lot of code and it doesn't really work and then sometimes we spend 2 hours trying to fix it instead of throwing it away. Because it's already there, right? Ah, but it's almost working. And so often it's it's it's similar to like fixing forward when your production is broken versus reverting it, right? Just reverting is so much faster just to fix it, right? And that's what you have to do with AI code a lot as well. So yeah, critical thinking, going back to your question about the skills, right?

46:38 Critical thinking and yeah, understanding all these probabilities and um constantly getting over ourselves with um all of these psychological effects of this manipulative technology, right? Going to be interesting.

46:56 Absolutely. And and of course it's not

Maintaining team control amidst accelerated delivery

46:58 just going to be us that need to be better in in, you know, as the human in the loop, but as a team everything's going to you know, the delivery's going to change. Um the the speed at which we deliver, the quality, um and and I guess from a delivery manager point of view, when everything's happening so so much faster uh around around us, what do we need to be more conscious of? What do we need to What do What Are there any new best practices or things that we need to be better at as a team in order to do this uh delivery um in more effectively?

47:34 I mean, all of the good engineering practices that we've developed over the decades, right? Still very much matter and Jenny AI amplifies indiscriminately, right? So if you have like a bad setup, you might just amplify that bad setup, right? And if you have a good one everything might go well, right? So this speed that you were talking about, that's going to be the linchpin kind of, right? Like can you actually in your organization with your pipelines, with your processes, can you support an increase in throughput, right? So, um when, you know, can you test faster? Can you fill the backlog faster? Um can you actually deal with that when you're starting from a continuous integration setup that is flaky?

48:19 Right? Or test setup that is flaky, right? So, um I think a lot of organizations are still underestimating foundation that you need and also good repeatable automation to make safety net, to make sure that things are still going well. And um yeah, I mean, me as a developer when I see AI go off the rails, I often roll back, like I was just saying, right? I try to like let go and roll back.

48:45 But when you are a delivery manager or product owner and suddenly your team is like constantly like turning out more stuff and all the moving pieces around here are moving even faster, you can't just roll back, right? So, um yeah, I have a feeling that uh we're still underestimating this the control that we're losing, right? Like, maybe that's just like I mean, as humans we want to be in control and maybe we also have to let go in some places and just let AI do its thing, but I have a suspicion that that's going we're going to run into trouble there with like when we just lose control over what's going on because so many agents are running around doing stuff for us, you know.

49:24 Yeah. Yeah, no, absolutely. Um

Preserving foundational debugging and safety skills

49:27 let's finish up with uh I guess the age-old question, the hardest question to answer. In terms of well, what what happens to me as a developer? What what What do I do? Do I Do I need to go be a farmer somewhere or what should I do? Should I How does my role look in the future?

49:46 I mean, as you said at the moment, I still see a lot of need for the classic experience and skills, right? When I was talking about the legacy migration and you called it forensics, right? For the forensics, you need to know where to look. You need to know you know to uh where where the corpses are in the basement, right? And include them in your data sources.

50:07 And you mentioned you mentioned about the junior versus the senior dev. I'd love to see like it's almost like I love to call it Martin Thompson used the phrase mechanical sympathy. And it's about about you know having that sympathy of how things are built underneath to actually be able to have that have that ability to actually understand how to use your tools or use the system. I think if you was to look at a junior versus a senior individual in that forensics style exercise, you know that the senior developer is going to go to the right path, the right pieces because they know what they're looking for. The junior will probably ask many good questions but have so many many emissions as well, right? And it's it's it's interesting that when you when we talk about that future developer role the senior developer of today will be well equipped for that.

51:00 How does the how does the junior developer learn some of those skills?

51:04 exactly.

51:05 Do they do they just follow the forensics of the developer of the senior developer? It's it's a different level of learning, I guess.

51:11 Yeah, yeah, yeah. And then I mean legacy migration is a specific case where you've always needed people who maybe are still familiar with some of like what is an application server, right?

51:20 Right. Yeah.

51:21 But then in the like let's say in just that day-to-day of building a current application, you need those skills when you're doing those risk assessments I talked about before, right? Like I constantly have to ask myself, can I trust this code? How much review should I do? You have to think about do I have a safety net? What can go wrong? What is the business impact? How critical is this piece of code? Is this complex? Is this not complex, right? Those are also very traditional developer skills, right? So you're better at like you're less reckless, let's say, maybe, right? And but at the same time like as as more experienced developers, we still have to build up that experience with AI to understand when can I trust it, when can I not trust it, and so on, right?

52:02 And I mean, my hope is that the people coming into the profession now will learn just the way we did. They will make mistakes and learn from them, right? Um and I just hope that the period in which like maybe everybody's a bit more reckless and like the you know, we just increase the throughput of those mistakes is going to be as short as possible and then we all develop a lot of critical thinking about this and it's going to be better, right? But I I mean, when I was an intern even before I had gone to university, I was an intern at a startup and I deleted the production database,

52:39 It happens.

52:39 I mean, it's like ultimately their fault for letting somebody who was just like a beginner There was just one environment at the time. There weren't even multiple ones, right? But I learned from that. I learned from that that I cannot just run a random script that might override all the tables, right?

52:55 you I bet the first thing you said was I bet the first thing you said wasn't why did you let me do that? It was oh crap, I've deleted the production database. How do I get this back? And then you you learn. Oh, okay.

53:06 Yeah, exactly. Yeah. The first thing The first thing that happened is I had the reddest face I ever had in my life. No, but that's an extreme case, of course, but that's how I learned, right? Like I've done things wrong or I've seen other people make mistakes in the past and then go like and then that kind of trained my intuition, my reflexes, my like oh, maybe you should look over there, look over there, right? So I hope that that will still happen and that still in teams will have that environment where we can help others not make those mistakes. That's my like positive view of how we will still learn this, right? But

53:40 Yeah. Yeah.

53:41 Let's see. In a way kind of like a lot of these things that you were saying about what we as developers will need to still do, remaining consistent, is great because those are the hard things. The things in which will change is how we go about doing that. And instead of us writing the code to to achieve those things or building the infrastructure or the the architecture to do those things, a lot of those things can be done similarly with AI. I think, you know, part of me kind of like thinks, well, will we get to those Will we get to those pain points and problems quicker?

54:14 Um you know, if I was a If I was a junior developer and I just allow AI to do whatever, I'll still hit those problems. I'll still learn about those kind of things, and I'll maybe even do it quicker.

54:25 But it might It might actually bring the pain forward, yeah.

54:27 Yeah. I I I just I just won't be I can blame AI versus me actually typing that command to do whatever in the production database. But but it's Yeah, it's It'll be interesting to see how that how that appears. But great to I think great great advice and great forethought in kind of like seeing actually the harder things that we need to do as developers today, irrespective of the implementations, are still the hardest things that we need that we will need to do in future. And I think those I think I agree I think those those those same needs will exist and and and be there for us for for many years to come, certainly.

55:02 Um Brigitte, this has been This has been very very enlightening, and it's good I think it's really good to talk about what is a real problem for so many, particularly in, you know, the larger enterprises, larger larger uh um organizations as well, that have to deal with legacy legacy code on a day-to-day basis. And and it's pretty a wonderful to kind of like hear, particularly in the the way you were describing how to how to get that information and really give AI as much as information as you possibly can, using AI as a as a tool as well to peer peer program and peer learn um from that. Wonderful, wonderful to hear to hear your your tactics and how you how you go about that. So, really appreciate having you on to the to the episode this this week, Brigitta.

55:50 Yeah, it was fun. Thanks for having me, Simon.

55:52 Absolutely no problem, anytime. And thanks very much for all listening. Looking forward to another topic next week, so tune in then. See you soon.