Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit

5,543 words · about 22 min to read

0:04 Today I wanted to have a really pragmatic and downto-earth conversation about AI, what is actually happening in our organizations, what you can expect to happen um and how agents are changing the game. And I thought in order to have this really pragmatic down-to-earth conversation, I wanted to take us to space. I do see a lot of parallels between the age of exploration and the space race and the age of AI. So last week I was talking with a CTO co-founder of a small startup. I was also talking with a principal engineering lead at a very big bank, highly regulated and we sat for a solid 15 minutes talking about all the cool stuff that we were building, how it's brought back joy, the joy of coding, and we just had all of these ideas and it seems like we couldn't build fast enough. There is so much to learn and so much to build. And it reminds me of this really lovely quote from Carl Sean that I love that somewhere something incredible is waiting to be known. And I think that this quote really captures what a lot of

1:06 us are feeling about AI and about the experimentation and just the possibility that is out there. This is the same feeling that we had with the age of space exploration, going to the moon, going to Mars. But it didn't come without skepticism. Why spend all of this money experimenting and going to the moon when we had lots of problems to solve here on Earth? We had a lot of wonder, but we also had a lot of skepticism because space exploration wasn't just about science.

1:36 It was also global. It was economic. It was political. Space wasn't a silver bullet to solve all of the problems that we had with humanity. But we also can't deny that when a man landed on the moon that it was a very pivotal defining moment for all of humankind and had a sense of wonder and had the world in awe. It was about redefining what was possible. And similarly, we have a lot of wonder and a lot of optimism and a lot of promise about AI. We can talk about productivity boosts and all of the hype around, you know, 100% productivity boost and all of our code being written by AI. We have the promise of perhaps the first singleperson billion-dollar startup with a person with an idea and an army of agents. There's a lot of optimism out there. Similarly, there's a lot of skepticism. There's a lot of skepticism in the corporate world about the real economic impact of AI given how expensive it is, given the environmental impact. There's also a lot of skepticism in a lot of different studies about the real productivity impact. In certain

2:38 circumstances, it can be really um it can really accelerate and in other circumstances it can actually slow us down and get in our way. It's hard to know what's real. But as technology changes, it is really good and fine to have that sense of wonder of exploring the universe while also realizing that we have problems here on Earth to solve. We have to learn how to balance that sense of wonder and curiosity with the acknowledgment that we are living in reality and we need to keep our our feet firmly planted on this earth. We need to understand how these experiments are actually going to apply to everyday companies. How are we actually going to improve the world around us? We need to keep the sense of wonder while also balancing it with pragmatism and beating the hype by looking at data. And so that's what I want to do right now. I'm going to share some brand new AI industry benchmarks

AI coding assistant adoption reaches ninety percent

3:28 with you. This is new data that no one has ever seen ever before in the world. Okay, it's coming right now to you. Um, I just pulled these down. I guess the the static team has seen them because they they saw the preview of my slides, but aside from them, no one has ever seen it. Um, this is though not really surprising because a lot of these numbers have not changed very much from the last quarter. So, what we're looking at here is a sample of 121,000 developers at over 450 companies. This data was pulled from November through February 1st, 2026. I I really just did this. Um, we're sitting around 92.6 of developers are using an AI coding assistant at least once a month to get their work done. And about 75% of developers are using an AI coding assistant at least once a week. When I say a AI coding assistant, most developers define that as cursor, Codex, Copi, you know, Claude, not ChatgPT necessarily. Um, but it is a bit open-ended, so keep that in mind.

4:24 When it comes to time savings, time savings is not the only measure of productivity impact, but it is an important signal. It's a good leading indicator. We're sitting around 4.08 uh self-reported hours saved due to AI tool usage per week per developer. This is not all too different from the number that came in Q2 of 2025. And then the number for Q4 of 2025 was about 3 uh six or seven. So this is kind of hovering around the 4 hour mark. And there's been a few articles, for example, from Google in the last year citing about a 10% productivity increase. And if we look at it in terms of time savings, we're kind

1 Highlight · Nearly twenty-seven percent of production code is AI-authored

5:03 of hovering around that 10% mark. It hasn't changed dramatically um over the last few quarters. What is changing and what is moving up very quickly is the amount of code getting merged upstream or in a customer-f facing environment that was written by AI that was merged without significant human intervention. We call that AI authored code. And in a sample of around 42,600 developers from that same time frame, uh, November 1st to February 1st, 2026, we're at about 26.9% industrywide for all of these developers. That's how much code is hitting production that was AI authored.

5:45 This is moving up from 22% in the last quarter, which is actually a pretty significant change quarter over quarter. And we can see that daily users of AI have crested over that 30% mark. So almost a third of their code is being written by AI that is actually being

AI cuts developer onboarding time in half

5:59 merged, passing through code review and getting into a customer-f facing environment. One of my favorite use cases for applying AI is to onboarding. And I had a bit of a hunch that AI was going to be a great tool for onboarding, helping connect people with information earlier and sooner. And I have all of this data and I thought, let me look at this quarter over quarter. And in fact, if we look at Q1 of 2024 all the way over here on the left side and fast forward to Q4 of 2025, we have about a half uh we have a half reduction in onboarding time.

6:34 This is looking at the time to 10th PR. So by the time a developer hits their 10th PR, that's a pretty important onboarding milestone that the industry has mostly aligned on in terms of onboarding and that has been cut in half now. And when we correlate that with the uptick of AI usage, it makes a really pretty graph. Uh AI is fantastic for onboarding. And this is not just brand new hires to your company. We've also seen plenty of evidence that this is for engineers who are moving projects or even non-engineers coming onboarding into projects. What's really important about this number is that there was a separate study done um by Brian Hulcat at Microsoft. He's the co-author of the space framework of developer productivity and they found in Microsoft's context that the time to 10th PR actually that performance sticks with an engineer for their first two years of tenure. So if you onboard faster that productivity gain isn't just onboarding it actually sticks with them for at least two years after they have started at the company. So this is a very important and significant uh trend

7:36 that we're seeing here with using AI to connect developers, reduce cognitive load, and get them onboarded more quickly into their code bases. One thing that's really important for me to call out, although I have just shared with you industry benchmarks, averages are just math. And as the polls move further away from each other, the average stays the same. Average does not mean typical. It does not mean what is going to happen to you. And it doesn't mean what a common experience is. One thing that is absolutely true, one thing that is common is that there is no typical experience with AI.

8:10 There is no typical experience with AI. It is it is extremely different in every single company because every company has their own problems and their own culture. This uneven impact can take us back to space for just a minute. So we can go back to the origins of the universe. We had the big bang and there was this massive release of energy and as this energy released the time the the space and time in between objects grows bigger right things are moving apart and for a lot of us the emergence of AI and AI coming into our organizations and in the industry has felt a lot like this big bang we've had this explosive

AI acts as an organizational multiplier

8:48 release of energy in the center of our world and things keep moving apart organizational performance is multi-dimensional and these organizations are just going off into different extremes teams based on what they were doing before. AI is an accelerator. It's a multiplier and it is moving organizations off in different

2 Highlight · AI amplifies existing organizational quality extremes

9:05 directions. The best example I can share with you of this is quality. Okay, so in this case, this is not every organization, but some organizations are facing twice as many customer-f facing incidents and this is from a sample of over 67,000 developers from Q1. So that same time frame of uh November to February. So just looking in that time frame organizations are experiencing twice as many customerf facing incidents at the same time at the same time companies are also experiencing 50% fewer incidents. So some companies have used AI they have a really healthy system it has amplified that system they are seeing fewer incidents they're moving faster they are accelerating with higher quality higher code maintainability higher change confidence. On the other side though, blasting off into the other part of the universe, we have organizations who were dysfunctional already. No, they're more dysfunctional. They're dysfunctional and

10:02 dysfunctional faster. Okay. Similarly to this uneven impact, organizations are seeing really uneven results like economically from using AI. there are a lot of steep drop off drop offs when it comes to using AI in a pilot context to production and then actually trying to tie it to profit. This is from uh an MIT study that was published in July of 2025 called the Gen AI divide. And what the study concluded they did a survey of 152 organizations was that right now where we are in the industry is that we have really high adoption, right? That 92.6 number. Um DORA also does its own research. We're hovering around that 90% adoption number. high adoption but actually low transformation because as it turns out transformation is really uncomfortable and organizations that were ready to give up on the cloud transformation on the agile transformation are also giving up on their AI transformations. It is really really difficult to look at your

11:04 whole organization and look at the problems and think we got to change something about this and that is what organizations need to do in order to actually see change to their bottom line. All of this to say back to my previous point we have 92.6 adop uh percent adoption among developers in our industry but adoption doesn't mean impact. Using the tool doesn't mean that it's going to actually advance your organization or do anything. It is an organizational problem that needs organizational change management. But that's not really what we were promised with all of the hype was like, hey, experiment with AI and then something happens and then we profit.

11:42 What happens though is that these tools were primarily deployed into individual coding tasks. And what this MIT study found in this high adoption low transformation is that when we apply it only to the surface area of a developer sitting at their desk there is a very very low ceiling of productivity gain. This is an organizational problem. If we want organizational results we have to think about it on an organizational level not on a coding task level.

12:10 Fortunately, our universe is expanding right now and that expanding is coming through the use of agents in agentic workflows. Our universe is getting bigger and so are all of the promises and all of the hype, but so is the possibility. So, let's go back to the moon landing, right? Like the ultimate hype was that we're all going to be living on the moon by now in flying cars like jetson style.

12:37 Um, similarly here we have a little bit of like crazy ideas. Um, Gas Town, if any of you have used it, um, there there's just like there's so much crazy stuff to do right now. Um, Gas Town is infinitely interesting to me. There are so many interesting things. Um, disclaimer, don't use Gas Town. It is unhinged. Um, we've got OpenClaw, Maltbot, Clawbot, whatever it's called. We've got Ralph loops. We've got all the stuff, right? There is so much experimentation and so much fun. It's just really fun to build. Um, but me building my nail polish matching like color scheme app while I'm sitting at the nail salon is not the same as a multinational bank being able to change their revenue because of AI. Those are really different things. Um, and I was at this retreat with Martin Fowler and Kent who are I think back there. Hello.

13:26 We'll talk about that a bit more later. We spent a lot of time trying to connect AI and the use of AI to bottom line, to profit, to P&L. And interestingly, kind of where we landed at the end was this question of like what is the value of innovation? Was it still valuable to go to the moon even though I'm not really located on the moon right now? And I would argue that yes, it is valuable to innovate. And that can get into some murky area because this is a business, right? This isn't just society and and doing things for the good of humankind.

13:56 we have to do them in an economic context and that can get a little bit tricky. So when we think about this quote something uh somewhere something incredible is waiting to be known. There is a sense of wonder and AI and space are both the age of exploration and it is so exciting. But the point of going to the moon wasn't that we all need to live on the moon. In fact, the point of going to the moon and the point of exploring and doing all this crazy stuff was to improve life on Earth. It was to use the space exploration and all of this wonder to apply it to the systems level problems that we had back on Earth. Not everyone wants to live on the moon. Um, but we have sunglasses, we have space blankets, we have barcodes, we have quartz watches. We have so much technology and so many improvements back on Earth because of this crazy age of exploration where we all went to space.

14:49 Even though we're not living on the moon, we've still used the lessons and applied it to our systems back here on Earth. And so thinking about agentic workflows, agents expand the possibilities of what we can build, how we can build it, and who we can build it for. Not everyone goes to the moon, and it's okay not to

Agentic workflows drive higher productivity gains

15:06 go to the moon. Not everyone is going to be building crazy stuff with Gas Town every day in in your enterprise context. And that's also okay because the experimentation helps push the boundary of what's possible and helps us think about solving problems in new ways. So, let's talk a little bit about how agents are being used in the industry right now. Again, this is new data that I'm sharing for the first time here. Um, agentic use is on the rise. There's not a lot of companies, honestly, that are so far ahead of the curve that they're already uh instrumenting their agentic use cases with really good telemetry.

15:38 This sample is a little bit smaller. It's around 3,000 developers at six companies. Keep in mind, these companies are ahead of the curve. They're already instrumenting their agentic workflows with telemetry. Um, we have about 80% of developers using these agentic workflows at least once a week with over 50% using agentic workflows every single day to get their work done. We talked about codecs I think in the previous panel. So on February 2nd, the Codeex desktop app was released and since then there's been a million over a million downloads by now. I got this data yesterday. I'm sure it's quite different by now. There's been a 60% growth in users just in the last week.

16:17 Um they also launched uh GPT 5.3 codecs uh last Thursday. They're processing trillions of tokens per week. Internally at OpenAI, 95% of developers are using codecs to ship stuff. And of the developers who are using codecs versus other AI tools, the developers who use codecs are shipping about 60% more PRs per week, which is very interesting. a data point, not the only data point, but it just speaks to the very high ceiling, the high possibility, the sense of wonder that we have with building all of the stuff with cool new tools like Agentic Workflows.

16:55 I want to bring it back to a nonAI startup though. Um, I want to highlight Haven Headache and Migraine Center. So, this is a company that's based here in San Francisco, actually just a few blocks away. Haven set out to answer the question, can we solve headaches with Zoom? And it turns out you can. Um, so if you're a headache sufferer, this might be useful for you to to learn about. In healthcare, it's really really uh crucial for Haven and their development team to distinguish between using agents for durable code or disposable code. One of the things that they're doing that's very cool since they are a disruptor, they are a small startup is using um using agentic workflows to rapidly prototype new custom uh like new patient workflows. So they're working on a patient portal building with Ralph loops taking uh linear and Figma artifacts changing it into a PRD uh you know spitting that out in JSON and then just having Ralph loops run. What they're getting though isn't garbage disposable AI slop. What they're getting is really high quality prototypes with really excellent documentation, excellent tests, much higher quality at a way faster rate than

17:58 they would have um if they would have built it by hand the oldfashioned way. The other thing that they're doing that I really admire is improving the standard of care for their patients by training a HIPACO compliant model on hundreds of thousands of symptom logs. So Haven meets you where you're at. You get a text message, you can log your symptoms and then they can um instrument your care, figure out what needs to happen from there. So they're training a hypocmplant model on hundreds of thousands of these messages so that those messages can be routed to, you know, medication refill or schedule follow-up appointment just meets you where you are. And the result of this is that they have 3x the industry average in customer satisfaction for a healthcare tool like this, but also real real meaningful clinical outcomes. So their patients have fewer headache days per month and also the severity of their headaches is much uh much less severe.

18:46 So good job Haven. In the enterprise, there are lots of examples of big enterprise companies experimenting with agent workflows. So there's an enterprise manufacturing company that's using it for solely internal developer purposes. They used C-pilot and Claude to build out a dev portal to accelerate uh developer onboarding. At Cisco, there's 18,000 engineers using codecs daily. They're using the uh codecs for complex migrations and also code review leading to a 50% reduction in the amount of time it takes to do code review.

19:18 There's a really cool paper as well by JP Morgan Chase's multi-agent framework for annotation, MAFA. If you Google that, you can find the source paper. It's really fascinating. Um, what they're doing is building out like a whole business of agents. So like a true multi- aent workflow similar to Gastown where each agent has a special job to do. What they're also doing in this model is introducing consensus among the agents. So they're taking all of these interactions and then they're annotating them. this was you know the intent what was it an FAQ what what were all these interactions the agents are annotating them and then there's another set of agents who are responsible for reranking and calibrating and validating the output and then of course we have to introduce consensus algorithms to the party because now we have multiple agents with maybe multiple different opinions about things um this is really fascinating and I believe consensus among agents is going to be a huge problem to solve in 2026 I spoke about this retreat. I was lucky

AI does not fix systemic organizational problems

20:16 enough to be invited by Martin Fowler and thoughtworks to the future of software development retreat celebrating the 25th anniversary of the agile manifesto of Gerge joined me. A few other folks who are here also joined me. We spent a day and a half up in the mountains talking about agents. That's really all we talked about about using agents responsibly, ethically, sustainably, how we can use them for organizations. And our conclusion even though there was so much interesting stuff, Steve Yaggi was there, we were working on Gas Town things, like there was a lot of experimentation happening, but the conclusion that we came to was that AI does not solve organizational systems problems. It only can do that when you apply AI to the system problem, which means you need to acknowledge that the system problem exists in the first place. AI is not a magic silver bullet.

21:02 Even though things like Gastown exist, even though there is so much sense of curiosity and wonder in the universe, um we kind of had a sort of off-the cuff conversation. Uh Kent Beck, Steve and I were just catching up um outside uh of one of the sessions in between conversations. And here's sort of where we summarized our thoughts. Organizations are constrained by human and systems level problems. We remain skeptical of the promise of any technology to improve organizational performance without first addressing those human and systems level constraints. We remain skeptical and we also remain human because the risk is if we don't address the systems level problems, we will just take them to space with us. We will just take them to space with us. We're not actually going to solve the human factors that are the driving force behind all of the constraints that organizations have right now. We can apply AI to those problems, but we still need to solve them. We can't just go to the moon and expect that pollution and garbage and traffic aren't going to be a problem anymore.

22:05 And so the question is not how to colonize Mars, but the question is how to get real organizational impact with agents and AI. At this retreat, we also talked a lot about common factors that we see. What do we see organizations doing? What are the common patterns that is kind of like the secret to to winning? What do they have in common? The first one is that organizations who win with AI and are winning with AI have goals and they measure their progress against those goals. Spray and prey does not work. Spray and prey, what I mean by that is just giving all of your developers licenses and hoping for the best. It does not work. I can say that very very clearly. I have a lot of evidence that does not work. If you can point AI innovation and that experimentation to a problem, have a concrete goal and then measure if you're reaching that goal, that is what winning organizations are doing right now.

22:57 Because as Spock has told us, insufficient facts always invite danger. We need to measure things. We need to have data. And I know this is something that's really difficult for a lot of organizations right now because developer productivity and engineering excellence are also really hard problems. And this is happening all at the intersection. So I have something that can help you if that is in uh a problem that you're facing in your organization. This is the AI measurement framework. This is a framework that I co-authored with Abby notto who's the CEO of DX.

23:26 This complements our core four framework which some of you might have heard otherwise it's in the impact column here. What we're looking to do is track not just usage and adoption and utilization of AI but then also translate that into real organizational impact. Is this changing your speed, your developer experience, your quality, your innovation ratio? Those are really important questions to connect the adoption to impact. Finally, we have to look at the cost. Are we getting a good deal?

23:54 Maybe some of us are for now. Um, and we need to understand as the cost of these tools keeps going up and up, is the investment the right one?

Developer experience is critical for agent success

3 Highlight · Rebranding developer experience to secure agent funding

24:04 The second thing that is helping organizations win is that developer experience matters now more than ever. Here is a piece of very unconventional advice that I'll give you is just anything that you were going to talk about with your leadership team about developer experience. Just call it agent experience and you'll get money for it. It's funny but it works. Um, it works because developer experience, feedback loops, um, you know, clearly defined services, great documentation, fast CI, these are all things that we have been screaming about for decades, literally, and we've been begging for pennies from our organizations to please let us

24:43 invest, please let us invest in developer experience. And we've been told no over and over again. Come to find out, in fact, these are the things that make AI really successful. We need to have really solid testing and quality practices. We need to have great documentation. These are critical for agentic workflows. It is disheartening that we didn't want to spend the money when it came to human engineers, but when it comes to robot engineers, we're okay with it. But that is the world that we live in and let's capitalize on our opportunity. So Devex matters more than ever. In fact, when we look at the data right now, remember we're hovering around that 4hour mark for time savings.

25:18 when we look at all of the other factors of developer experience like AI time savings is not going to make up for bad meeting like bad meeting culture and lots of interruptions and um you know developers who are constantly being pulled out of their work unplanned work interruptions outages those kinds of things AI will not make up for that we can use AI to help solve that problem but AI in and of itself is not going to make up for it then when we look kind of in the bottom half build and test wait time toil and dev environment we put all that together We realize that just the time savings from coding task speed up isn't going to get us very far. But what will get us far is when we can take AI and point it at those problems. Can we use AI to help reduce meeting frequency?

26:01 Can we use AI to improve CI weight time? Can we use AI to reduce dev environment toil? That is what winning organizations are doing right now. They are putting DevX at the center of their universe and seeing AI as a tool to fix systems level problems. They're doing it also on an organizational level. If you want organizational outcomes like revenue, P&L, time to market, you have to think about AI on an as an organizational problem, not as an individual problem that your developer needs to solve at their desk. It has to apply to workflows that span entire value streams.

26:39 back to that MIT study when we looked at the barriers to organizational adoption uh or the organizational barriers to AI adoption, they weren't technical. This wasn't about the models necessarily. It wasn't even about the tools that wrap the models. It was about things like change management or lack of executive sponsorship when you have an executive team saying go with AI, but they themselves have never cracked their laptop open and fired up windsurf or cloud code or codeex. um poor user experience, just very unclear expectations about AI. Those are the things that get in the way.

27:13 If this sounds familiar to you and perhaps your organization could do a better job, there's two things that I want to point you to. Um the first one is the Dora AI capabilities model. These are models that kind of communicate and help you get ready for AI. So think about this as an AI readiness model or an AI capabilities model. This has in a crazy amount of data from organizations that Dora studies. They do a lot more than just the four key door metrics. Um, finding correlations between practices that organizations have and good outcomes with AI. So, if you use AI and have a good clear and communicated AI stance, you are going to do better organizationally than a company that does not have one. You can find this at dora.dev. It's the Dora AI capabilities model. There was just a new paper that came out last month. Last month, uh, Nathan is here who leads Dora over at Google Cloud. If you want to talk to him about this, he's probably the the guy.

28:03 Um the other one is the thoughtworks forest framework. This is similar to the AI capabilities model kind of a different flavor on it. Um if you go to thoughtworks.com and look in their white papers you can read through this but these are both really solid wellressearched industrybacked AI readiness models to help convince your leadership team if you need that um or just help you do an internal audit of are we doing the right things to make ourselves ready to you know reap the benefits of all this experimentation.

28:31 The last thing is that organizations who are doing really well with AI right now are experimenting by solving real customer problems. Again, space exploration and going to Mars is great, but that is not sustainable for your whole entire organization to be experimenting with going to Mars. It just costs too much money. It distracts too much from the core business problem. It does not serve your customers. So, keep experimentation going. Other experimentation can be really laser focused on real customer problems that you have. And that is how you're going to see the organizational results.

29:03 Somewhere something incredible is waiting to be known. There is so much possibility of how we can build what we can build, who we can build it for right now with AI and agents are just accelerating this. They are expanding our universe. We are definitely in an age of exploration. The urge uh the thing I want to urge all of you to take with you into the rest of the sessions today is to find that balance between a sense of wonder and a sense of awe and aiming for Mars and aiming for your moon colony but also understanding that we need to solve the problems here on Earth and we have to live in this reality. So please stay grounded, stay skeptical, stay human, most of all stay pragmat stay pragmatic.

29:43 Thank you all.