Stop Maintaining Your Code. Start Replacing It

12,109 words · about 48 min to read · YouTube’s automatic captions

Code is a liability while the system is the asset

0:00 the code that we have is a liability and the system is the asset that we're building. Phoenix architecture takes the same exact idea, but it says ideally what you should have is some sort of a specification for the system and then the code you should think about is just an implementation detail or a build artifact. And the reason it's called Phoenix architecture is like the metaphor of the Phoenix that's destroyed and rebuilt and you know, rises from the ashes. But the entire architecture is about constantly replacing every piece of the system, but not replacing the system as a whole. The goal here is to be able to deploy code in production that was generated by AI of some sort that humans never reviewed. We know that's what's going to happen because we know human behavior and it's way too easy to do that now. So there people are always going to do the easy thing. So let's figure out systems that make the easy thing okay to do. You have a statement that said evaluations are the real code base. How do you think about durability?

0:51 What is it that as I build a key or as I build up anything persists across? That's a good question.

1:09 Back in November, we hosted the first ever in-person AI Native DevCon in New York. This June 1st and 2nd, we're bringing it to London. It's two days built for AI Native developers and engineering teams. One day full of hands-on workshops and one day full of practical talks on agent skills, contact engineering, agent orchestration, and enablement platforms and how teams are actually shipping AI in production.

1:34 Join us at the brewery in London near the Barbican for all of that plus networking, parties, giveaways, and a room full of people building the future of AI Native development. You can also join us from anywhere in the world via the live stream. As you're listening to this podcast, you get 30% off your ticket with code pod30. Just head to ainativedevcon.io and we'll see you in London.

2:01 Hello everyone. Thanks for tuning back into the AI Native Dev. Uh today we are going to go uh I don't know if it's far into the future, but definitely looking into the future or the concepts of it uh with Chad Fowler who is a VC today, but don't hold it against him. You know, he's a today at Blue Yard Capital uh but uh he really has this sort of rich technical background on it. You know, some of you might know him as as the the person leading Ruby Central uh at the time when Ruby was all the rage it feels like uh feels like a while ago. Uh

2:32 but then innovative, you know, I think you were CTO at uh Wunderkind. Not sure how to even call that, which is sold to Microsoft uh and you were CTO for a variety of of departments and sort of areas for Microsoft then uh and generally I think a fairly well-known uh thought leader in the space. Um and specifically, you know, for for this podcast, what pulled you in is that you've been writing some really interesting blog posts and like little storylines and perspectives on the future of software development and where that is headed in the Phoenix architecture blog, which is inconveniently posted on aicoding.leaflet.pub.

3:09 Every time I have to like find that. Uh but it's really really good uh blog, really really good ideas, which we will talk about today. Uh so Chad, thanks for coming onto the show here and sharing some perspectives. Yeah, thanks for having me. Um it's a totally exciting time to be talking about software development. So I want to do as much of it as as I can. Yeah, yeah. I'm fully fully agree with it. Like and suddenly interesting, cerebral, fast-moving uh a lot of this excitement and never boring. Super interesting.

3:40 Yeah. So I guess before we So we're going to talk a lot about the Phoenix architecture and I want to maybe like dedicate some specific focus on on what you mean by this. Uh but maybe for starters, I know you there was a bit of a like a origin for all of these things thinking and you were you were sort of thinking about code is disposable and and these types of uh views for a while now. So can I for starters just have you tell us a little

Euthanizing legacy systems inspired disposable code

4:03 bit the origin story of How did you come to this line of thinking and um how how that came about?

4:09 Yeah, I guess you know, it started with um in the the early 2000s, somehow I ended up in this line of consulting and full-time jobs where my my job was to euthanize an existing software system. So come find some system that's like doing good work, but can't keep up with the demands like the technical demands or the the business demands um uh posed to it.

4:36 And so I'd come along and kill the systems and I started getting really tired of that because I was thinking about like what about my own systems that I create? Like think about how much time and effort and emotion I put into my work. How can we build systems that can outlive us? Like why why are we building all these systems that are dying within like five years? It's insane. And so I started thinking a lot about legacy and what that word means and also just like biological biological metaphors for software development because I thought, well, I don't take very great care of myself as a human, but I'm still here.

5:06 How is that possible? You know, we take we spend all this money on software systems and they still sort of get corrupted and die you know, fail under their own weight. And so uh a couple of sort of key insights came to me. One of which I didn't really act on much over the years. The first one is just about infrastructure and I started thinking about, you know, how we in the like '90s, we'd be excited about uptime for a given server. You know, we install these massive big Sun servers or whatever and I remember one that had been up for like two years without being rebooted. Terrifying, but back then we

1 Highlight · Constant change proves system understanding

5:38 were we were proud of it. Yeah. And I realized that like if you want to be able to change things, you need to do it all the time. And so, you know, this sort of behavior, whether it be an infrastructure or software, would lead to situations where you just couldn't progress the system because you're not practicing change. So the first insight that I had in this sort of legacy thing is let's just kill things over and over and over so that we can be used to killing. And I started with infrastructure.

6:04 And now it's just like old hat, the idea of immutable infrastructure, but like never ever modify a server, always replace it. Which of course you couldn't do with physical hardware, but you can do with containers and you know, EC2 instances is what we were doing back then. Right. But not not just even for the purpose of uh of kind of learning and the agility, but also to to know that you can do it, to really truly know that you're able to recover from a server failure is to do it all the time intentionally.

6:32 Right. Right. And why is that? Why like why is that the rule? It's because if you cannot change the system, it means you don't actually understand the system. Mhm. So, you know, if you're changing it constantly, you understand what the requirements are. Just you have to, right? So the same should be true in software, but it was much more expensive, you know, 15 years ago or so to just throw away code all the time. But I was thinking a lot about like what would it mean to have uh disposable software, immutable code.

7:01 So as soon as you write like one unit of work, whatever that is, you never modify it again, you just throw it away uh because the unit of code itself is small enough uh and like bounded enough that you can understand what it's supposed to do trivially and therefore you can safely replace it. So this is the sort of backstory that when when you pit it against where we are in terms of like LLM generated code, it starts to click and you say, oh okay. Uh and we can go into that in more detail later, but that's sort of the the origin story of this thinking. So I I I

7:33 love that and I love the the analogy to immutable infrastructure because I think so many DevOps analogies are helpful, at least to me or to people that have kind of been through the DevOps change to to be able to to kind of apply those to AI and try and forecast where it's headed. You know, the fact that servers are statistical creatures, right? And they you know, they don't always behave the way we want them and so we we need to deal with a world in which it's not deterministic and to and to move those.

8:00 So I love the software to or rather immutable infrastructure concept and applying that to immutable software uh and think, okay, what what have we learned about immutable infrastructure that we need to now kind of represent here? Right. Yeah, there's also like as you were talking, I was thinking there's also this thing back in the day like when I talked about how proud I was of my two-year uptime on a server, I also knew that server's name and I knew its IP address and I like set it up myself physically. I cared about it. You know what I mean? Yeah. So I was like attached to this thing.

8:35 And when you get to immutable infrastructure, you can never be attached to like one running instance of your your software stack. I I think the same sort of lesson applies to code, too. Yeah. We have all developed,

Treating code as cattle rather than precious pets

8:47 especially developers, this almost fetish-like uh worship of our code, forgetting that the code is not actually the thing we're trying to build. It's just the implementation of the thing we're trying to build. Yeah. Yeah, no, I I love that. That's you know, in in DevOps there's a whole ethos of cattle versus pets, right? You have the the ones that you're nurturing as pets, uh which is what you're describing and it's it's um it is a really interesting thought to say how do you think of code as cattle uh and you know, you you just have plenty of it and you keep swapping it, which was just not feasible before. Just like immutable infrastructure was not really feasible or practical, I guess, except for very unique circumstances uh before the cloud enabled it. Yeah. So I mean, how would you how would you make it feasible? Um even back then, you know, as I said I was like starting to think about this, the first rule when when I was working on Wunderlist was you can do code code in any language you want as long as it's no more than this big.

9:45 And so, you know, I'd literally say this. So it just means

9:47 trying for a small amount. Just between like

9:49 Yeah. Roughly roughly like it fits on a page of your editor. So if you have really good eyes, you can have a you know, You know, have more code cuz the font can be smaller. But we actually did this and we did this sort of like radical microservices architecture. And there were at least a couple of cases where code was disposable because of this. Um we had like uh consistent calling conventions and sort of just, you know, we had like a way of dealing with mutations offline and we had a way of dealing with REST and CRUD sort of stuff through the the system. And it was all consistent. Um And all the code was tiny. Which meant we were able to replace things. So the two versions of this, one I had written a service in Haskell. Uh and I was one of only two people who would tolerate even looking at Haskell. I loved it.

10:36 Write-only code.

10:39 Yeah, well, yeah. In fact, in this case that was the problem. So the we had to change this code so infrequently that it became the build system became stale underneath it. And after we were acquired, we could not recompile the code anymore. It still worked. It was great. But the like toolchain had broken underneath us cuz we didn't have that rigor of constantly changing it. So there was a problem with it at one point and someone just said, well, we're just going to rewrite this in Go or something. So they rewrote it rewrote it in like an afternoon and deployed it. It was fine, you know.

11:11 Yeah. And the other thing was we we had deployed a major release of Wunderlist, which was like the, you know, sort of crown achievement of what we did technically. And when we did it, we had literally hundreds of server instances running these heavy Ruby on Rails servers. Because that's what we knew. It was like a consistent choice. But then afterward, over the course of like 3 months, we replaced something like 70% of the code base. Uh with Clojure, Go, Rust, etc. And made it, like, I don't know. I think it cost like 25% of what it did at launch to to run because we were able to replace all these systems that were running Ruby.

11:47 Because they were all just tiny little things with consistent calling conventions and easy to plug in. That was our first attempt at doing this sort of disposable code thing. No, no, I love that. And I think some of the kind of clear modularity goes all the way back to sort of Java design partner design patterns. You know, just thinking about, you know, abstracting of factories of factories and kind of creation of those that at least from a dynamism of the software and the ability to hot load different components will come in. And I guess we still think of the code itself often times as the thing that is precious even if we modularized it within the system. And I guess, you know, this is a AI land or AI era now. And so clearly that is now changing. So maybe tell us a little bit about the the the Phoenix architecture.

12:33 I I I love in general a lot of the

Phoenix architecture codifies regenerative software

12:35 storytelling you have on the blog. But, you know, some of these visual analogies are are helpful and and fun. So you talk about Phoenix architecture, you talk about regenerative software. So what's the I mean you've you've already alluded to some of it. But tell us a bit about what's the philosophy behind that. Sure, yeah. So, you know, if you think about what I was just saying about consistent calling conventions and and generally conventions for like what the interfaces would be, you know, how you think about uh CRUD operations on data. And you know, this doesn't just apply, of course, to CRUD-based REST apps. But um these are all ways of constraining a system so that you understand how the components interact with each other so that the components could then be uh potentially implemented in any language or, you know, framework or whatever. You still have the idea of how the system works. And one thing I used to always say to the team at Wunderlist is the code that we have is a liability and the system is the asset that we're building.

13:35 Uh and the Phoenix architecture takes this and sort of codifies it. And I also have a not yet released, but eventually open source project called Phoenix, which is like the actual implementation of all this stuff I'm talking about. Um It codifies it and it says uh any software in the system you should think about like it's a build artifact. So, you know, imagine you have a makefile or ant file or whatever the dependency-oriented build system thing as you know.

14:03 There's always some um sort of top-level asset that you start with. Usually it's code in a high-level language. And you run a thing and it know it understands the path and it can compile all the way through. It can invalidate any object code that's changed. You know, if you're talking about like a truly compiled system or a VM type system like JVM. Uh and it can replace just the paths that need to be changed. But the top-level like asset that you're

2 Highlight · Treating code as a build artifact

14:28 building in that case is the Java code or the C code or the Rust code. The Phoenix architecture takes the same exact idea. But it says ideally what you should have is some sort of a specification for the system. And then the code you should think about as just an implementation detail or a build artifact. And the reason it's called Phoenix architecture is that, you know, you these like the metaphor of the Phoenix that's destroyed and rebuilt and, you know, rises from the ashes.

14:55 But the entire architecture is about constantly replacing every piece of the system, but not replacing the system as a whole. Because you need that sort of consistent shape and calling convention for for interacting with the system you know, generatively. And so um yeah, that's sort of the key thing of if if we want to be able to change systems safely and rapidly and we want that to be how we interact with our systems.

15:23 They're just like very change um friendly. Then like I take a lesson actually from extreme programming, Kent Beck, all those people in the early Agile days. If something's hard, just do it all all the time. So if testing is hard, then do it all the time. If code review is hard or expensive, then just do it all the time by doing pair programming. The idea here is if change is hard, then just build it into the system so you have no choice but to constantly change. And you can also never actually look at the code. And and I guess I I should sort of back up and say the goal here is to be able to deploy code in production that was generated by AI of some sort that humans never reviewed.

16:03 Uh and this would have sounded more crazy 6 months ago. Now a lot of people are doing this. But at scale and like big enterprise systems or very complex older systems, of course, you can't do this yet, you know. So if that is the goal, and why would that be the goal? Because that's what's going to happen. We know that's what's going to happen. Not because like we're looking at the future and we're looking at technology trends, but because we know human behavior and it's way too easy to do that now. So there people are always going to do the easy thing. So let's figure out systems that make the easy thing okay to do.

16:34 Yeah. I think it's also the opportunity. It's not just that it will happen. It's also uh uh the the means of staying

16:40 competitive is to be able to tap into that and to take the human out of the equation. But I, you know, it So as someone has been saying, you know, code is going to become disposable a couple years ago and and, you know, kind of was dismissed as heresy not that long ago. Like sort of 2 years ago. Today doesn't feel as far-fetched. Uh but let me ask about a few challenges

Iterative specification replaces detailed upfront design

16:59 kind of within within that world. So I think the the the most blatant one is uh what is what is sufficient detail in the spec? And at what point does that sufficient detail just become a different piece of code uh that that you've written? Uh and all you've done now is like create some sort of template that's at the top. How would you define the the right level of specificity uh in the spec and and which parts you're happy to delegate to the to the implementer uh that is the agent? That's a good question and the answer changes weekly kind of, you know. Like I mean, since you've been talking about this for at least 2 years, it means probably 4 years ago you were messing around with code generation and in the frustration loop constantly. Like, oh my god, it's magic and then it's not magic anymore. Fails, you know.

17:46 Uh and I think the last time I felt that way was like September or October with the tools available. And now I don't feel that way anymore. But I know it doesn't sound like I'm answering the question, but the reason I'm saying this is there's a whole lot more you can one-shot with very little detail these days. Um however, uh like forgetting about the fact that it just keeps getting smarter and smarter, I I think the answer for me is that um I want to iterate until I see the thing change, right? So uh like I'm working on a new IRC network, for example, a reboot of IRC right now. I didn't one-shot it.

18:25 I've got all sorts of interesting technical things in it. It's called Freak, by the way. Um And when I when I worked on it, I didn't create a spec that had a great level of specificity. But I iterated until I had a spec that was enough. And I knew it was enough because I see the evidence in the software. So it's kind of like you go back to the early 2000s, we had this this thing called Naked Objects in Java where you just like generate Java objects and it creates the UI for you on the fly. And the whole idea was a new Agile way of working with soft with customers and software where you could just like pop things open and show that to them and iterate like that.

19:03 Sitting with the customer instead of like going on these long project things. Interact with the data model. Yeah. Yeah, and I think it's the same sort of thing now where you can iterate and iterate. The problem this creates though is if you want a system that can survive change over time, you can't just regenerate the entire thing constantly. Yeah. So you need to be able to iterate, but you need to have some way of being very clear about which things are changing. So yeah, like there are two issues. One, um if a user knows how to use an application and it changes every day because you've re you've changed the spec and regenerated it and, you know, re-reimagined an entire UI, that's not okay. Even if the UI is great, even if it's better tomorrow than it was today, users have to actually like understand how to use the software.

19:48 Um and the other is you just don't want things that are like core to the technical foundation of your system to change constantly because they might be wrong, right? And that's where this level of specificity matters a lot where one intuition, like so an example of this is I'm building on this IRC network. So I've got IRC server that implements the protocol and then I've got protocol layers on top and another layer and then I've got clients, you know, and client SDKs.

20:15 At the very bottom level, I want to get that right quickly and just set it in stone and probably never change it unless there's a mistake or we want to upgrade the IRC protocol, right? And so the best way to do that would be if there were some set of specifications that we iterate on that we can literally tie through some like build graph back or to the to the code that gets generated and it never has to change unless the dependency tree changes before it. So we can change the the protocol all we want in the beginning and then we sort of lock it because we never change even the like vague specs we created that happened to create a thing that was good enough. This sounds pretty scary, I know. And then you just, you know, you work in the the layer that that makes sense to do a lot of change. There's a term, I don't remember the person's name who coined it, but a term called pace layers that sort of uh codifies this idea of different speeds of change at layers of a software architecture. Yeah. I think so so so I've done a fair bit of

21:16 thinking of this and in the system framework that we've since kind of parked as we focus on Tesla and context management. And we can get a little bit into into the delta there. Uh we we had a lot of uh these elements and we we talked precisely about that in that you will create a spec and the spec can be at whatever level of resolution. You you reach software that you bless by using it and looking at it and sort of deciding that it is good enough. And then from here you have the question of stable what you want to achieve is stable regeneration. Uh so you know, you you when you regenerate it, you want stability. Uh you might have a button.

21:52 You might not care what color it is and you're happy to delegate that to the to the agent, but you don't want to change in colors every every version. And so you want some stability over there. Uh and so you want that and and the the challenge of course is figuring out what are the aspects that you want to um stabilize because uh because increasingly like those those change. Like maybe you want the button to be the same color unless it has a good reason to change because you've just changed all of the themes colors and so the button should also change uh in that process. And we actually introduced to use the word lock, you know, we we had in the system tests for uh for the different uh parts of the spec and we you could lock a test to imply it.

22:36 And so so it is very interesting um and I I I I think probably a bit of an unsolved and then interesting interaction model as well of of if you were to create these different things. Let's say your information architecture allowed that. Uh what is the what is the sort of the user interaction, you know, that that you would have with these assets? Um and maybe I'll I'll I'll throw one more which is I think there's also an element of who made the decision that is more important. If I said something, me the human in control who is paying the bills uh and whose neck is on the line, that should be, you know, more holy than if the agent has made a decision and that decision should be more easily overwritten because it was just making

Agentic systems shift focus from languages to architecture

23:19 it at the time. So there's a lot there's a lot to sort of figure out. I don't know if you're are you would you anticipate like new programming this is like a a frequent question, you know, that I get as well. It's like would you anticipate these things manifesting in in in new programming language paradigms or in or these just patterns that will eventually be applied by the agents? They will manifest in programming languages, but I think that's also a step along the way to where it goes because I I think what we're going to find is like any programming language we create is going to be trying to solve problems that LLMs can't currently solve with with plain language uh and then they'll become obsolete.

24:01 Like my first assumption when when I started doing stuff with chat GPT, you know, in early early chat GPT days was oh we should be using like Haskell or Owl and RSS, you know, like ontologies and I had all these ideas about static typing being better and blah blah blah.

3 Highlight · Tracking human intent with cryptographic hashes

24:17 It's just all doesn't matter actually. Um but I think what what we need instead is different programming um paradigms and patterns that aren't languages. So, you know, my my work right now on my implementation of this Phoenix architecture is focused on going from raw human specs to refined um they're called clauses in my system where you can like canonicalize actual intents out and then you can do hash-based tracking of these intents all the intents all the way through with cryptographic hashes which gives you that build system thing, but it also gives the ability to add provenance because I absolutely agree like capturing intents of the system uh and capturing who made them uh and like when and why are really critical both in like knowing what to regenerate and also in just like figuring out how you would remove things from a system.

25:17 That like to me that's the biggest problem because we're generating things at such speed, you're going to end up creating like absolute masses of garbage. And if the agent did it and not the person, you know, it's probably okay to delete. Uh you know, there's a lot of other factors you have to check. But like on my IRC system freak, I actually have two parallel web apps by accident. I have a really nice one that I've been iterating with and then one that the agents just decided they were going to make and it's linked some places so I have to remove it.

25:46 Um but like at least I know in this case that I made this decision. But I think what's really interesting is the combination of what we were just talking about with specs etc. and then like capturing the the human interaction with the LLM as the code is being built. And there's a bunch of work going on here of doing this. Um and to me that's the important thing is like how do we figure out how to map from

26:09 intent all the way through to uh object of code, whatever that means, implementation unit in as clear and replaceable a unit as possible. Then we have the ability to do this regenerative software thing for real.

26:24 Yeah, for real. I think super interesting and I I guess my my thesis also would be that there are actually there are sort of three layers of human condoning, you know, or like blessing. There's the explicit instruction that came from the human developer. There's the explicit review. I I actually read this and said yes, which is a bit tricky because did you really read it? I don't know, you know, like you technically it says is this okay? Should I go? And you said yes. And then there's the you know, no human interaction. The agents just decided those. Uh and uh yeah, we had a lot of those. We by the way referred to them as shadow specs. Um there's the spec and the shadow spec uh which are the decisions you never made but have been made for you.

27:03 Yeah, that's that's a good term. I like it. And I like I must admit most of the work that I do I've got mostly shadow specs with like really high-level stupid instructions these days and I go YOLO, you know. So my review my explicit review I did not read it. I scanned through it and I'm like yeah, that sounds right. Exactly. Just replaced it. Yeah. And there's again there's kind of these two levels that I don't know how to sort of name them. There's there's levels of shadow like, you know, the shadow and the dark spec, I don't know. But you know, the ones that you in theory reviewed and the decisions that were never presented to you. They were just made and you could have proactively gone and reviewed but you didn't bless.

27:36 Like asking about programming languages reminds me of a conversation I had earlier this week with a real language

Compiling to system architectures over languages

27:43 innovator. I won't say who it is cuz I didn't ask if I could, but um we were talking about specifically as I take all these stupid ideas I have and I try to make them into real code that I can create projects with. One of the first things that I ran into was you know, I keep saying like consistent calling conventions uh and ways of dealing with uh mutations in the system asynchronously. Is there some sort of message bus thing? Like there's sort of these things that happen in my head for what you need for a Phoenix type architecture to work.

28:14 And the immediate immediate thing that I run up against when I started generating this is oh I actually have to choose like what language and framework. Let's say I'm going to build a web app. Like what MVC framework is it going to be in? What programming language? What runtime? And I reach the conclusion that I should just decide this for the entire system for any code that's generated. So like we can start with web apps. I'll just decide like okay, what's the best framework for generating it for generated code for a web app in AI today?

28:45 Uh I think there are two answers to it. One is whatever the LLM knows, so probably like two years old old Next.js or something, you know, React. I I not things I like. It makes me sort of ill to say this. Yeah. But it's whatever it is that has the sort of the the sweetheart of the weights at the moment. Exactly. Yeah. Which which tends to sort of persist historical uh uh popularity. Yes. Well, now it's really going to lock in forever because that's what we're going to keep regenerating forever.

29:12 getting duplicated and it feeds itself. Yeah. But what really matters now is uh you know, like I I was all into Ruby on Rails and evangelism and all this stuff back in the day, you know, as you mentioned. I would not do that today. I would not tie myself to such a thing because what matters is you have the code has to be able to be correct. It has to be as cheap as possible to run. It has to perform as as well as it can.

4 Highlight · Compiling to system architectures, not programming languages

29:37 It does not have to be human friendly. It doesn't have to be trendy. It doesn't even have to have unit tests, [snorts] you know, you don't have to do peer review Haskell, you know, the uh it's okay. So so what I realized is I'm this Phoenix architecture like I don't think you will decide when you use it what the like framework is or the language that it generates. You shouldn't. That should also be a thing that's completely open to change.

30:03 So, what we'll be compiling to is more system architectures. A shape of the system that allows us to have pluggable, replaceable components because of the shape of the system because of the calling conventions and the consistency and probably some sort of like type system. There will be these sort of properties, but they won't be about the programming language. They'll be about how this thing runs and how the components interact. Yeah. I think um so so that's that's a good kind of tee up for for a challenge that I have in my mind. So, uh clearly I've been on this of the spec-centric

30:35 software bandwagon for for a while. Like literally I've been sort of devoting kind of my time in in Tessal uh to pursuing this vision. Uh and uh and like probably the most significant way in which I feel like my thinking has has uh I want to say evolved or at least changed. You know, maybe it's regressed uh over time uh is is the move from from more of uh thinking about a spec and some some uh data format definition in which uh you properly capture your intent and and the definition to to wanting to get the agent to just make good decisions.

31:18 Uh and I feel like when you when you work with humans and like with people on your team the the best developers that you have on the team uh you or even like you know, yourself and your future self on it. It's it's not so much that you expect them to kind of truly kind of represent these uh definitions of your software, right? Or sort of you know, the historical decisions that you've made and that need to blessed and all that. But rather that you expect them to combine intelligence and information. You know, it's combine uh you know, just like smart decision-making and kind of conclusion drawing uh with the right context, you know, using a term that I think is good for humans and agents to to uh put the emphasis in the right spot. They they should know how much you care about finance and budgets versus how much you care about um uh you know uh speed and and latency. You know, how much you care about functional

32:19 programming, you know, versus how much you care about, you know, Ruby. Uh how much you care about security versus, you know, like just sort of all of these different preferences and even your uh you know, your kind of business surrounding. So, it's sort of an understanding of your constraint and sometimes those constraints may include it has to be written in this language, right? Maybe if you are building within Microsoft and you know, someone condones it, well, this has to run on Windows.

32:44 Right. So, that's a constraint and that limits some of your options. Um So, I don't know. Like I I really love and and still expect a representation of these things to to manifest, but I wonder where is it that we even if we're thinking ahead, we're not thinking sufficiently ahead and we still like the idea of a data model that can represent robustness, right? Or some some semblance of determinism when really it's more about making good decisions.

33:18 Yeah, I mean I guess like just um a random thought from what you were talking about is I think instead of specs, we need to be thinking about building knowledge graphs that uh yeah, that sounds like a little bit probably too inflated, but you know, knowledge graphs of all of these things whether it be the the values and and and variance and intents of the the person that the beautiful thing with LLMs is they can actually get they can mine this stuff from you very easily. They can say, I feel like you're missing this, you know, which is sort of how I did it with with the tests I've been doing with this Phoenix architecture so far.

33:55 Invariants are part of the system and like metrics of success are part of the system. And you're right that there's like a you know, I don't know, deploy target or build target or whatever. Like clearly there's still a Windows app sometimes and there's still a Mac app sometimes. Um the way I'm thinking about it is there's probably just like a set of patterns that uh I'll say I create for myself with this Phoenix thing, but there's you know, everyone's going to have their own uh probably center of gravity of ways of doing this that they adhere to, but for mine it would be a set of patterns that apply to uh native macOS apps, native iOS apps for now, web apps, you know, um TUI apps for consoles or whatever, server-based things.

34:37 Uh and and even, you know, like distributed systems. So, then I have an architecture that I compile into which can include things like platform. Um you know, most of the cases for me I'll be pretty agnostic about platform and language. But then you could have different constraints that you you add to the system. Yeah, very hand-wavy wavy abstract stuff cuz we're we're not there yet, you know.

35:03 I mean, the other thing was like we're talking about apps that manifest, but maybe there's just a data substrate and apps manifest on the fly, you know, it could be visual or it could be LLMs, whatever conversation. Yeah, there's probably still an efficiency element there. So, I really like it. Like I did also like I don't want to challenge. I feel either way we need these data structures uh for it and and there's, you know, I think another lesson learned a little bit is that you can't be too far, you know, too far out in the future. You have to think a little bit about it. And so, I think the Phoenix architecture is is already fairly dramatically innovative and a bit of a of a mental switch to to embrace, you know, for anybody in the current software development world. So, even if there is even if it is a stepping stone, you know, as you as you think ahead, it is it is uh very likely to be a kind of necessary stepping stone. I also really like how it enables something else that

Enabling malleable software via local-first data

35:54 I think about, which is personalized software. Uh and the idea that uh you cuz I I I find increasingly, I don't know if you feel the same, which is I use a bunch of tools that I that I really like. And I don't I don't really have a problem with them. I don't want to vibe code an alternative even if I could. It's it's good. It's good. I don't mind paying them a little bit, you know, it's what I do. But what I increasingly get frustrated is that I can't customize them. Like I want to add They're not doing They're not have this feature and I'm annoyed. And it's almost to the level that's like, you know what? I'm going to write one of my own just because I want that feature. And I guess I'm imagining a world in which uh you the author can produce a piece of software that has some of these definitions of what is what you can muck about with and what you cannot. You know, what is the essence of this piece of software and it has to be embedded.

36:41 And what is it that, you know, Chad, if you want to sort of add a different UI, a different load out, you know, for this. Maybe this is one that we do, you know, we can do cuz APIs and all that. But you can modify it. Um and like have you thought about like the Phoenix the Phoenix architecture or or whatever the I don't know how to kind of call the the whole thing, but like that type of platform being an actual distributable that this is the software that you're sending off to you to users? Yeah, I think so. Although my uh it's not that I disagree, but the way I've been thinking about this is more from the data model up. Um and it comes from like the local-first community. Uh specifically um well, I think Ink and Switch wrote a a good thing called malleable software about some of this. Um but you know, like in the local-first community, it's all about your data is local. It's on your device. People write applications that interact with that data. And then they, you know, you have CRDTs etc. for for synchronizing this stuff.

37:39 Um so why wouldn't you have like a shared data model across uh and I'm sort of not answering a question you didn't ask, but imagine you had a shared data model of all the sort of common things you need to do. And anyone could write software that allows you to interact with your shared data models. So, it could be your productivity stuff like meetings and tasks and emails and stuff. Um if it's not in a cloud and it's not on some private server that you have to deploy to, but rather it's a software that you run locally, then manifesting new interfaces to these things is sort of just like an assumption.

38:14 Because third parties, you know, maybe one one company writes software that you start using. Let's say Linear, which is actually local-first software. Um someone else could then come along very easily and write the second interface to that same stuff and Linear can't stop you. They wouldn't want to. You know, that's part of the point of doing local-first software. And maybe I'm just like an old uh stupid software developer that's, you know, still chasing this idea of reusability and and the semantic web and everything. But you know, what if we could now think industry-wide or at least in pockets of consortia about like doing local-first as as a default and having shared data models um by default. You know, schemas, JSON schema, whatever it is.

38:58 Uh schema.org that is. So, then new vibe coded apps just talk to this stuff and there's an assumption that it's there. I think this is really interesting. Um the the the other idea of like constraining which parts are are malleable is interesting. Although I don't see why there should be a limit to which pieces of the software you should be able to change. Uh though maybe you know, from an IT department perspective, I I do see it cuz, you know, someone calls and says my software doesn't work cuz I changed it, that's a bad thing.

39:32 But part of it is not so much the constraint about what you're allowed and not allowed to change. Like at least a portion of it is is just, you know, like if you if you who in Wunderlist and you created and you've spent this deep thinking about, you know, choosing the right functionality or visuals or like means of it or the things that will drive the correct kind of to-do list handling and so I I don't want to lose that. I just want to add this feature. So it's more like de-risk me when I add my sort of experts to my own social network button that didn't exist on it because, you know, I wanted to extend it. Can you Can you help me preserve, you know, Chad's brain, you know, when you when you when I got the software from you and just modified I mean, like big software does this, right? Like CRMs and all those you would customize your organization. I just find it interesting to think about uh Yeah, it is a great idea because, you know, one of the things that we all when you start talking about regenerative software, the immediate like uh pushback is you can't just regenerate the whole thing. The users won't know how to use it. So it's true that if you could like define even

40:33 pace layers within the UI maybe like this panel and this panel or these types of panels are things that are always extensible. It makes sense and you're right. It's like a lesson from the '90s or '80s, right? You know, like uh this has been a thing in enterprise software forever. It usually not because they want to preserve the experience because the experience is usually not very good, but everyone needs to customize them. Yeah. Yeah, to differentiate. And I and I I love the idea of pace layers and and to be to be clear, I am a huge fan of reusability. I think even if you get to the point in which LLMs can create and agents can create everything from scratch with a very high uh reliability, that is so massively inefficient, right? I want to create a to-do app and it needs to build a kernel. I I mean, why why would it do that? And, you know, familiarity between different systems, consistency within enterprise systems, just mutual learnings, right? When there's a bug, I don't want that everybody to kind of shave and, you know, have all of these problems. So Yeah, but the the efficiency thing is a a really interesting one. I was talking a couple

41:34 weeks ago with um one of the founders of Radical, which is this open open distributed GitHub sort of replacement. Um he is now working on something called Radiant Computer. And it's like a ground-up operating system that he's building. And he's thinking obviously a lot about LLMs and and he wrote a piece uh about how you can sort of like reduce the surface area of how you interact with something, let's say an operating system uh to the benefit of efficiency of LLM use. It's part of what he's doing is like everything is private by default and you can run local models, etc. Of course, you're dealing with like local power constraints and the fact that the models aren't quite as good as the frontier models.

42:17 And it's it's an obvious thing to reach, but somehow there's this beauty in in the the insight that if you like you start with an open playing field and then you create this constrained language that actually expresses what the system's supposed to do. And I think this is like a version of what you're talking about. Then it becomes a very easy thing for the LLMs to interact with it. And you can do this over time with every system where it starts with this open field and then it becomes, you know, it congeals. Maybe this is like what programming languages are in the future. You asked our, you know, about new programming languages. I don't know if there's another guy uh Guy Steele um who is a you I'm sure you know of him. He's a well-known programming language researcher.

43:00 He did an amazing talk at OOPSLA in like '98, I think, called Growing a Language that I like every generation I I reintroduce it to people because it's always relevant somehow. But the talk starts with him uh defining a subset of English from scratch that he can use for the talk. And it's all this big lesson on building up vocabulary and building up foundations you can then be expressive.

43:27 Uh and maybe that's what what we end up evolving toward because in this agent this like rapid change rapid uh software creation environment we should be able to create tons of reusable vocabulary like this. It should and we should be able to easily differentiate between what's reusable for me, what's reusable for my organization, and what's reusable for the world. And the ideal state is that we like figure out markets for this or ways of sharing this stuff. Yeah, no, absolutely. And we we've gone through all of that type of thinking as well at uh at Tessell in the early days. It's like he create a an HTML to markdown converter for me. Sounds like a simple spec. Which HTML do you mean?

44:06 Uh which which markdown do you mean? Uh on it? You know, to be loaded in a browser. Which browser? So and and and and um and and these things rot over time, right? Like whatever, you know, if you interpret it today's, you know, LLMs will interpret the thing from 2 years ago. Well, I guess actually maybe it stays the same cuz it's persisting itself, but in over time the definition, you know, in 5 years now. Hey everyone. Hope you're enjoying the episode so far.

44:33 Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

45:05 All right, back to the episode. So I want to I want to ask you though about another set of posts that you have there. There's more about uh evaluations and durability. And so if if if everything is regenerated and as we sort of we've been talking here, there's some not that that's not always ideal and there's some accumulation of learnings and preferences and and distributions.

Evaluations serve as the real code base

45:28 Um you know, you have a statement that said evaluations are the real code base, you know, they are uh in one of these posts. I guess how do you think about durability? What is it that as I build IP or as I build, you know, build up anything um persists across Yeah, so I I do say this thing about

5 Highlight · Evaluations and invariants replace traditional code bases

45:48 evaluations and it might be an overloaded word where it sounds like I'm talking about some like modern uh definition of evaluations in the context of LLMs or ML and I'm not uh although I also am. I'm really talking about invariants, metrics, constraints, all these things that that we were just talking about, you know, um like to take a stupid just because it'll be easy to talk about. Let's say you have uh I don't know an order record in a database and and you're just talking about like the CRUD operations for orders in a in a e-commerce system.

46:24 Um you know, there would be like [clears throat] correctness requirements obviously. There might be side effects that you expect when an order is placed that should also happen so that you can like measure the fact that that happens. There might be speed constraints. There might be memory constraints or IO or you know, all these sort of like system level things. Uh and and to me like this set of stuff and this is kind of why I was talking about knowledge graph earlier versus specs cuz I think you have to create this knowledge graph which may be pretty pretty inscrutable that we probably need tools for dealing with this, too.

47:00 But it captures all this intent that's both business intent and and technical intent. Now, in my old naive way of doing things

47:09 [snorts]

47:09 which is still the way I would do it for now. Um like I would always talk to my teams about uh focusing on mean time to resolution of problems versus mean time between failure. Which was another way of saying like don't worry so much about testing everything up front, but let's just make sure everything is really well instrumented so that we and we can easily roll back. And so like my way of changing software is YOLO in production.

47:37 Yeah, you know, just to sort of overstate the point. Yeah. And then immediately respond to the live metrics and the you know, the real state of the system via intense observability at all layers including business and and technical. I think there's probably some aspect of of that still Right. at least for now uh in a system that's going to be changing rapidly like this. Um the other thing is you can create tests uh and you can do it both before and

48:03 after the fact. You can do it really rapidly with with AI now. So I think there's a thing where uh and you were talking about like locked tests, I think, in in your way of thinking. Because if you test otherwise, you don't know what to what to trust, right? You you you take some spec, you create code, you create tests, they disagree. Who's right? Exactly. Yeah. My way of thinking about it right now is there's a there's like at least a layer of tests that like everything is locked.

48:30 There isn't differentiation between locked and not locked. Of course, I'm not writing tests anymore. I'm not writing any of the code, but, you know, so I'm explicitly talking to the agents about the purpose of the tests. And the purpose of the tests is based on our current under understanding of the system like in my IRC system, for example. I I will go through runs where I have it just like look for all of the possible edge cases we might not have thought of for a certain class of bugs. And I get multiple models multiple times to do this.

49:01 And then I review it and I have it create tests that discover bugs. And then I have it work until the tests run all the time and I keep this layer locked all the time. So that's sort of like the second way I'm thinking about um evaluations now. I I think probably it goes further and further when we can collectively spend more time on specs to canonicalized requirements to invariants and this sort of chain because that gives you something different to reason about in a more automated way. But right now I think just this sort of metrics plus tests is the way to go. I'm a big believer in evaluations. I think in the um like first year, if you will, of Tessell, I was thinking a lot about tests as the things that persist. And you'll have tests, you will lock tests.

49:47 This is the behavior of the functionality. Increasingly, in in my current worldview, I I think more about making good decisions. And and so the tests that I want are more the tests of my agent's behavior as opposed to the tests of the system that has been built. Uh and those are class I don't know if you'd call anything classic these days, right? But those are AI evaluations of saying, "Hey, is the agent behaving as I am?"

50:12 And I guess I I perceive that to be a higher leverage point because you are you're making the developer that you want versus making the system that you want. And the developer that you want is actually much more powerful than a system because they can create the system that you want at a at a fairly high fidelity. Um and what's interesting is that the same concept that you said right now is true, though. Like there's still a a little bit of you have to capture and aggregate what are the evaluations that are that represent what you want. And you probably have to accumulate them.

50:45 Uh and over time you need to curate them, which is still like a you know, we don't know how to we don't know how to deal with that yet, you know, we have to think But but at least evaluations speak in slightly higher level terms of like when when this happens, that happens, you know, like the behavior uh the the developer behaves like this. It's still a volume challenge of of those do you do you assess? Um but then once you build once you get that, the instructions, the specs actually are malleable uh themselves because they are um they also change with the models, right?

51:19 Like the way that you would instruct Haiku is different since that effectively is different than the way you would instruct Opus. Um and we see already that, you know, Opus is very creative, but it's a bit of a smart ass and it doesn't follow instructions. And you know, Codex is less creative, but it is more obedient, you know, in that sense. And you know, Gemini has its own strengths. And just each of these things is different. Uh and so when you when you think about these like again, if you really continue down the same rabbit hole and you continue you you you move from programming languages to program programmer language, right? It's the

Shifting focus to context engineering

51:52 language with which you instruct the agent versus the language with which you define the system. That's really an interesting approach. And and I I feel this like I switched from Opus to another model trying to arbitrage credits uh a week ago. And I had a very frustrating day until I woke up the next morning and realized, "Oh, I changed the model. That's why it feels like, you know, 2025 again." Uh I blamed the model. I probably should blame myself in the way I interact with the model. However, um uh that actually sort of makes a point.

52:25 Like when you said try and make the best programmer, I think this is a really cool approach. And it's absolutely the opposite of the way I think about it. I just assume the programmers are both idiots now and going to be geniuses later. And they're going to progress at different paces depending on the model. And so I'm I lean toward trying to create a system of code generation. And maybe that's the programmer. I don't know, but it's not an intelligence, you know, but a system that allows me to plug in the most idiotic LLM here and the genius here and still have some results that I feel I can I can trust. But also decouples me from thinking about the models underneath.

53:05 They're just going to keep changing and getting better. Of course, eventually all this stuff that we're talking about is going to be irrelevant, too, and obsolete cuz the models are just There's a different point in the future, which is you know, having kind of a a more intelligent conversation by a couple of AI models than what we're sort of having here. But yeah, and I think for this work, I think you need both because I think amidst the team, you need some form of definition of the outputs of that, you know, intelligent programmer that that captures the definition of your system and makes it easier for the programmer to be, you know, less intelligent or even even just to capture the alignment on what the system does.

53:42 And you know, making that increasingly LLM friendly so it's easy for them to

6 Highlight · Context engineering replaces software development lifecycle

53:47 understand and kind of apply and represent instructions. And then you separately need the instructions of the of the developer. And I I I think to me, context engineering is about that, right? And I think we started writing and we'll probably talk more about the context development life cycle. And I think I think the agents will handle the software development life cycle. And we need developers to sit in the context development life cycle as we develop context. And the context really is the operator's manual, right? It is about how do how should the agents behave and what your intent is. It'll be interesting to see the interplay there.

54:19 Yeah, a friend of mine, Jess Martin, is working on something called context library, which is a way of automating building up context of all of these these decisions that you make. Yeah. With the goal being kind of like what you're saying, run agents all night while you're asleep and they make decisions based on probably what you would have said because they have enough context in the system built up.

54:41 The way that's usable by them. I see a lot of that. And then I I do wonder the reason I say context development and I think it's a different view on it, although I time will tell which one is right, which is there's this spontaneous spontaneous, you know, observe me and understand what I do. But then what I do today is not the same as I would have done a year ago. And let alone 10 years ago. And so like the the ultimate approach is context is a thing you develop. It is not a point in time. And we all know like at one point during the LLM kind of iterations, all the rage was learn my code base. And then we learned like there's a bunch of stuff in my code base that I do not want replicated. In fact, most of the stuff in my code base I do not want replicated cuz it's been accumulating garbage. And garbage in, garbage out.

55:29 Uh and and so it's not that it's not useful, but you want to develop your sort of practice. So it's interesting. I guess uh maybe it's a good point to sort of

55:39 try and conclude the conversation in the point of timeline. So what is your thesis? Like you're building put aside sort of the the guide to the programmer even side, the switch from current day software to immutable regenerative, you know, Phoenix architecture style software.

AI software adoption will be dual-paced

55:58 Like what what do you think is a is a practical or what is your anticipation in terms of the uh the the path of adoption for software actually changing to be working like this in the industry? I I think it will be both really quick and incredibly slow. And the the slow part is the realization we need to all make that we have systems that have been in development for decades and no one understands them.

56:25 And most of us nerds out on the frontier are not thinking about them so much yet. I think some of us start are starting to. Um so like uh I think, you know, if I were to go start a company now, I would be doing the company this way today. So, you know, greenfield code base right now. Now is the answer. And I know some people who are doing this to, you know, varying degrees. Uh but I also know of development shops in, you know, fairly forward-leaning companies that are barely touching AI at all. And they're resistant. And there's a huge portion of the developer population that's afraid or just not paying attention. Yeah, just like when Java happened, they didn't want to do Java.

57:08 They didn't want to do Ruby and Python, you know. Uh and uh you know, they just want to have fun and and go to work as as little as they can. That is the truth about most people in their jobs, which is fine. That's, you know, humans are humans. Um but organizations are backward. Organizations are slow. Most people don't know how to deal with this and they're going to be looking at like large consulting companies to try and figure it out. So, I think it's a long, long time before there's a mass adoption of AI-generated software up and down the stack through, you know, the largest enterprises.

57:41 Um but there's going to there's a lot of interesting work that should be happening probably in the coming year or two because so many of us in companies I'm talking to, you know, from my perspective as a VC are having these same conversations about like trying to capture intent. And what is a spec versus intent? And where can intent come from? You could mine it from corporate email systems for, you know, the last several decades. You've got all that stuff captured. You could mine it from source control system, project management.

58:11 So, work being done to figure out the legacy intent of its legacy systems so that you can then start automating maybe like strangler fig pattern breaking pieces of an application off. Um but yeah, I think it's going to be like insanely fast on one side and surprisingly slow on the other side.

58:31 Yeah. I think that's a good it's almost always like the right answer into like any diffusion of technology. I do what one of the things that I do feel and maybe this is in sort of Tessell on the company building ethos side is that one of the reasons that we've kind of focused on on the sort of, you know, train train the developer versus, you know, define the system in in the offering, you know, there's there's a whole bunch of reasons some of this stuff needs to, you know, be open source, needs to sort of be part of the agents, you know, there's a variety of reasons. But part of it is that if you train the programmer, then you're sort of simulating the human behaviors, right? Like in these ancient systems that are so fragile, there are still humans, they're operating in there and they're making decisions.

59:14 Uh and those decisions are, you know, by and large these companies are still alive, right? Like they are they are growing, they're modifying it. And so there's some logic by which they are making this work. And so we felt like it's even though conceptually as we talk about the it's a higher layer and it's maybe a further future to think about training the programmer versus training the there's actually in some aspects of it a more short-term applicability of it because that better programmer can help you do better on the systems even if it doesn't fix the fragility or it doesn't make those systems you know be able to rise up from the ashes.

59:51 Yeah, I agree. I mean I think you know, it's like what I said about everything's going to be obsolete that we're talking about at some point. I think there are waves where we will solve these problems in the way we're talking about today and it's totally valid and it will be useful for a long time. But the industry is just going to go so fast we'll think oh God, what we're talking about now is silly, but it's still useful. But it's maybe not the future anymore, you know, the beautiful thing about today is we can go so fast.

1:00:17 We can deal with all these different phases. You know, and we can have road maps for our companies that where we just say okay, we'll get to that when it's a problem. Okay, now it's a problem. Boom. Next week we got a whole new release, you know. It's like a miraculous time to be a software developer as well as a terrifying one. But yeah, I think it's it's where we get to be like steering [snorts] through this crazy time. It's really exciting. Yeah, yeah, definitely sort of fasten your seat belts, but it is it is an exhilarating ride.

1:00:48 Chad, this has been super super fun, you know, fascinating, future-looking, you know, sort of deep on it and and I think right in many ways. So I'm really looking forward to to the open source version of it and to hear more about about this development. I don't know if you want to say if like if people are are keen to to find you to invest in their sort of you know, invention of sort of future of software development, where can they find you? Probably Blue Sky is the best place. I'm Chad Fowler.com on Blue Sky. Sounds good.

1:01:17 So thanks again for coming on to the show. Thank you. Very fun.

1:01:21 Thanks everyone for tuning in and I hope you join us for the next one.