Maggie Appleton: Agentic Coding Has Outgrown the Text Box

11,444 words · about 46 min to read · YouTube’s automatic captions

0:00 I just feel like this like very textual simple interface isn't cutting it and I want agents in there proactively doing stuff. And I do think agents prompting us more would actually help relieve our cognitive load which a lot of us I think are struggling with in the age of agents. And you can't solve it with a multiple choice text question. If you're asking me to pick between a bunch of NPM libraries, give me like a comprehensive report of how these compare. Don't wait for me to prompt you.

0:26 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts Guy Perjani and me Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. Yesterday, you were writing code. Today, you're prompting agents.

0:52 Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works.

1:13 This November, we're bringing it back to New York, bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. As a podcast listener, you can get 15% off your ticket using promo code pod50. We have so much planned and we can't wait to have you join us in New York.

1:45 Hello and welcome to another episode of the AI Native Dev. Joining me today is Maggie Appleton, a staff research engineer at GitHub Next. And we're going to have a great conversation tackling such topics as how humans and agents interact and what are the handoff points between them when we think about planning, when we think about development, and so on. So, we've got a huge number of topics and depth that we're going to talk about. Maggie, welcome to the podcast. How you doing?

2:12 Good, thank you. Thank you for having me.

GitHub Next explores exploratory interfaces and software factories

2:13 Oh, absolute pleasure. Now, Maggie, tell us a little bit about your role and what you do at GitHub Next.

2:18 Sure. So, technically, staff research engineer. That is the title of almost everyone on the team. I'm actually a designer. I'm a front-end engineer, as well, but I'm not the best. I'm much more at home making like exploratory interfaces, visual design. I came from a background that was like an illustrator, art director, and then moved into more front-end stuff working on like dev education. And then I I just love a lot of things like theory of programming and like researching it, what's the future of software development, which is why I get up next is a very good place for me to just kind of do interesting experiments about what GitHub should work on next.

2:51 Yeah, no. I was just going to say, what is GitHub Next? But for for people who haven't heard of it, tell us a little bit about the kind of It's like a lab research space, right? It's super interesting. Tell us a little bit about a few of the interesting projects that are are being worked on and that have come out of GitHub Next.

3:06 Sure. So, yeah, GitHub Next is is the labs R&D branch. So, like, pretty standard like innovation, what the usual product engineering org wouldn't work on because it's too risky or too far out there. This lab works on like big swings, further in the future sort of stuff. And this lab originally made the first GitHub Copilot, so like, the first ever AI coding agent before anyone else had made one. And of course, now there are plenty, but this was like the OG. I wasn't on the team, so I'm not taking credit for that.

3:33 But there's tons of always tons of interesting projects. I feel like we're in an exploratory phase right now. So, everyone's prototyping like mad. Every morning I wake up to like five new prototypes to try to understand and and figure out if they have legs. Like, a lot of stuff's early. But some of the big projects out of the lab at the moment is like genetic workflows. You had Don on a little while ago to talk about essentially software factories or like he's trying to lobby for the phrase software farms which I like more like you're harvesting the crops of agents all the time it's less mechanical it's more like growing.

4:02 Yeah.

4:03 And then I'm working on a project called Chopin which is all around how we plan with agents and collaboratively with other developers and other members of your team so not just you and an agent alone but you and your whole team planning more rigorously together and more visually because I'm big on like diagrams visual explanations like tapping into how graphic design can help people understand complex concepts.

4:25 Mhm.

4:26 So for me that's a big piece of it but other people working on the project are more excited about you know the multiplayer piece and and tapping into like letting all your local agents work with it so this there's layers to the project.

4:36 thought you were going to say like Chopin it was going to be an orchestration piece or something like that but I suppose with multiplayer it kind of is yeah.

4:42 a lot of stuff in there my assistant is the engineer on it he's from Poland so Chopin is a very famous Polish um composer so he loved the name cuz it's all it is about orchestrating agents there's multiple agents in there you can kind of hand off the implementation and see how the work flows from plan through the finished product so there's a few things to it but the piece I love is the like the the collaborative planning the visuals the like rich rich artifacts to help developers understand what they're about to ask agents to do and then when the work's done trying to comprehend what the agents did.

5:13 Yeah.

5:14 The two big bottlenecks I think in in current work.

5:16 Nice. So you're in the github.next space is it github.next or github next?

5:21 github next

5:21 github next

5:22 Like the next github.

5:23 github. I presume I presume you refer to to current like you know the mainstream github is either github old or github

5:30 current and then I in my head there's a github preve which is like legacy working with legacy clients or something. If you say if you say no I'm just going to imagine that anyway so that's

5:40 we kind of call it dot com.

5:41 dot com okay yeah.

5:43 Cuz it's like we we don't tend to work on dot com cuz dot com is like a big a big thing right with a lot of real clients, needed to stay the same. So, we tend to work on things that are like more prototypes on the side or might end up rolling up into .com, but not directly on the surface.

5:56 Yeah. Nice. Nice. So, Maggie, one of the things I love is going through past

Scaling developer teams requires more than local agent speed

6:00 comments that people have made. And and yeah, dragging up the past and and and and showing the dirty washing on the on on show here. So, one of your lines was nine women make a baby in 1 month. Uh and that's a great I love that logic. I love the line. Um so, concretely, what is a what is a team look like that's, you know, shipping five features a day?

6:19 Yeah. Uh so, this line came out of a talk I did uh that was about a collaborative developer uh platform that we had made called Ace or prototype that Shapon is kind of related to. And it was essentially just like Slack, but with um micro VMs for code sandboxes in real time and agents in there as well. So, like all-of-you-in-one uh environment. And this is now like happening a lot more with their products now, like Buzz from Jack Dorsey and like even Clojure getting into Slack is like kind of going that direction. This was maybe like uh 6 months ago that we had a prototype of in this direction. So, it's nice to see other people agree with our theory.

6:53 Um and uh the problem we were trying to solve was that uh everyone's going really really fast with their agents locally in their terminals or their local desktop apps. So, everyone on your team is accelerated, but like scaling up one person doesn't lead to like good collaborative products because everyone's off on their own branches doing their own thing. It's really you get carry merge conflicts eventually or you get features that like other people didn't agree to or didn't get to put input on.

7:18 Mhm.

7:18 So, one person could build something and they didn't really consult like the designers or the PMs or the back-end people. And then they have to like throw away the work after they've built it cuz it they weren't aligned up front. So, this metaphor of like you can't nine women can't make a baby in 1 month. Like

7:33 you

7:33 not possible.

7:34 No.

7:34 All right. Okay. Okay. Okay. I was questioning

7:36 Unfortunately, unfortunately, I know. I would expect labor. Um but, you know, in order to to do big things, big ambitious things, you all need to be aligned together as one. You can't split up the work amongst all of you and think that it's going to lead to some cohesive product in the end.

7:51 Mhm. Awesome. Uh one of the other things that I think most people will probably agree that humans uh are similar to this as well, but you mentioned that agents are shissa CSS. Uh and I and I would say that's because they've learned very, very well from us being really bad at CSS as well. Uh but honestly, what what what would you say the current generation of

Current models need better visual reasoning capabilities

1 Highlight · Why current AI models struggle with UI and design

8:10 models are typically very bad at that we're not acknowledging as as an industry?

8:15 Yeah. I feel like I I do hear people complaining about agent design skills is I think the term they would roll that under. Um and when I hear people say they're bad at design, what I'm actually hearing is models are trained on language, right? And and of course on code, so they're trained on CSS. But, the way that a human designer like looks at designing something, they are mostly working with visual input. They're working with what we would think of as like classical graphic design. So, you're trying to say, "Okay, what's important for someone to see on this page and what do they need to do?" And then in order to achieve that, you use things like space, contrast, priority, color, grouping.

8:51 Um you like use the principles of of how humans interpret objects and colors and values in space to like make the person see the right thing in the right order, click the right button, understand what's on the screen.

9:03 [snorts]

9:03 Um and agents haven't been trained on visual inputs. Like we can do, you know, image generation and pass images to them, but they're really not good at translating like visual reasoning into code in the way that a traditional human designer does, right? Um so, sure, it doesn't help that the CSS is all probably pre- pretty crap in the training data, um but I think it's much more this missing piece of that they're not multimodal enough. They're too heavy on language, and so they're unable to like see what we can see. So, I often, when working with models, trying to prompt them to do designs, they will just like make elements overlap cuz they can't see. Even when take screenshots, they don't really understand how much space needs to be between things. Like, what does it mean for something to be visually aesthetic? They don't have an understanding of aesthetics, especially because it's um historical and cultural in the sense that like what we think looks good changes over time.

9:52 Yeah.

9:52 So, there was an aesthetic in like 2010 that if you had a website in that style now, we would think of as very old school. Like, oh, that looks like you sliced that sidebar out of Photoshop and like stuck it in, you know, like we used to do.

10:02 Mhm.

10:03 Um and now the aesthetic is like linear with cell, clean, kind of modernist. But, that will change over time. So, we can't even like bake into their weights like what good aesthetics are cuz that evolves.

10:14 Yeah.

10:15 But, we might be able to do it through, you know, skills or just like giving them more relevant data for models at a certain time period. But, it's like a a more fundamental issue with the fact they're so language-based and they don't have this visual reasoning capacity.

10:27 Yeah. Do you think Do you think the visual reasoning will need to uh account for cater for agents as well these days cuz more and more agents tend to almost be going to the browser to do things at times where APIs and things

10:40 like that don't exist. Um is that is that something we need to cater for? And I guess a follow-up question is, in terms of what you see as fixing this,

10:48 Yeah.

10:49 do you think general models can will learn to do this or do we need like some kind of specific model to uh to

10:56 we'll probably get general models there. I assume that the labs are doing more multimodal training. Like, I assume, you know, I mean, given the speed of things, maybe 2 years.

11:04 Yeah. Yeah.

11:05 You know, if it's slow, maybe five.

11:06 Yeah.

11:07 I definitely am not in the camp of like, oh, models will never overtake designer jobs. I I don't know they'll take their jobs, but like in the same way that we've been able to automate away large chunks of the software engineering process. Or I should say like specific tasks within that

11:20 Mhm.

11:20 uh job. In the same way, I hope actually there's a future where I can describe what I'm trying to achieve to an agent and the interface it makes is at least usable, like baseline aesthetic. There's enough space between things, the text is readable, there's not a purple gradient on it, like there's like some baseline that I think we're really near and and within a year or two we'll get to. Um and then there's the question about like do you do you think like agents will need to browse visually? I feel like we're mostly just going to redesign the web and websites and like I know there's like other ways they could browse, but I think we'll just they'll have APIs and they'll just like use agent methods cuz text will always be faster. Visuals are faster for humans because our you know, brains are built differently to you know, I say agent brains not the same, but the way we process information is different to them and we like visuals and space and gesture and movement in real time and they're very like static language, you know? Mhm.

12:11 Yeah, just give me one line with no spaces, no white space. Yeah, yeah. Um

Designing better boundary objects for humans and agents

2 Highlight · Agent MD files fail the boundary object test

12:15 okay, one other thing that you've mentioned is boundary objects and and and you borrow this from Susan Leigh uh Star's sociology. Um so, most engineers have never heard that term.

12:25 sure I haven't.

12:26 Tell us a little bit about what it means and and why um I guess Agent MD files fail that test.

12:31 Yeah. So, yeah, so this is like a classic sociology paper. Like I studied anthropology, this is like one of the things you read in 101. Um and it's from 1989, it's not that old really, but it was the study of how um Berkeley's Museum of Vertebrate Zoology first got started, which was like early 1900s. And these sociologists studied how many disparate groups of people came together to found this museum. So, you've got academics, you've got funders, you got biologists, you've got like trappers and naturalists and they all had like different levels of uh training and expertise. They had different priorities, some people care of course more about funding and and rules and others are just concerned about like biology and capturing specimens.

13:10 But they had effective boundary objects to collaborate is the theory um that they worked with here. It's like they had specimens and labels and field maps and everyone read these objects differently, they like worked differently for each group, but they enabled effective collaboration. And you could point tons of examples of this out in the world. You just think about any any huge project that brings together many different kinds of cultures and communities of people, they need effective objects to collaborate around. Um but the objects can mean different things to different people and take different forms for each kind of group that it interacts with.

13:40 And this applies to to humans and agents pretty well. If we treat agents as a kind of being with a kind of way of seeing the world, and it's a very different to the way humans see the world and think, and we don't need to perfectly understand each other's worlds because that's probably impossible. Um I always kind of refer to the fact that, you know, um we are humans embodied and we like talk in real time in space. We're very social. Agents have none of these qualities. You can give them cameras and mics, but they're still not going to understand what it is to be an embodied being in the way a human would. Um but they don't have to. As long as we have effective what what we currently use as right here, agent MD files, skill files, MCP servers, effective boundary objects to collaborate with them, then we can actually achieve things together without needing perfect shared understanding.

14:24 So is that context like an agent MD, is that an effective boundary object?

14:27 Uh for the moment, it's sort of okay, but I don't think I think um it serves agents well because it's it's a text file. Everything that we do is like very text-heavy. We are reading these huge outputs in the CLI. Even the fact that current agent coding tools are so CLI focused, I think it's like preferencing the agents' preferences and not the humans in terms of understanding and being able to communicate. Um so we are having to read and write masses of text, which is not very effective for humans.

14:55 We are much faster visual-spatial social reasoners. It would be much more effective for you and I to like plan out some complex architecture problem with a whiteboard in real time, right? Drawing diagrams, talking, pointing to things, moving around. It's much less effective for us to like type words into a document and that to be our way of expressing what we want and problem-solving together. Um so the the big thing I'm trying to push for is like we like they're fine boundary objects like these text files, but we could make them preference humans far more by making them social, real-time, interactive, visual. Um and I when I say interactive, I mean like it's not a static text file. Like we have beautiful like systems like the web and browsers where we could make everything interactive. You could be able to like pop poke around an interactive diagram, make a a mock-up of your UI and compare it to what you might want to move sliders and move little things. So I like in design we make jigs a lot which are like little little control panels just to solve one particular problem. These are the sorts

15:55 of things when I say visual interactive that I mean is like make this stuff easier for humans to understand in a way that a wall of text is not. It's very difficult for us to comprehend complex systems in walls of text.

16:06 Yeah. Yeah. And it's kind of like leads on to the final quote which you'll be happy to hear which is when you described humans and agents as looking at looking at each other through a keyhole. Uh and this is really interesting. It's kind of like that you

16:21 know that problem of of really not being able to to kind of like see as much or understand as much of the the entire entity as from both sides as as as you might want. Who's job is it to I guess widen that keyhole? Is it the the model builders? Is it the tool builders like yourselves? Yeah, or is it the humans I guess learning better how to prompt, how to provide that information and context to the agents?

16:44 I think it's it's mostly tool builders, I think. I would maybe include model builders in this in the sense of like actually the moment the model builders and the tool builders have so much overlap and you if you think about the fact that Claude and chat GPT really dominate in terms of like they kind of own the interface and the model. Um not to say that there aren't other players in the space like of course GitHub Copilot is another one that like gives you tools but doesn't own the models. Um at least in terms of widening the keyhole that letting letting the agent see more of our world.

17:14 Uh at the moment it is like we're just like funneling text through this small hole. Um I mean I can kind of touch on like interpretability is like the study of how we understand how models think and that's much more what the model companies do, right? Because from the outside we can't do that. That is like foundation labs are really the only ones in a position to do meaningful interpretability work. And I think that stuff is important to for us to see how agents reason. But to get back to the boundary objects thing, it's actually not as important for us to understand their worlds more comprehensively or even them ours as much as it's important for tool builders to make these kind of environments that humans can do good thinking in and then create artifacts that you can pass off to agents that are optimized for the agents as well cuz they want to you know, they can happily deal with reams of text just tons and tons of files like nested skills that just contain all kinds of things, but that is really difficult for humans to work with. I think it's the responsibility of the tool builders to make better interfaces and environments for humans to like figure out what does the system do? What do I want it to do? Like do how do I design a reliable

Moving toward proactive agents and richer interim artifacts

18:16 architecture that will withstand the test of time? Those need different interfaces to just like text editors.

18:21 Yeah, it's down to the developer experience or the human experience now in terms of how we actually how we actually interact. Um let's talk about planning. Let's talk about planning and design, I guess. Um how does that change when agents actually rather than um I guess being on the other side of the keyhole where we're just providing information to them, how does that change when they actually become collaborators? So um roles, responsibilities, how they help, and how they become part of the team. Is that something that we want? Is that something that, you know, we actually

3 Highlight · Proactive agents and breaking out of multiple choice

18:51 want it to be much more human-oriented and actually maybe we get a bit of input from agents. What's the best dynamic you think we should have?

18:58 I definitely think agents active participants in planning is really good cuz it kind of gets into not like, you know, I think we should have a thousand layers of agents and everything. There's like the agent over here who might implement something, but they can be far away from the fact that like when you are trying to plan something out, there's a whole stage where you need to like understand the current state of the code base, um and then figure out what you want to change, and there can be all kinds of helper agents in there, and we already do this. I think in a in a way that is inefficient.

19:25 Where like we're in our CLI, we're like, "Okay, I want to plan something." Like go into plan mode or whatever. And the agent's like, "Okay, I'll spend more time like asking you questions." Like I'm sure if you you've probably used like Matt Pocock's grill me skill, like great infamous at this point, where asks you like 40 questions in a row, all multiple choice in this text editor. And for almost every question, I feel like each could be its own like research thread or like rabbit hole I need to go down to say like, "Okay, you're asking me to pick between a bunch of NPM libraries. I don't know. Give me like a comprehensive report of how these compare and how they relate to the problem I'm trying to solve." Or like you're asking me about visual design.

19:59 Show me visual design options. Like show me my design system. Do we need to change the design system for this design to work? Like every one of those is a complex problem to dig into, and you can't solve it with a multiple choice text question. You need to make it its own rich artifact. But this is where agents can help. Like every one of those could have a bespoke interface designed for you with agents. Which is also why I care about agents getting better at visual design. It's like I really do want generative UI to work out, not for the end product, but for the interim developer experience. Um so like I just feel like this like very textual simple interface isn't cutting it. And I want agents in there proactively doing stuff, too. Don't wait for me to prompt you. Be like, "Oh, I see where you're going. You're trying to work on something with auth. I'm going to go research a bunch of like cutting edge practices or like what are the new libraries you could be using? I'm going to go do web search ahead of time so that by the time you've reached this topic, I know where going to get to in the plan." Like I already have reports ready for you with bespoke interfaces ready to go. So a lot of I'm exploring

21:01 this with Shriram is like how do we get agents to look ahead, be proactive, do the research ahead of time, present you with decisions like as you move through without you having to always be the one to like prompt it. But

21:13 Yeah.

21:13 I think people have been not controversial about this like, you know, get the agents to prompt you because it does think like, well, the agent sometimes forgets to ask you things or like it can't it's not as good at critical thinking as we are at times unless it's prompted to be.

21:26 Mhm.

21:26 Um so you don't want to like let the agent drive too much, I think is the worry. And at the same time I do think agents prompting us more would actually help relieve our cognitive load, which a lot of us I think are struggling with in the age of agents. You're just constantly juggling seven different sessions and trying to remember what you've been doing. I really want agents to be like, "Hey, every morning, here's a summary of what we've worked on.

21:47 Mhm.

21:47 Because of these, here I think are five good next steps you should consider." You know, it doesn't I don't have to run with them, but I want them to prompt me more. I want them to be doing more sense-making across the work I'm doing over the course of a day, a week, a month, and saying, "You're you're forgetting this. You haven't looked into this. Good thing I looked into it for you. Here is my results." Like, I'm looking for that.

22:07 And there's a couple of areas I want to dig a little bit deeper in. One which we'll get to is the single player versus multiplayer on that planning cuz I think the dynamic changes and how you actually interact with the agent changes. But I want to touch on junior developers because this is I I

22:20 what I what I think this [snorts] can lead to is actually a really good path for junior developers to actually, rather than, you know, the worry is, "Oh, a junior developer hasn't been through the trenches like like a senior developer has. They don't know the pitfalls. They don't know what to look out for or or how to design or plan well." Now, if you have an agent here helping with that planning and understanding the pitfalls or at least having the skills or knowledge to to to recognize what a senior developer would do or the pitfalls that a junior developer might walk into, how much how much will will that do you think um kind of bridge the gap

22:55 Mhm.

22:56 between you know, the senior developer

Bridging the gap for junior developers through shared agent visibility

22:59 junior developer uh, wisdom that that sometimes lacks?

23:02 Hopefully a bit. I mean, if we think about like pair programming must be totally dead by now.

23:06 Yeah. Yeah.

23:07 Which is how you used to learn.

23:08 It's probably still alive, but it's a pairing with an agent versus

23:10 Yeah, okay.

23:13 Um, but this thing of like everyone like vibe coding their own apps, right? They don't even know what questions to to ask. They don't even know to ask about security or they don't even know to ask about how to deploying something like that.

23:22 Yes.

23:23 Yes.

23:23 Um, but but if you ask the agent the right questions, you can of course get good quality answers that like teach you all about those very important topics.

23:30 Mhm.

23:31 But I would hope like in when we get agents to move into these more real-time collaborative spaces when we're doing our development work in our of these like local silos, I the hope is that juniors and like everyone on the team just gets more visibility into everyone else's agent sessions. That's like one piece of it is the way we've designed most of these systems is open by default within a team. So like everyone can see everyone else's agent chats.

23:53 Mhm.

23:54 This is like we got of course feedback this is a bit controversial that everyone's like I don't want people to see what I'm asking the agents, but if you're on a team, you're building the same product, you really it should be open.

24:03 And it should be sharing as well, right? Because you you want that level of, uh, you know, one person not being super effective and other people being, you know, ineffective because they they they they don't understand the prompting just as much and so forth.

24:15 Exactly. Exactly. Seeing how other people prompt the agent, seeing what kind of questions they ask. But even if you were able to like like I'm I don't do back end or I I if I do back end it's very bad, but I get to work with people who are like very senior back end people. If I could actually see them writing plans in real time and seeing what they correct, what questions they ask the agent, where they ask the agent to go do research. Just like I think that's the way that you would get junior developers to actually learn is to be in that sort of environment of like they see the quality of the level of plans that a senior engineer would hand off to an agent and they understand more about what's gone into it. Versus like I don't even know what they're doing at the moment other than sitting alone with their agents not asking the right questions.

24:55 Yeah. And I and I wonder I I even if if the junior dev, let's say, even needs to know some of those things as in needs to know to ask the agent some of those things or whether it's actually kind of like built into the agent that other the you know the more senior developers are asking those questions or are you know offering opinions in these areas. I should make sure this is also considered here and I should probably tell the junior developer, designer, planner that this is what I'm doing and this is maybe this is an important area or something like that.

25:31 Yeah, I'd be surprised. I'm sure people have skills for this already.

25:34 It has to be it's going to be more of a guidance or skill thing as well.

25:37 Yeah, or maybe they'll bake it into the models where like models now I think that we're finally getting a a handle on memory where it really has an understanding of your profile and what you're good at and bad at. And I think if it could recognize someone was a junior, I would hope the model was to some degree trained to ask more proactive questions or like understand. But it conflicts with the with the rule like you know the human's always right. Yeah. So, I don't know. Do we like put junior developers in a position where they don't have as much authority over their agent? That seems troublesome. So,

26:05 Maybe it was instead of saying you're absolutely right, maybe it would just say ah you're you're probably right. Yeah, yeah.

26:10 I mean this is like cuz for a while I was a bit worried about the whole thing of of agents always validating our ideas. I'm like I I wanted an agent that would like enable my own critical thinking and my own reflection. So, at some point I put in my Claude system prompt like you should question me on everything. Like don't let me just assume I'm right. But it was so extremely annoying the way that it behaved afterwards was every single thing I said even if I was like I'm researching new kitchen scissors to buy. It would be like well, you probably aren't correct about your assumptions of what you need.

26:42 Just I just turn off. Never mind. I regret this.

26:49 Um let's Let's some single player versus

Balancing single-player flow with multiplayer background agents

26:51 multiplayer here as well. So, as an individual, if I was to think about planning, yeah, I you know, just interacting directly with an agent makes a ton of sense that I can kind of like ask some questions, have a good back and forth. But of course, let's say you and I were to do some planning, exactly what you said, we'll we'll be in a whiteboard and we'll be talking and drawing and and and visually planning. How does an How does an agent best interact there? Is it Is it something that they should validate in post? Is it something that they should almost like, you know, hear our conversation, understand our assertions and and why we're doing things, and then give commentary?

27:27 What's the best place to use it?

4 Highlight · Classifying team conversations into structured artifacts

27:28 Uh okay, the thing I'm playing with at the moment is is a little bit of them like watching quietly the chat. So, in the prototype we have in Chopin, we have like a chat and a and a document. And the document is more like the the artifact you're making, the plan you're making. And the chat is of course free form chat. But when we when I'm planning with my engineer, like we are also usually on a call talking. And at the moment we don't have it hooked up with the agent like reads our transcript, but I do think that would be a good direction to go in. But at the moment what I have is like we're typing in the chat. And of course like, you know, Jeff is like the the hottest thing ever.

28:00 Of course, the minute Jeff came out, I was like, "Perfect. We've been doing this badly with models. Let's go." Everything you type in the chat gets classified as like, "Is this a question, a proposal, an opinion? Is this a constraint on an existing decision?" So, the agent is constantly trying to understand what's happening in the chat and it's putting things into the plan in a structured format. It's like marking like, "This is a decision you need to make. Here are the options you've laid out for that decision. Here's like the goal that you stated in the chat." So, you're trying to turn this like very continuous free form human conversation into a more structured artifact that you will work towards handing off to a implementation agent.

28:36 So, this starts to head in the right direction for me like we're trying to figure out how to put in like TLDraw whiteboards to be able to draw. And I need to figure out a way to hook up something like a real time call transcript to it, but my ideal like planning environment is free form chat with someone where we just like hash out the details, but the agents of these background agents classifying and then when they spot something like we have an open question that could be a web search they need to go do. I want them to proactively write that research report, put it in line in the plan. So that's the next piece we kind of want to work on.

29:09 Yeah, really interesting. And I think the handoffs that you mentioned there when the agent interacts is super super critical. Um as a parallel as an aside, I've been I've been using Claude, I must admit, on voice

29:21 on voice chat while I've been cooking recently. And it's been it's it's it's not great at timings. But it's good with it's good with telling me, "Oh yeah, you should add this or that." I I kind of like show it the fridge and I'm like, "Oh yeah, you can add this and this and this and add that." And one of the it's great when it's just me. Then when my kids walk in or my wife walks in and just say something, all of a sudden Claude's like answering back in my ear. And it's like, "Okay, that's not great. I don't want you to interact at this point." And I guess the one of the one of the questions here would probably be like recognizing there is a there is a point in time when I want an answer from you or I want a contribution from you.

29:59 And it's almost like keep like milestones where it's like, "Okay, we've we've got to this stage. What have we forgotten? What do we need to add? What are we What are What other constraints should we be aware of?" And then get it to get it to come in at that point.

30:10 Yeah. Yeah, I know. I This is why I've struggled with that real time chat interface. Like I've tried it with ChatGPT, but it's the same thing. It's like someone I'll be like out in the park walking with my kid and then trying to talk to it, but then of course you like look weird like you're chatting to this agent.

30:24 And then someone gets too close or they come talk to you, you're like awkwardly like hang up a call with an I don't know. The whole thing socially doesn't work for me. Um and it's this

30:32 Really, I love it. I love that. I really love it socially, yeah.

30:35 Yeah. Um but it is like I think the way I always try to design these like these collaborative like agents in real time systems is agents as in the background. I don't want them interrupting, especially if I'm talking to a co-worker. Like they do not have the right to like come in. Do you know like in the beginning when we've had all these um agent transcribers in like Zoom and stuff and they would like look like a person in the screen and I was always like they do not have the prominence to be a person in this call. They are like a a background helper. Um so I haven't really explored much where the agent interrupts in any sense. I definitely want it to be like it's doing continuous background work in a way that you can always uh discard it. Cuz it might do some research that you're like actually not relevant. Like my opinion I already have experience in this. I'm picking this framework. I don't want to hear about your research report. I think it's something like we have to assume all the agent work can be like just disposable.

31:27 Like it'll become cheap enough and intelligence will be cheap enough that this is all just like fine for it to constantly churn and try to suggest things and then we can dismiss as needed. Um but I don't I don't I haven't yet figured out how they could really be active participants in a way that doesn't feel invasive, annoying.

31:44 Yeah. You almost need like that Jev decision that the agent thinks should I interrupt here or should I not interrupt here? Am I Can I Can I contribute something that that my that the users

Formal verification and rigorous handoffs before execution

31:53 here would actually find valuable at this at this point?

31:57 Yeah.

31:57 Um let let's talk a little bit about handoffs. Huge human to agent handoffs and agent to multiple human handoffs as well. Um I guess what what do you find what what do you find is the biggest problems with handoffs today? Maybe around content specific context specifically.

32:13 Yeah. Um it's interesting actually. We're getting someone to come work on Shipon who's really into like formal verification. So So at the moment right we all like write this plan and we're like, you know, you might ask the agent like have I thought through everything?

32:27 And like it's usually just going to be like sometimes actually it does have good answers. Like you can always try and sense check yourself. But it's it's almost certain right you hand this plan off and then the agent always finds things during implementation, right? I always tell it like, "Okay, you can make decisions as you go cuz you're going to need to. Um if I'm doing it where I'm not around, I just want you to log them. At the end I want like a really nice visual report where you summarize everything you've done and why you did it and a list of all the decisions you made along the way."

32:53 Um because we just it's so hard to comprehensively cover everything up front. It always finds bugs or edge cases or like, "Oh, we didn't consider what happens in this in this state." Um formal verification systems are supposed to help with this where like you might take a a plan that you've made with a coworker and it's to your the best of your ability comprehensive, but then try and turn it into a formal spec where it really tries to cover every single possible use case. And that also means that after the implementation, you can compare it to the formal spec and like validate whether it actually did the work it was supposed to in a way that's harder to do in kind of more free-form like natural language plans. So, that's one interesting direction is like could you just like computationally verify that the plan was completed to a certain degree.

33:35 I mean yeah, I do think the other one is just like their reporting on the other side helps with you figuring out like, "Okay, this is this is what went wrong along the way." Yeah.

33:45 And and how much does this kind of lean into shared understanding before delegation? Cuz I guess I guess you know, is that shared understanding should that be between multiple agents, many humans? Where where do you draw that line between shared understanding?

34:01 Yeah. Um I think we've explored this a little bit, but probably not enough in the sense that I know in some companies like you really want everyone to not have like seen or checked the plan, but to have consensus that before you hand it off, especially um I think if you're being ambitious about your agentic runs, they can be really expensive. If you're really like, "Okay, we've planned this feature thoroughly. Like we want to run like I mean I love orchestrator agents." I know everyone does, right? But I had always had mine to a powerful orchestrator and I say, "You use, you know, less powerful models as needed to do the sub-agents to implement and test and review everything. Um so you have your orchestrator who you know is going to run the show. Uh you want to make sure that you haven't planned incorrectly. You want to make sure the plan isn't like um when the when the run's complete and you've already spent the tokens, you don't want the designer to come back and be like, "Well, that doesn't fit into our system." Or you don't want your infra person to come in and just be like, "We can't deploy that for some reason." Um getting your whole team to sign off on a plan, I think is more of a heavyweight thing, but that I imagine you would do in bigger companies. If you're not a small

35:02 startup and you're spending significant numbers of tokens on like really good quality implementation and checks and stuff, you want everyone to be like, "We agree this is the right thing to build, it's the right time to build it. PMs agree like this isn't going to flop in the market." Like I I think we'll get see more of this like formal approval up front before we hand everything off to implementation agents.

35:23 Yeah. And and so that's very much human kind of like agreement.

35:27 Yeah.

35:27 Do do you see that as almost like spec driven or do you see this as something that's kind of like a combination of other other pieces? Cuz I know you know

35:36 Yeah. Uh the team the GitHub Next team has has done some previous research before I joined on spec driven development, where they were like, "The spec's the software. Like how is it different?"

35:45 Yeah.

35:45 I think that can become more true with the formal verification methods. I think with natural language I'm a bit more skeptical cuz it's like what is code except for precisely stated, you know, like rules of how software works.

35:57 Mhm.

35:58 And the spec is like one level up from that, but people will argue like, "Well, that's what code is." It's the code is a spec. Um which I I kind of agree with. Um but yeah, I do I do feel like we're just inevitably moving towards towards spec driven development. Like how much is anyone really going to read the code? Again, as long as like the tests verify that it does what it says it does. Yeah, I I feel like we just move into interacting with these higher level artifacts. So these like plans and these like um after an completes a run, what we would currently do in like a PR description, but we obviously need new nouns and like new primitives there.

36:32 But, some sort of report of after your all your orchestrated agent and all your implementation agents have done some huge run, it's like you don't want a line-by-line diff.

36:42 Yeah.

36:42 You want a really high-level understanding of what happened, and and you can also have agents, you know, verify that final report actually reflects what the agents did. You can have many layers of checker agents all through this. But, you'll just be reviewing these high-level inputs and high-level outputs. And you won't have to worry as much about all the noise down below because we'll have figured out ways to actually make this quite rigorous and testable.

37:04 Hey, everyone. Hope you're enjoying the episode so far.

37:07 Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But, for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our

Software craft remains the ultimate differentiator

37:35 guests and build an even better product for you. All right, back to the episode. Let's talk about implementation and how we actually now build with agents. Um I I said I wouldn't quite you again. I lied. I think this is one of your quotes. In a world of fast, cheap software, quality becomes the new differentiator. Was that one of yours?

37:57 I feel like everyone said that at this point. I don't know if that was me, but I've definitely said it.

38:01 Okay, that's cool. That's cool then. We'll say it's one of yours. So, quality is notoriously hard to to be able to measure, I guess, and validate. Or even reward in companies as well. How do you get a team or an exec to to pay for I guess the craft that they don't see in a dashboard when speed and delivery is becoming I guess one of the one of the more sought-after pieces efficiency in in organizations.

38:31 I think it'll happen naturally because I think before you really had to fight for craft for the time to fine-tune something, but this was in the time where software was was compared to what we have now infinitely slow. Just you would spend weeks planning something and then like months implementing it even like a dashboard or something which now okay, you can bang out a version in an afternoon that sure is not going to be as like um verified that it's like good quality and there'll probably be lots of bugs to solve. But we're in the age where you can assign a bunch of like autonomous agents to just like run a bug fix program like go try and use the software on a loop find every bug file a bug fix it have another agent review it like this is like the software factory stuff is really solving a lot of this quality control QA.

39:15 And what is that what is that now?

39:18 Yeah.

39:19 It's just like chickens, right?

39:20 Yeah, it's all just chickens all the way down. Um I feel like craft yeah, craft becomes the differentiator, but I do actually think craft is more within our reach now. I certainly feel far more empowered as both a designer and developer like today than even 6 months ago that I can do much more ambitious things and to a much higher quality level than previous we could have ever done. It's just I just was not as capable. We would all just make kind of like okay software frankly in the past. I don't think quality's degrading from what I see around. You could you can look at like people who like vibe up slop of course it's bad, but really if you look at like the big companies it's not getting worse the software. It's actually getting bug bug reports are going down. It's like things are shipping faster. We're going through a weird transition period where where exacts are demanding more speed because that you have agent tokens you can just build stuff faster, but I think they're quickly going to realize like deploying agent tokens effectively towards like reducing bugs and also like leveling up

40:22 the quality of your interface and like being able to explore more options and like use a test them with more people and then figure out what actually works. It's so cheap to prototype these days. I feel like we get faster insights into stuff.

40:33 Yeah.

40:33 Um I mean, I don't know how you measure it, but I hope someone can come up with some quantitative measurements for like how the quality of software improves, but I have no doubt it's happening.

40:42 Yeah. So, if we if we look at what's speeding up and what's slowing down and where the bottleneck is, then the how something is actually built becomes cheap and it's probably largely defined by models and agents and and and and loops. Um the the the the I guess the the how we build something in terms of the more of the planning and the design

41:03 Mhm.

PMs guide the 'should we build it' while roles remain distinct

41:04 is is likely more human-led or or certainly human-signed off. Um and probably agentic assistance needed, required, used. Uh but one level above that, the should we build it? I guess.

41:19 Yeah.

41:20 And that that becomes the the harder thing because in a world where in a world where building becomes cheap and we can and everything is possible, what do we do? Who's equipped on a team to really answer that? Is that the engineer, the architect, or is it is it more leaning into the PM?

41:36 Yeah, I think it's the PM. I don't know. It's like I feel like PMs get a bad rap, but like I worked with some really great PMs where like they make far better decisions than any of the engineers, but we should do. Yeah. And it really is. When I say PM, I just mean someone who actually is like doing thorough research on the state of the market, on like what other products are available, doing comparisons, like when it's someone's full-time job just to care about like are we building the right thing in the right way?

42:03 They kind of do become the arbitrators of like what we should build. I think before you would have had designers and engineers more helping make those decisions because they could speak to feasibility more, But now that you you can be more ambitious and build more stuff, I feel like there's less chance for them to say no.

42:17 Yeah.

42:17 In the past, it would be like, you know, wouldn't it be nice if we could build this was like most product conversations. And now it is possible for a small team to build much more ambitious stuff that they feel like you could really can go after the ideal version of things and not always we had to make the compromise and build the crappy skateboard and not the car kind of thing.

42:36 Mhm.

42:37 Now I feel like maybe you can't, you know, always build the car, but you can build something like the bicycle. I don't know what the metaphor is here, but you can build more ambitious things and should let PMs kind of like guide guide the decision making of what to build.

42:51 And do you see how do how do you see those two roles like scope creeping, I guess let's call it scope creep given that we're talking about PMs. But um how do you see the the the PM role given you know, the development role engineering role is becoming more accessible.

43:08 Yeah, yeah, yeah.

43:08 Um and

43:10 there's this thing of like we won't need developers, the PMs will just like prompt it.

43:14 Or is it or is it that the engineers actually become more of the PMs as well? Is there Is there a merge happening there?

43:19 Maybe. I still feel like you need distinct roles. Like unless you're going to say the engineer's going to become really good at user interviews and market research and product comparison and strategy, I feel like that's too much to burden one person with. We always want to try and collapse roles in this industry, but the engineer still has to be in charge of like is this architecture scalable? Are we going to like fall over if 100,000 people like join at once? Like I think you still need someone who's in charge of owning the engineering and like keeping really high quality

43:47 Mhm.

43:48 product. And then that is a totally different job to user research, market research, product strategy. And I think it it's like would be a bad idea to collapse them because it's too much for one person to manage.

43:59 And they have I suppose different outcomes in terms of an engineer would be more focused on the quality and the and and the architecture and the design, whereas the PM would actually care much much more about the why should we build this and and and who's this for and focusing more on the user and the outcome of the product, yeah.

44:14 Yeah, and it's the same with designers. Like, designers aren't dead despite everyone going on about it. Like, the designer still advocates for like what's the best interface for the user? What do they need to do? Like, how do we make this look and feel the right way? And that is again, I feel like a completely different skill set to what the PM does and the engineer does.

44:27 Yeah.

44:28 Yeah.

44:29 Okay, so so let's kind of combine a couple of these and ask a ask a a little

The future belongs to ambitious prototypers with generous token budgets

44:32 bit of a teasing question here, which is if we could start our careers again, but in today's world, um you know, we're we're maybe just finishing university, coming into uh coming into the world of industry, what kind of role do you feel, given where AI is today, you would pique your interest in terms of in terms of, you know, you being you wanting to be a builder, what would your role be?

44:59 Oh, I mean, I've always been a prototyper, really. Like, from from early on, um trying to like build websites like as a as a teenager. I was always just trying to do like weird scrappy ideas. And I think I would have I I would love to be like 21 right now.

45:17 Um because because I would always come up with these ideas and sketch them out or like design them in Photoshop, but like I could never take them much further.

45:24 Yeah.

45:24 Now, I would just be flying. I would frankly, I would get myself into whatever a junior role at whatever company would give me tokens. Tokens are like the currency. I'll get paid nothing as long as you give me

45:35 Pay me in tokens. Your tokens, yeah.

45:36 Pay me in tokens.

45:39 And then just like build and I'm I mean, I am doing this now in the time that I do have. Just build as much as possible. I think the lesson I keep learning over and over is like whatever idea I come up with, make it more ambitious. Like, I've never in my life done things like build like an eval harness. And like I'm now just being like right, let's just try. I'm just going to plan it out with the agent. I'm just going to attempt. And in the attempting, I learn so much, right? And then failing and doing it for the fourth time over. I'm like learning so much each time. I would just get somewhere where it's like you have generous token budgets and you're just willing to to put out as many prototypes as possible.

46:13 Not really trying to make them that robust or not trying to turn them into products. I don't know. I think there's people who try to like take everything and be like how do I make this into a business? Maybe that's just not the kind of itch I like to scratch. I would just be trying to make as many small usable things as possible in order to learn. It just introduces you to the full spectrum of engineering. Like I have I always had to stay very narrow on front end and design because there's so much to learn outside of that. But now I can just be like, well, let's learn about what database I'm going to stand up.

46:41 Like let's learn about how I'm going to deploy this. Like

46:44 I'm I'm coming at it from the complete opposite angle actually cuz I I yeah, my background is much much more on the back-end engineering. So and I was always terrible at design and front end.

46:52 Now's the chance.

46:53 All of a sudden I I I'm like, oh my gosh, I'm I'm I just realized I'm a front-end designer. I'm a I'm a front-end engineer now. Um and it just unlocks, right? Now to ask the question which I hate to get asked but I love to ask other people uh which is the the getting the crystal ball out and trying to foretell how how the future will go. In a few years, let's say cuz I hate you know, years ago I would have said 10 years but in in a few years let's say let's say let's say six or seven years.

47:21 Okay, okay, hard. Okay.

47:22 How do you see that role that you just mentioned?

47:25 Yeah.

47:26 Um I guess not the entrepreneurial but the the the creative the the rapid application developer, the trying to trying to build those types of applications. How do you see that standing up to the test of time of just five, six years?

47:41 I would hope I would hope in the process of building lots and lots of stuff and being ambitious because agents allow you to. You would learn enough that you should be becoming very powerful with agents in a way that I think those of us inside the AI bubble forget how slow technical adoption is outside of it. Um like cuz we're going at warp speed.

47:59 Mhm. But if you go talk to most people around, they're like vaguely know chat GPT. Like now they're hearing a lot about the AI safety concerns. Like like the bubble like things are filtering out of the bubble. Um but still I feel like if you got started early and were very ambitious and really understood the capabilities of what models can teach you and allow you to do, you're still way ahead of everyone else. I think it's not like those people would become that their skills become like useless because the models advance so much. I don't see it going that direction. I don't think at any point models will be able to like read human society and culture and like make decisions about what the right thing to build is. I don't know if they'd ever get there. Not in the way a human really could. So it's more like a human who can like look around them and say what problems can I solve in the world is always really going to have something valuable that the agents can never have.

48:48 Why do Why don't you think an agent will be able to make a good decision on that?

48:51 I don't think they have access to the right inputs. I feel like they are not part of a culture in a society in a way where they can see what the problems are and also the fact they don't have physical embodiment. I know that's coming. People say it's coming soon with robots and cameras and stuff. But even so, having a robot present in a social situation changes it in a way that like they I don't know they just like can't go out in the world and be part of companies or or events or social groups in a way where they would see the same social problems we would.

49:20 So do you think it's not a it's not so much a model capability, it's a it's a it's a data problem.

49:25 Yeah. Yeah, they don't have the right data. And I don't think you could ever scrape it. How would you scrape? I mean you just think of like whatever these successful businesses, it's like they understood there was an unsolved problem for a group of people in a certain context. And I'm not sure how you get the data of like being in a certain place and realizing like oh these people really need like if for a simple version like there's no bakery on this street where lots of people would maybe buy from it or like I see there's some problem where like uh people don't have access to to like water in this town. Like that even that level.

49:57 Um you need a human on the ground with social connections. If you think of like how sales works, it's all like social relationships. It's all like who knows who. It's like who who convinces who to buy something. Models I even though they might good at social engineering, they're never going to have the social relationships you need to actually build a real business. Like solve an actual problem for other humans. Um that's the piece that models can't replace. But a human with that context, with like I've spotted a problem, I understand how I might solve it. I now have can leverage agents to actually solve that problem. That's that's actually going to be someone who has a lot of power in the world. And power in a good way, not in a like get all the money way, but in a like solve problems for people sort of power.

50:38 Yeah.

50:39 Maggie, we're at time. Um I wonder if we can get nine technical people to to record uh a podcast episode in 5 minutes. Just to just to wrap up a nice nice little ending to it. Um Maggie, this has been really good fun. I really really appreciate your time um and and insights as well and wisdom. Where can our listeners go to learn more about Chopin as well as uh GitHub Next?

51:02 Sure. So, githubnext.com. We have a little write-up on Chopin, just like a very early preview of what we're working on and we'll put more updates on there. Uh and then I have a website, maggieappleton.com, where I put up writing and research and everything I'm working on goes on either the GitHub Next site or my personal site.

51:16 Amazing. And where's where's where should people listen on socials for for or are you

51:20 Yeah.

51:21 Are you not uh Are you not on X? Are you on X?

51:23 I'm I live on X, unfortunately.

51:24 call it Twitter because I'm I'm on Twitter.

51:26 It was It was Twitter from the beginning. Yeah. I live on Twitter. I love Twitter. It had like a a dip, but it's still the best place to like AI news, AI people. It's great.

51:35 Amazing. Maggie, thank you very much for for for joining us today and it's been a pleasure to chat with you.

51:40 Yeah, thanks so much.

51:40 Awesome. And I hope you enjoyed that uh listening as well. And thank you very much for tuning in. See you next time. Bye for now. The AI Native Dev is brought to you by Tessell, the package manager for skills and context. Your hosts are Guy Podjarny and me, Simon Maple. Our producer is Tom Dawler. The AI Native Dev is not just a podcast, it's a community and we host monthly meetups at the Tessell offices in Central London. Visit tessell.io/community to learn more and I hope to see you there.