Agent Experience Is the New Developer Experience | Sean Roberts

5,284 words · about 21 min to read · YouTube’s automatic captions

0:00 Agent experience is an extension of developer experience. You know, developer experience, it's greater than the sum of all of its parts. It's not a question of whether or not agents are working on your codebase or on your platform. They are. The question is whether or not you're supporting those. But where we are today is this thing I call this mini oneperson band problem. So figure out what they're using. Figure out what's working, what's not. And the reality is if there is a gap in knowledge, it will confidently make things up or decide for you. To combat this, context files are where you need to spend your initial time. And if you couldn't simplify something, make it obvious. One practice that I really enjoy doing is like an agent therapy session. Like what should we do better to make sure that you're going to get this right next time? And specifically update the context files to make sure that happens. Ephemeral environments are something I'm a huge fan of. At Netifi, we do deployments like pre-prod deployments or preview deployments and they're amazing feedback tools for everything that's not specifically code related. As Bon Scott said from AC/DC, it's a long way to the top if you want to rock and roll. It's a much more

1:01 enjoyable ride if you're with your friends and with your band and you're going together. Before we jump into this episode, I wanted to let you know that this podcast is for developers building with AI at the core. So whether that's exploring the latest tools, the workflows, or the best practices, this podcast's for you.

1:26 A really quick ask, 90% of people who are listening to this haven't yet subscribed. So if this content has helped you build smarter, hit that subscribe button and maybe a like. All right, back to the episode. We have a second uh 30 minute keynote uh before we jump into main tracks and that's that uh next keynote is going to be from Sean Roberts from Netifi who's the VP of applied AI. Um now this is a this is a really interesting session because I think when we think about how we have built software over time and we think about our core kind of like ICP when we when we build that software we think about the user and the user being very often a human a developer or or just a consumer of our services but times have changed with the adoption of agents being you know so in our face with everything AI it's important that when we when we think about you know

2:28 providing these services providing these these tools, we think about agents as users of that as well as developers. And so this session is really going to kind of like compare the two and talk about how we can plan with agents in mind as our users. So uh please welcome on stage Sean Roberts.

2:49 Thanks for that.

2:53 All right. Hello everyone. Um as Simon said, I'm Sean Roberts. I lead the AI programs at Netlefi. Um, [snorts] and I've been working trying to make the web better, more secure, faster for a very long time. And especially at Netlefi, I've been really trying to figure out how we can provide the best agent experience that we can for our platform, but also trying to figure out how the web and agents can work really well together.

3:20 And so with that, you know, let's dive into the talk today around agent experience. And I don't know the makeup of the crowd, but if you don't know what this is referencing, that's not a me

Agent experience is an extension of developer experience

3:29 problem. Um, you know, take a picture, share with your friendly neighborhood AI agent, and they will tell you it's definitely in the training data. Um, but really what we're talking about today is how how agent experience is an extension of developer experience. Um, a quick raise of hands or or horns here. Um, who is a senior plus dev or in charge of a codebase today? A bunch of you. Yeah.

3:56 Horns. Yes. Uh, all right. Raise your hands again if you feel like you are a steward of the developer experience of your codebase today. Yes. Yes. All right. Well, thank you for your service. Uh, this is often a very thankless task. Um, but it's incredibly important because um, as we know, you know, developer experience really is the sum. uh it's greater than the sum of all of its parts, right? To have a great developer experience, you really need to have good documentation, onboarding, there's intuitive design that goes into it. It's an intentional act that you put into it. Like, but the reality is is if you're allowing people to develop on your platform or on your codebase, you have a developer experience. That does not mean it's good. It just means you have one. Just like every software has a user experience if they have users.

4:49 And with agent experience, it's very similar, right? Um we have hit this critical mass where it it's not a question of whether or not agents are working on your codebase or or on your platform. They are. The question is whether or not you're supporting those. And when we talk about agent experience, we do mean that kind of holistic experience that an agent has interacting with your system or your codebase.

5:14 And to illustrate this, right, um again, you know, when we think about developer experience, we think about these developers over here who are handcrafting beautiful error-free artisal code. It's great. Um and and the the quality of them going from here to here is really determined by the quality of the developer experience. And now we have developers who are delegating work to agents um to do parts of this on their behalf. And when we talk about agent experience, we're talking about this connection here. So the how well are they able to achieve you the same amount of of work. Um, and and this connection is all predicated on how well the agent experience is.

6:00 And I think it's very important to emphasize we're talking a lot about the agents experience here, but this is all to benefit the human developer on the other side of it. This is nothing to do with replacing developers. There are developers delegating work to an agent. And the more successful we can make them, the more successful we can make them, right? That's the whole point of this. It's also worth noting that when we talk about agent experience, we're we're talking about it today in in regards to a codebase, but an agent experience is this discipline that spans much broader, right? Um going forward, e-commerce is going to have to figure out how they serve their customers who are using agents to buy on their behalf.

6:45 um flights, how how they you know book flights and and airline tickets when an agent was used to delegate. Um so it's not specific to us but it's very important as right now agents are best at working with our code bases. Um but as Guy described they are have a lot of room for improvement. So one more time who who's in charge of the developer experience of their codebase here yet? Oh, a lot left's hands. A lot of people decided not to. But now, everyone of you holding your uh your hands up and probably everyone in this room really you're also in charge of the agent experience. It doesn't like maybe that used to be a question mark before. It's not anymore. Like it is a part of

Overcoming the one-person band fragmentation problem

7:32 serving developers is acknowledging the fact that developers are using agents to also work with your codebase. Um so you have this charge. Now it's very straightforward but um there there's you know having an agent experience and using agents isn't a new thing but where we are today is this thing I call this mini oneperson band problem like we've all you know have gone off our teams individually have gone off figured out what works for them and they've kind of culminated like kind of like their view of the world and like they have their whole setup right um and then two days ago Gemini 3 came out and someone with 10 years of experience using anti-gravity IDE said you should stop what you're doing here and then switch to this and so someone will and then now your team kind of you know expands its surface area and so forth and then you know there's pro there's going to be another tool next week and the next you know a few weeks and and we get this kind of sprawl we we have individually

8:32 went out figured things out and and we've built this this is the kind of mini oneperson ban problem that we see with with developer AI native tooling today. How many of you have gone to a band where there's like four or five of these playing at the same time or go gone to a concert there's four or five? Yeah, zero of you because they can't. And but you know because these are this is not a band. These are three individuals maybe trying to do their own thing. Maybe towards the same end but this not a band. Okay. Um [snorts] this is a band. So a again ask your AI.

9:09 Um this is a band they they have purpose-built um you know tools. They are all trying to come together for the same goal to to make um great experiences for for their customers, their audience, right? And that's really important because together they're able to do far more than any one person band, right? Um, and you know, for those who like the orchestra metaphor better, there you go. Um, all right. So, now what what is it that we actually need to do in order to to have a better agent experience um to to improve the developer experience of our code bases?

9:55 Um, and thankfully Guy did a lot of the examples and homework for me earlier, so that's awesome. Uh, but I'll dive into the the other parts of it. I I think what's really important for us to to make sure that it's clear on is there's no specific solution. I wish we could say this, if you do this, you're done. You know, pack it up, let's go home. Uh, but there's not. Agent experience is a discipline, not a specific tool. Okay?

10:22 And just so so if you're if you're thinking about, okay, I have to support agents, so I just need an MCP server. that's all I need. Check the box done. That's thinking far too narrowly about this problem. And whether it's MTP, ADA, ACP, these are all solutions that are kind of under under the umbrella of an agent experience. Um, but the other reality is we're all still figuring it out. We have a lot of ideas about what works today and then we're going to prove those wrong tomorrow and then we're going to keep iterating on it. But we're right in the middle of it. And so where should you actually start? So this is going to be my recommendation around the kind of crawl, walk, run stages that you can take on to to start either start your journey or expand it going forward.

11:11 So the first thing like if if your team is just a bunch of one person bands um or not doing any of this at all, start with figuring that out, right? Like I think that there's a hunch, yeah, my team's using AI, whatever. Like like we we're AI native. It's like, no, like like figure out what they're using. Ask them, figure out what's working, what's not. And and this doesn't have to be a big effort. Get a Google form, whatever, vibe code it, and just ask them what what agents are they using? What's working well? And to be clear, don't ask them the question that we asked them a year ago, two years ago. We're like, have you tried any agents? Like, no.

11:50 Like, everyone's probably tried an AI tool, but we want to know is what people are using every day or every other day. What's working? what's not. And it's important to understand that for for many reasons, but not least of which is the fact that there are lots of AI agents and tools and combinations of them. And then all the work that you're going to be doing to um improve the agent experience. This helps scope it much better. Just understand what people are doing. If they're using spec driven development, that might be a little different looking of an agent experience than if they're using some some other tool or they're really into stuff like lovable or bolt, things like that.

12:29 So, audit audit what people are doing. Figure out what's going on. Um, and then start to build your internal community, right? All that really, you know, has to all that is really involved in doing that is getting them together every, you know, infrequently, but get them together to talk about the issues that they're running into. Talk about the tools that they're using. Um, if you really want to learn something cool, come up with a problem or take something some take some ticket off the board and say, "Hey, I'm going to solve this together and I'm going to watch you solve it your way. I'm going to solve it my way and we're going to we're going to learn together." And that's some version of of pair programming here. Uh where there's also an AI agent involved and that is so illuminating in terms of like how what people understand differently about the different uh the code base how they approach the problem and and it will also start to build up your internal community um to to build towards an agent experience.

Context files are the first line of defense

1 Highlight · Context files prevent confident model hallucinations

13:26 So taking the next step forward, right? Uh you know the agents that you need to support now you need to help them be better. Guy did an a fantastic job of explaining this for me earlier with with with some data around backing up that uh the LMS themselves are not sufficient in terms of understanding your codebase. Um and and the reality is is if there is a gap in knowledge, it will confidently make things up or decide for you. It's not always made up false. It just might just be totally different. Like his example of picking a theme that was not the right theme um to combat this, right? Context files are where you need to spend your initial time, right? Um either you know depending on the tools, right? agentsmd files, steering files, cloud code, uh, cloud files, cloudMD files. Um, these are going to be very important um, nuggets of information. And you know, some rules of thumb that I think about

14:26 is if uh, if you could if you couldn't simplify something, make it obvious, right? So many code bases you see like, okay, to run a test suite, it's this some magical command that you got to run. That's not obvious. If if you have those, make sure that they're in your context. Um other areas are like if you have a decoupled architecture. So you have a front end um that's decoupled from your backend which might be decoupled from your your data layer etc. It doesn't necessarily know those connections and you need to make those clear or it will make those connections it it'll make them up every time. Um [snorts] and that's going to be incredibly important.

15:09 Um and and and especially whenever you get into this world of as people are um iterating on this and they start finding that hey it's it something that we thought

15:22 was obvious wasn't it's going to keep making those things up and adding those into your um uh agents files and and steering files is going to be incredibly important. The next thing I mentioned um is is the core dependency uh documentation. I'd mentioned the core dependency of the the front end has a dependency on this back end. It's over here. Here's how you use it. Here's how you find information about it. And that's incredibly important, but there's also this this problem of um dependency uh documentation and like third party modules, things like that, even first party separated modules and and and guy went in uh a lot of detail about that.

15:59 This has been bothering me for so long. Um because you know the reality is especially with you know brownfield type of projects is I I have developers or so the model is trained on version one of my API the latest version is version three but all my developers are using version two and so like it messes it up nine times out of 10

16:22 [snorts]

16:22 um and and the the tooling for getting this right isn't great today. I was actually, you know, writing this, I was very stoked to see Tesla's, you know, registry and things like that that they're doing. Um, and so I'm gonna explore that more. I think that's very exciting. I definitely agree that there's actually, um, more onus needs to be put on the open source community to deliver these things. The way I kind of see the Tesla registry is like, well, if every uh, dependency had to have its types hosted somewhere else, right? like really the types, the documentation for it really needs to be there. But we also have the second problem where especially open source repos there's people who build on them and then there's people who use them and those are different documentation and so like even it still gets a little bit harder and harder the the more you try to optimize but again we're all figuring this out together so that's fine. Um if if you use context 7 that's also something that uh people people use. Uh,

17:23 I've I've run into issues with it not doing very well on the versioning side of things. Um, but regardless, if you have a key dependency that's a third party module, make sure it's in your context. Make sure you're using it right, especially if it's things like data based databased um uh because there's it's just a such a dense topic and tool set that um it ends up getting pretty confused with those types of tools. So make it clear and especially for very large code bases um the concept of knowledge graphs um there's many different implementations of them cognition released uh deep wiki and and code maps recently but it's this you know the reality is is if you have a really large codebase and you're asking it to do things every time that you try to ask it to do something it's got to kind of figure it out on its own and the larger the codebase the longer that takes the more expensive it is the more likely is it's going miss parts or get confused. And this is where knowledge

18:23 graphs really come in where you can kind of pre-roduce this kind of understanding of the overall architecture, how things relate

Establishing continuous feedback loops and agent therapy

2 Highlight · Agent therapy sessions for continuous improvement

18:31 to one another and summarize those. And so before it tries to do things, it consults there first. And those are very helpful for larger code bases. Uh feedback loops are going to be incredibly important. Um, so if you're if you're already getting your internal community together, working with them to say, "Look, whenever we're having an issue, if if it if a human had to manually intervene in an agent uh agent's workflow, get that documented.

18:57 Get that into context." Um, if you're talking to that senior engineer who's been there for 15 years and has all the tribal knowledge in their head, if you every time you have to consult them, if that doesn't go back in documentation or context, I don't know what you're doing. But like that that's this internal feedback loop. You have to have some forcing functions to make sure that your agents understand because unlike you, they they can't go ask uh necessarily that that uh that tenure developer who has all the information in their head.

19:28 So establish feedback loops with your internal team. One practice that I really enjoy doing is I have this MC MCP where it's like, okay, if the agent's really going off the rails here, um, I have it look back and say, what could we have done better here? It's like an agent therapy session. Like, what what should we do better to make sure that you're going to get this right next time? And specifically update the context files to make sure that happens.

19:55 And then especially like once we do that um assuming it makes sense at the time having it try again and then go from there. Uh I have a lot of good results with that as well. [snorts] I'm sure most of you if not all of you

20:08 are using AI review tools in in your um uh code bases and your CI pipelines and such. the so I won't go too deep into that but like the one thing that I think is an addition on top of that is not only having your your uh AI review tool figure out what's wrong with a thing but also suggests when there are patterns that were missing from your context files that would have prevented this from happening in the future. So like this is you're getting towards the step of trying to automate some of this feedback loop.

20:38 Ephemeral environments are something I'm a huge fan of. Um, you know, at Netlifi, we can we we do deployments for like pre-prod deployments or preview deployments. Um, and we can do as many of them as as the agent needs or as the developer needs and they're amazing feedback tools for everything that's not specifically code related, right? Right? So if it's like the style of a thing or the layout of a thing, if it's just making stuff up, you can both have the the human review these things as or as well as you know, if you want to give access to the the agent to review these things on your behalf. These really and then combining these things. So an agent figures out, okay, I messed up the layout here. I'm going to go fix it. I'm also going to make sure I document this in the context. Now you're really starting to see a this full end to end picture. And this is ultimately what you kind of end up with with with a good agent experience and feedback uh life cycle. So developers can delegate works to to to the agents. They generate the code and this again this being the the

21:39 agent experience that they have access to here. They are better at doing this because the feedback loop of making sure the context is good um uh appropriate um for for the jobs that they're doing and so on. they can, you know, you have your your reviewer agents figuring out, hey, this should have been done differently. I'm going to make sure that they know that going forward. Um, I can also make sure that they can review the the actual changes, not just the code, but also the the output of that code and and then make suggestions for doing better going forward.

22:14 Okay. And as we get into the um

Scaling with shared subagents and centralized configs

3 Highlight · Offloading work to shared markdown subagents

22:18 you know you're really advancing your your your the agent experience of your codebase um shared sub agents is a really cool unlock. So shared uh sub aents these are going to be like guy mentioned earlier um these are as simple as just markdown files of defining hey this is a specialized agent because by default all these agents are generalists right and as we know with full stack developers they can get the job done but a specialist is probably going to get it done a at least a little bit better if not completely better um these are your specialized sub aents you define them in a markdown file and then so you can say okay This is our methodology for design.

23:00 This is our meth methodology for copyrightiting. It has our voice and tone in it. This is our data migrations sub agent. This is they are specialized at doing this. And what's going to be really um the the big difference that you're going to see here is that when you ask it to do something, you have sub agents. It's going to say, "Oh, actually it's better for this one to go do it." And it will kick that one off. And it has specialized contexts. Now imagine you had five specializations that you wanted to really hone in on and tune and you had to throw all that context into a single cla MD file or agents MD file.

23:38 That does not work well. And that's kind of why we have these now. We figured out that that doesn't work well. You overload the context. It gets confused. It starts to steal attention and and things like that. So this is uh shared sob agents are are very important and and and really allow you to have a shared way of doing things across your code bases and across um your different developers. You have your context files, you have your uh steering files, and now you have uh sub aents files. You're immediately going to run into how do I distribute

24:11 these things? Um because as soon as you have more than one repo as as guy was mentioning these things are are you know there there are things that are repo specific that always should be in the repo and then there's going to be things that are global. Your voice and tone for your brand that's a global thing. Um your brand styons all that stuff global. How you might you know do reviews all those things are global. Um, but how do you share that across your your developer base, which might be, you know, dozens or, you know, potentially thousands of people? Um, and my my preferred methodology personally is to centralize these things into uh the globals into a shared repo and then having a CLI that people can run to update them and do that those things. And um I also learned through guys talk that they they have this capability on the Tesla um platform as well. But I think that once you like this is a good problem to have. Okay, we've defined rules. They're working really well in our code bases. We

25:11 actually need to propagate this so that we're working better across multiple code bases. Um, and centralizing these outside of a single agent as guy also mentioned is incredibly important, right? Because like if it doesn't like like you did this audit step one, unless they all said I only use cloud code and they only all they all only use codeex, then picking one agent is a recipe for disaster. Also, if they're all only using one agent, you you're probably already setting yourself up for a future problem. Um but uh yeah so so centralize these um if you don't use a platform like Tessle you can use centralized GitHub repo that also has a CLI to run to to do the syncs and I think really on the bleeding edge

Running evaluations on agent experience

26:01 of of these uh optimizing your agent experience um we we we have these opportunities to run evals on the AX of our of our code bases and if you haven't used or done any um agent evals all they really are are putting an agent under certain conditions asking to do certain task and evaluating the performance of it doing those things but have you ever considered taking now cloud SDK or codeex SDK throwing that in eval saying hey try to do some stuff on my on my repo and then help me figure out what I should tell you to do that better you're doing these things wrong um you know guy and the Tesla team have done a lot of this on the open source side of things, but this um doing this for your own repo is incredibly uh important, incredibly validating.

26:51 It will really help you close the gaps between what's just clear in your documentation today and what's still missing. In addition to helping you fill gaps, it also helps you solve the the the one-way ratchet problem or it's a Asmov's law of bureaucracy. I totally knew that before yesterday. Uh um which which says right like um laws and regulations once they're enacted monotonically just grow. They never contract and they never substantially get repealed or or or reduced. [snorts] And we are seeing that happening with config or with with context rules and context as well. They're just growing.

27:34 We're like ah I missed this thing. I'm going to keep adding to it. So, but how do you safely feel confident in peeling back some of those requirements and how do you do how do you um you know you fix this issue but you cause other issues. How do you feel confident about not doing that? And I think eval for for the agent experience is going to be incredibly important. Um cool. So we covered a lot of ground in terms of you know the developer experience. We we all understand that's our charge now. We're we're we have been in charge of the developer experience for our code bases, but agent experience is no longer optional. That's a part of what we have to deal with now. Um [snorts] not because we want to support agents, but because we want to support our developers who use agents. Okay.

28:21 and making this I was really um stoked on on using the band a whole metaphor because I think what's really important about bands is the size of them doesn't uh determine the success of them, right? It doesn't matter how big your band is, but what does matter is how well they can perform together and and produce uh great outcomes. And so when we think about all the things that we've heard with like, oh, if you use these tools, you'll be a 10x developer. So now we get one one person band and this other 10x developer wants to be their own oneperson band. But what's really magical is is having um a 10x team, right? And that really comes when when we're all sharing knowledge and we're all working together. Um, I as as the the as Bon Scott said from AC/DC, um, it's a long way to the top if you want to rock and roll, but I would amend that to say it's a much more enjoyable ride if you're with your friends and with your band and you're going together. So, with that, thank you all.

29:23 Thank you for coming here. Drink some water. Go make good Adrian's grace.