What OpenAI, Stripe & ElevenLabs Devs Do Differently Now | AI Native Dev

13,858 words · about 55 min to read · YouTube’s automatic captions

0:00 So much software is yet to be produced that like we now have the option to do because implementation is so cheap.

0:07 I am talking about why CICD is dead. When you think about how engineers are using traditional CICD tools, those were built for humans pre-cloud and you get this massive volume stuck in PRs that we never saw with humans.

0:23 Now we're seeing about 1300 pull requests without human assistance be generated and land per week.

0:30 You can't have so many different agents taking versions of the same code and merging them.

0:37 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts Guy Pajani and me Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

1:01 Back in November, we hosted the first ever in-person AI Native DevCon in New York. This June 1st and 2nd, we're bringing it to London. It's two days built for AI Native developers and engineering teams.

1:14 One day full of hands-on workshops and one day full of practical talks on agent skills, context engineering, agent orchestration and enablement platforms, and how teams are actually shipping AI in production.

1:27 Join us at the brewery in London near the Barbican for all of that plus networking, parties, giveaways, and a room full of people building the future of AI Native development. You can also join us from anywhere in the world via the live stream. As you're listening to this podcast, you get 30% off your ticket with code pod30. Just head to ainativedevcon.io and we'll see you in London.

1:54 Hi there, Simon Maple here and on today's episode of the AI Native Dev Podcast, we're here at AI Engineer London. This is the European event for AI Engineer. Probably one of the most premium events here in the AI Engineer space. So, just take a look at the sheer number of of sponsors and the quality of the folks that are going to be here in the Expo Hall. And the Expo Hall, of course, is right behind me here. We have a whole bunch of wonderful people. We're here to talk to attendees as well as folks who are here speaking and also on the Expo floor. We'll hear about what's new, what's happening in the keynote, and also what people are here to take away and learn. Let's go in.

2:36 So, joining me is Steve Kaliski, an engineer at Stripe. Steve, how are you?

2:40 Good. Thank you for having me.

2:41 Ah, pleasure. Enjoying AI Engineer so far?

2:43 Yeah, I love it. It's really awesome to just like you're online so much reading about what's happening, but to actually see the people here is super cool. So.

2:49 And what are the people here like, right? Because this is one of the premier events in in the AI engineering space, AI development space. Um, do you get a higher class, a higher a higher level higher levels of adoption of developers, you know, using AI in their environments? What kind of conversations are you having?

3:06 Yeah, I I think what's interesting is like you know, every day like there's a new announcement about some new models, some new technology, right? So, everyone here is like really operating at the leading edge, right? Everyone here is like really trying everything the second it comes out, everything that's new. So, it's really interesting to sort of like get the frontier perspective of how people think about new tools, new technologies, and so on. So, it's really exciting.

3:26 Awesome. So, tell us a little bit about Stripe and how Stripe is adopting and using AI in your products.

Stripe automates development with internal agents

1 Highlight · Stripe's internal coding agent lands 1300 PRs weekly

3:31 Yeah, sure. So, uh Stripe is using AI in a few ways, right? So, helping build Stripe, helping our customers build their Stripe integrations, and then also generally providing, you know, economic tools to agents and and things like that. So, I can go off uh through all three quickly. So, the first one is how we're building Stripe with AI. So, we have an internal coding agent called Minions.

3:51 Okay.

3:52 And uh you know, sort of like other coding engines you might have seen but it's very tuned for Stripe specifically.

3:57 So it's a model and an agent?

3:58 Yeah, so we're using you know common models like Claude and open AI and so on. Our harness is forked off of blocks goose and then we run it internally within our pre-existing development environment. So you know maybe I see a Jira ticket come in or feedback from a customer I can click a single emoji in Slack it'll spin up an entire replica of Stripe you know as I sort of normally would have done but it'll now run an agent loop off of that prompt and then it will use all of our internal docs and tools and CI and so on to try to resolve that prompt.

4:29 Oh nice.

4:30 So now we're seeing about 1300 pull requests without human assistance be generated and land per week with the only intervention is sort of the typical code review process.

4:40 So is there still I was going to say is there still human review?

4:42 reviewing all the code yes.

4:42 Yeah yeah do you think they'll get to a stage where human won't be in the loop

4:45 for a bunch of these like lower priority less critical

4:49 Yeah so I I I think there are certain changes where um there's not a business logic effect right? So you can imagine something as simple as a typo in the docs in general might already be something you'd be comfortable with landing today already right? Um so I I assume we'll start that base case but then increasingly we'll move up towards you know features that have a monetary impact where the sort of risk reward blend of it landing faster is probably not in excess of having a human review it to like double check that it's right.

5:20 Yeah yeah.

5:20 So I think there'll be a balance where some things land you know automatically in sort of the same way you know even with GitHub today with Dependabot doing upgrades and checking at CI I could probably just let that land.

5:32 There are things that would you're you're happy to accept because it's lower risk.

5:34 But then on the flip side like you might not want that to land right? So the Axios and you know all those other kind of security things that have recently where you like might not want an automated package update to go through so it'll It'd interesting to see how that plays out, but I don't think anytime soon.

5:49 Yeah, yeah. And I'd love to hear a little bit more about your the the environment that you use there to to to to build within Stripe. How do you How important is context as part of that whole structure?

5:59 Oh, I mean, it's everything, right? So, you know, like you know, like the literal context, like Stripe's been around for, you know, 10 plus years, right? There's 10 plus years of context that doesn't exclusively live in the code base, right? So, um you know, any kind of integration with a payment method or network or um some sort of uh you know, business logic around how we treat funds, like that has a legal and uh compliance and all these other kind of contextualizations that we try to manifest in the code base, but also live in documentation and live in, you know, our knowledge base and so on and so forth. So, to be a human engineer at Stripe, I have to read all that content.

6:35 So, if we want our agents to be able to code effectively, needs to have that context as well.

6:38 And how do you manage context? Cuz obviously there's there's so much of it, and it's not just, I guess, context about the code, but also, I guess, your policies, your ways of working, how you want code to be to be written, and the CI/CD process, all that kind of stuff. How do you How do you manage all of that? Cuz there must be so much.

6:53 Yeah, so I I think this is part of one of the advantages of of us having built this kind of thing at home, right? Which is, um you know, the engineering you do at Stripe obviously looks different than the engineering you do at another company or at a startup or totally different uh business domain. So, you know, may- maybe 80% of coding at Stripe follows some, you know, uh blessed paths, right? Maybe I'm making an API change or change to documentation or change to introducing a new currency or so and so forth. And because we can sort of pattern match to, you know, one of those pathways, we can make sure that we're just introducing the appropriate context to really nail that.

7:29 Yeah.

7:29 So, the context you need to make a docs change looks a lot different than the context you need to make a cross-border, you know, uh money transfer kind of thing, right? So, um you know, us understanding the type of product we're building and the type of changes we're making, lets us, uh, you know, thoughtfully put in context in advance. Instead of saying, "Hey, here's like the entire thing. Good luck." Uh, which would be impossible for a a normal person to do that. So.

7:53 Well, listen, let's wrap up, but I want to ask you one final question. What is it you want to learn here at AI Engineer this week?

7:59 Yeah, so I I, you know, for me, you know, obviously there are a lot of people who are Stripe users, so I'm excited to talk to them, just to get feedback so we can make our product better. But, I also think, you know, to to that first point, like everyone's working at the bleeding edge, right? And it's like impossible to keep up. So, being able to hear different perspectives and how people are thinking about the tools that are coming out, and how we can best use them, and those kinds of conversations seems very useful.

8:18 Amazing. It's been wonderful chatting, and I hope you enjoy the rest of AI Engineer.

8:21 so much. Appreciate it. Appreciate it. Thanks.

8:24 Well, we're lucky enough to chat with Madison Ford, now one of the speakers at AI Engineer here in London. Madison, welcome.

8:30 Thank you so much. Happy to be here.

8:31 So, tell us first of all a little bit about, uh, who you are, what you do, who you work for.

8:35 Yeah, so I am an investor. I'm a partner at NEA, which is, uh, one of the biggest and oldest VC firms. We do have a London office, but I am based in San Francisco.

8:46 Nice.

8:46 Um, although I do have a lot of investments in Europe. And I actually am, uh, former AI researcher. I used to work at Meta, and led data and AI teams. So, I have a little bit of a unique background that more aligns with the people here versus venture. Um, so yeah, very excited to be here.

9:03 Awesome. And what is it you're talking about here at the conference?

9:06 Yes, so I am talking about why CI/CD is

Traditional CI/CD breaks under agent PR volume

2 Highlight · Traditional CI/CD breaks under agent-generated volume

9:09 dead.

9:10 Oh my gosh.

9:11 That's actually the reaction I get from

9:14 Um, but no, it's a really important next category for us to think about in agentic software development, because, um, today everyone's thinking about agent native engineering. I mean, I actually have a an investment in that space that's really exciting called Factory. They're also speaking here, uh, this week as well. And when you think about how engineers are using traditional CI/CD tools, those were built for humans pre-cloud, and you get this massive volume stock in PRs that we never saw with humans.

9:45 Humans were submitting two, maybe three diffs a week, and now it's just thousands, maybe tens of thousands, and you start to get to a place where your code base is merging them is just it's impossible. You can't have so many different agents taking versions of the same code and merging them. So, we really needed to be more inference-native.

10:06 So, it's almost like is it is it replacing our existing code base kind of software?

10:11 I think that's where we're headed, but I think right now it doesn't quite look like that. Right now, I think the best approach is to have a complimentary layer where you are speeding up the build times, development times. There's a company called Namespace that I invested in, and we're actually speaking together on this. And Namespace helps to speed up your GitHub action builds, Docker, anything in that ecosystem so that you don't have to be sitting there waiting for your agents to deploy.

10:40 But, I do think over time this will likely replace GitHub and other CICD tools that are just not built for this infra.

10:47 It's interesting as well, because obviously it's such a it's such an important part to GitHub's business to you know GitHub code being such an important thing and those workplaces. Going to be very interesting to see how or if they're even able to change fast enough to

11:03 Well, I you know, there's a lot happening at Microsoft these days. If someone takes a look, we've been seeing a lot of reorgs, a lot of leadership from GitHub is are leaving. Like Thomas Dohmke. Yes, and I'm an angel investor in Entain, so big believer in that.

11:18 Entain I mean, that's going straight out and saying we are going to be the new GitHub.

11:22 Which Entain really makes sense. I mean, Microsoft's had the opportunity for a long time, and they haven't run with it. So, like why not have a new tool that actually is built for the cloud, built for agents, capturing new code and and creating a better developer experience. You know, GitHub has been the the friend of developers for forever, but it's not that user-friendly.

11:44 Yeah.

11:44 It's not that agent-friendly.

11:46 Specifically with the it's the agent-friendliness, right? The agent experience.

11:49 And then Harness, too, right? Harness, Circle CI. I mean, GitLab, like so many of these tools are just not really scaling to agent-native software.

11:58 So, let's go through a bit of your portfolio. Let's let's Let's hear. What would you say is the startup that you're most excited about that people have never heard about?

12:06 Okay, love love that. So, I mean, I have to give a shout-out to Namespace, but I already talked about them. That is probably one that has the least maybe brand awareness, but so much developer love. I mean, Warp is built on top of them. Sierra, Decagon, OpenAI is a customer. I mean, they they have so much love. But, I'll put that aside. What's the one that has no one has heard about yet? I would say probably Golden Analytics. This is actually super interesting because Golden is just launched yesterday. It's a stealth company. And the idea is to bring the golden era of data and analytics to builders. So, this is the first tool that was truly created to be end-to-end data autonomy for everyone versus just the data team.

12:59 And they're really trying to blend data autonomy, as I was mentioning, analytics and semantic understanding of your data. But, then the third piece, which I think is the most compelling, is design and storytelling. So, how do you actually ask questions of your data and get full end-to-end stories where your data is being You're finding the right data versus just using the data you have. So, I think it's going to be a really big unlock because I'm very bullish on structured data getting its moment in the vibe coding era.

13:26 And that name again, sorry. Go go go go.

13:28 Golden Analytics.

13:28 Golden Analytics.

13:29 Just launched yesterday. Check it out.

13:31 Amazing.

13:31 Really cool product. They have a big waitlist, so.

13:34 Yeah, yeah. Excellent. Sounds good. Uh um what are you here at AI Engineer to learn about? What are the things that you want to unlock?

13:41 Yeah, so I mean, I I'm always thinking about as I was mentioning before, kind of category foresight. Where when you look forward in 1 to 2 years, does the existing infrastructure start to crack? And that's where I want to be understanding both for my portfolio companies, um but also for the enterprise heads of data and heads of engineering I work very closely with. Um so I'm looking for where is this not going to work? Where are things going to break?

14:10 Um who are the thought leaders of that new era? And I mean, just to get deep into some of it, I think foundation models for tabular data are an area I'm very interested in. Um domain-specific foundation models. Um harnessing engineering I think is already a huge topic, but just trying to understand how that sits in the ecosystem alongside OpenAI and Anthropic. Um so yeah.

14:33 do you think when talking about those foundational models, do you think do you see obviously the billion-dollar question a while back was around general models versus specific models?

14:41 Yes.

14:42 And we saw companies like Poolside, a little bit quieter these days, but Poolside they you know, they were created they were they were putting huge investment in this massive massive uh amounts of money that they were valued at. Do you see that now as the unlock, particularly with my awesome things like that being announced as like killer models which have been trained very specifically? Do you see general models fading away for specific models?

15:06 I think general models will always be very helpful for orchestration and routing to other models. And maybe that looks like mixture of experts where there is certain expertise on small models. That's actually a part of my bet with the tabular data space. For For I think data's a really good example of one where foundation models will not be able to come in and build precise structured data, but the tabular data architecture might be one that a large language model is able to route to.

15:34 So overall, I'm actually very bullish on small specialized models for different domains. I don't think coding is necessarily one that will be in that, you know, specialized domain, but I do think there's a big opportunity for harness engineering there. Some of the more specialized areas I'm excited about are material science, not possible with a large language model. We have a company called Cuspaider, it's actually based in London.

15:59 So sovereignty and language specific models, so like reflection in the US, Sakana in Japan, Mistral in Europe, I think are very interesting. Um And then we have another one that's foundation model for CAD, so think about mechanical engineering and product design. Very difficult problem. This is not something you can just release a, you know, large language model or even a coding model to do, but it's very precise for that use case that's built for the multi-modality.

16:31 So I definitely think there are huge markets that haven't even been touched by AI yet with small specialized models. We're being very thoughtful about betting in a lot of those categories right now.

16:41 It's been wonderful chatting with you, Madison, and enjoy the rest of AI

Small on-device models enable custom fine-tuning

16:44 Engineer and good luck with the session.

16:45 Thank you.

16:48 We're here with Omar Sanseverino, who is the developer experience lead at Google DeepMind. Sounds amazing. Tell us a little bit about what the role is and what you do at Google.

16:58 Yeah, of course. So I lead all of the model launches at DeepMind, so Nano, Banana, Bio, Gemini, and of course Gemma. So last week we released Gemma 4, which is the latest addition to the open model family of models. So yeah.

17:10 Amazing. And tell us a little bit about what each model is trying to address. What's the Why would someone use one of those specific models?

17:17 Yeah, of course. So, uh so there are models that are for Gen Media. So, we just released Lyra as well for music generation. We have VEO for video generation, Nano Banana, which is pretty much for image um editing and image generation. And then we have Gemini like the core famous Gemini model, which is a flagship model if you want the best capabilities, the most knowledge, all of these gigantic, high-performance coding capabilities. And we have Gemma, which is as I was mentioning like the open models that we have released.

17:47 Last week we did Gemma 4, which is the most capable model that can fit in a consumer GPU. That means that these are models that you can run in your own devices. And that can mean your own desktop computer, your own laptop, and even the smallest Gemma models can run in your own phone. So, trying to pack as much intelligence per parameter as we can.

18:06 Amazing. So, so the one the key feature of Gemma is its size.

18:10 It's its size and also the control that it provides you. So, you can run these models in your own hardware. And you can even do something called fine-tuning, which means that you can modify the models for your own use cases. So, let's say that you're working in a setup where the information has a data type that is very tricky like legal setups or maybe like a specific medical type of data that must proprietary models have never seen before. So, you can pick Gemma and you can modify it for your own use case. And it and it can work quite well. And then you can serve that model in your own infrastructure and your own devices.

18:40 Amazing. So, you did a launch just recently. Did you say yesterday or this week?

18:43 Last week.

18:44 Last week. Last week.

18:45 So, last week on Thursday, so that was 6 days ago. And now we have over 10 million downloads already. Yeah, yeah. And over 1,000 Gemma 4-based models built by the community, which

18:55 I suppose you had launch partners and things like that. What what what are people starting to do with Gemma already?

18:59 Yeah, so what is critical for us is that when we release new models, people can use the models with whichever are their favorite tools, right? So, we collaborate with Ollama, with Hugging Face, with Unsloth, with Llama.cpp. So, pretty much no matter which is their tool, people should be able to experiment with the Gemma models. And it's the same strategy for Gemini. We work with LangChain, Llama index, and so on. And people are building very exciting things. So, people are running the models in their phone, they are doing even agentic stuff within the phone. So, as part of the launch we did an application called the Google AI Edge Gallery. You can install that in iOS, you can install that install it in Android as well. And it allows you to do things such as adding new skills, even like controlling your phone, like turning on the flashlight and doing like different UI device control things, which is quite exciting.

19:46 Interesting, interesting. I'd love to lean the conversation more into context now as well, which we see it feels like context always opens models up in terms of getting the to get the best out of models, you need to almost like customize in context and skills and things like that. Seem like a really good way to provide that information about how to customize your usage. With the different models that you just mentioned, would you say context is as important with with various models or would you say there are specific use cases where it's it's so important and others where it's less important?

20:15 Yeah, yeah, that's a great question. So, let's talk about Gemini and Gemma. Of course, Gemini has a much longer context window. You can pass videos of 1 hour, you can pass very long documents. In the context of Gemma, these are small models designed to run on device. And the issue is that the context does consume your GPU, so you cannot just provide like a huge context without having to burn much more compute, right? So, for us that that part is quite important. So, the smallest Gemma models support context of about 220K. The larger Gemma models support 256K, which is nice for on device, but it's still like much shorter than what you could try with Gemini.

20:49 So, the context curation is actually far more important to actually

20:52 important

20:52 make sure the context is small enough for the model to actually use correctly.

20:56 And the other part is also thinking efficiency, right? So, there are different open models and how many tokens or how much thinking or reasoning the model needs to do to get to an answer can vary per model. So, that's something that we also look quite a bit. So, making sure that when models are reasoning, they don't generate like just like a super long chain of thought, right? And that they can actually generate like a dense like highly useful highly like dense kind of information in the context.

21:21 Yeah, yeah. So, we're at AI Engineer here in London. Great to be here. What are the things you want to take away from the conference?

21:28 Yeah, so the first part is to just meet with the ecosystem like all of as I was saying like for us it's critical to have ecosystem partners to collaborate with them to contribute to the open source ecosystem. So, it's great to meet with all of them in person. And then the other part is just like listening to all of these exciting use cases like what people are building. The last time I was at AI Engineer conference was 1 year ago in New York.

21:49 In Paris or is it Paris? Yeah, yeah, yeah.

21:51 Yeah, in New York. New York. It was a yeah, in the winter of last year. Just in the last year so much has changed, right? Like it's so exciting now to see like what people are building even like compared to 6 months ago or a few months ago. So, now I want to see where we're standing 6 months from now, right? It would be quite exciting.

22:07 Well, there's a lot of exciting things coming out of Google Gemini as well and very very keen to hear what's coming and what's what's what's going to be in the future.

22:14 Yeah, yeah. For us what is critical is to also like build based on the feedback from the community. So, pretty much for Gemma 4 for example, we we gather all of this feedback from the last year. So, people were for example asking for function calling and agent capabilities, a better license. So, we changed our license to be Apache 2 actually as well. So, pretty much this is a kind of opportunities also to gather as much feedback what we're doing right but what we're doing wrong, right? And how we can iterate and we're executing as a startup as much as we can. So, trying to release models, release new capabilities, release new tools as quickly as possible to get feedback and iterate based on that.

22:48 Amazing. I'm actually having a wonderful chat and enjoy the rest of AI Engineer.

22:52 Yeah, thank you so much for having me here.

22:53 Cheers. Now, I'm here by the 11 Labs phone booth and I'm going to go in and try it out. It's going to basically going to ask me a number of questions about British AI. Let's see how I do.

23:03 What classic party That was Oh, uh 2016. Was it chess? We're back here at the Eleven Labs booth and joining me Boris Starling, who is a growth engineer uh at at Eleven Labs. What is a What is a growth engineer?

23:25 Well, uh growth engineer is a little bit of everything, little bit of marketing, of engineering, and a little bit of fun projects like a booth.

23:33 that booth that booth was the you were the creator of that. You built You built under the covers. Tell me a little bit about the technology behind that.

23:40 So, the hardest part was actually reverse engineering the actual phone. We used cloud codes to reverse engineer the low-level protocol of it, and uh then we connected it to Twilio, and uh connected Twilio to Eleven Labs agents. And that's how it works.

23:56 That sounds cool. And it really was a phone line. You would You were actually calling uh uh made a connection, and then you were there was an agent at the end that was that was answering back. It really was Michael Caine, right?

24:06 It is. It is. We have

24:07 the license his voice.

24:09 Amazing. Yeah. Cool. Cool. So, tell us a little bit about Eleven Labs, what Eleven Labs does.

24:14 Well, uh we started as a text-to-speech company. Right now, we do everything. We have a AI agents platform. We have a creative platform with a bunch of uh tools for creators. Um and um yeah, we do we do a little bit of everything now. We even have a music product now.

24:31 Oh, amazing. And actually, one of the things that I I heard Mati uh who Mati who's on the who's on the uh podcast, uh co-founder of Eleven Labs, uh one of the things that he said was uh voice will be the way that developers, engineers will engage with AI. How much do you use Eleven Labs internally during your development and during your engineering?

24:52 Yeah, this is a fun one. Actually, we we've built a tool internally to use the voice with Cloud Code before Cloud Code actually integrated the voice into their product. And uh we also have a tool again like internally um to use voice with Open Flow, so that Open Flow can call you and you can call it and it's like truly conversational.

25:15 Yeah. Amazing. Amazing. And what's new in in Eleven Labs these days?

25:20 What's new? Well, we're growing really fast. It's uh we have a hundred

25:24 a round as well recently as well, right?

25:26 Yeah. Yeah. We recently raised a new round of eleven billion valuation.

25:30 Wow.

25:31 So

25:32 Yeah. Congratulations. Awesome. So what are the some what are the some of the conversations you're having with uh people at AI Engineer today? Are you Are people mostly using Eleven Labs interested in kind of like integrating voice into their product or development?

25:44 Yeah, people are very curious about the both. They use it as an entry point to show the what the agents, the Eleven Labs agents, are capable of. So usually people come go through the quiz and then I show them the actual agent that they spoke to and I can I I can even show them the actual conversation that

26:01 Oh, wow. Amazing. Amazing. Uh Boris, it's been great chatting. Uh I hope you enjoy the rest of AI Engineer.

26:07 Thank you, Simon.

26:07 No problem.

26:08 Nice to meet you.

26:08 Cheers. Awesome. And we're here just next to the Open AI booth and we have uh Ryan LeCompte who is a member of technical staff at Open AI joining us. So Ryan, welcome.

26:19 Thanks for having me.

26:20 Is this your first AI Engineer or you've been to plenty in the past?

26:22 my first one.

26:23 This is your first one.

26:24 Yeah, excited to be here in London. It's been a It's been a great crowd.

26:26 And uh where are you based normally?

26:28 Uh Seattle.

26:29 Seattle. Oh, okay. So it's a bit of a

Harness engineering replaces direct IDE coding

26:30 trip bit of a trip for you. Um so uh Ryan, first of all uh what are you talking on here today?

26:36 Uh talking about harness engineering and sort of the arc of implementation that my team and I have been on over the last nine months to remove ourselves from the process of writing code as much as possible. I kind of have like banned my team from opening their editor and it's been wild to see how quickly we've been able to stack capability into the code base to let agents do more and more of the software job.

26:57 And that's really interesting cuz you'll get you'll get different types of developers. Some who like that scares them completely. Like I don't My god, that's my that's my happy place, you know, in an IDE writing code. But realistically, I think there's going to be different types of developers. Those that who are creators that will absolutely love this because the the the the job of writing code can be happen faster and it can get taken away from them and they can create faster. What's your reaction to or or I guess what's your team's reaction to whipping out the IDE from underneath them?

27:24 I am a a Rubius. Ruby was my first love, so metaprogramming is kind of a thing that is like in my bones, right? I want to kind of lift myself up a level, use high-level extractions to kind of do the job for me. And I do feel like the harness engineering is really similar to this, right? I am working through a super high-level primitive in a coding agent to do the code production itself. So, the levers I have around me are different. Uh it's all around refining output rather than generating it directly myself.

27:52 Right.

27:52 A trick that I have used though is to just My team is all external hires and they just go full yolo insane into this thing. This is the way we do it. This is why we're at a frontier lab to invent the future.

28:03 Amazing. So, talk us through some of the some of the principles of harness

3 Highlight · Harness engineering optimizes context and attention limits

28:06 engineering then.

28:08 Fundamentally, the models are limited on two things: attention and context. In order to maximize attention and limit context, we want the code, the process, the test to be the same as much as possible, and we want the agents to largely cook with the minimum amount of instructions, giving it context just in time in order to efficiently respect that context and let it operate over very long time horizons.

28:34 Instead of front-loading all the context into agents.md, I want to just-in-time inject how to remediate a linter failure at the time it happens. And this allows us to have like much wider periods of black box behavior where the agent is reasoning and cooking and writing code and resolving its own failures, which allows me to kind of look at the beginning and the end of a task rather than the messy middle sort of thing.

28:58 Yeah, interesting. And And so, how is that done then? Is that done through skills and things like that? Some Some of that almost like progressive

29:03 disclosure of context into the agent?

29:05 That's right. That's right. Uh we use skills for this. We actually uh have very few skills in the code base that we want to centralize around. I think we have five or six. All the engineers on the team contribute to those. And this way we can kind of optimize the human involvement in the process to not have to change too much. We have a fixed number of entry points into the model. Uh and then it's all about figuring creative ways to inject prompts into context, right? Those skills are prompts. Error messages from tests are prompts. Review feedback, which comes mostly from agents, also are prompts. Uh and it's all about getting them just-in-time injected and making sure that they are somewhat reliably followed, right? Like the models create text, we need to figure out ways to give them text.

29:48 So, in order for this to be successful, does it reflect that it's all about the skills being accurately written accurately crafted so that the so that the agents are are are essentially guided in the right way? Otherwise, they're never going to get to that end point.

30:04 Yeah, yeah, yeah. This is why uh I don't really use plan mode very often because

30:09 I'm hearing that more and more actually. It's surprising, right?

30:11 these big plans that I like really don't want to review and I hit yes to accept them. But if those plans actually encode instructions that are wrong, like I'm pushing the agent in the wrong direction.

30:23 we learn as we go that will actually affect how we've done that original planning.

30:27 Yes.

30:27 Very interesting.

30:28 Well, the the time I want to spend is on like the most ambiguous, hardest work over the course of building a product, right? And I don't necessarily know the right shape at the beginning. And the agents operate the same way when they're writing code for a PR. We want them to kind of figure it out as they go to kind of go from a lot of white space in front of them to a nice refined high quality output.

30:49 Yeah.

30:50 And getting feedback along the way is a big part of that.

30:52 Amazing. Now, you're also going to be talking about exactly this topic at AI and Native DevCon, which is happening June 1st and 2nd right here in London.

31:00 Super jazzed about that.

31:02 Which is what? Like less than 2 months away now, so not too not too far away. Um what what is your what is the reaction to to to this session though? You know, when when you present this session, do you are developers typically on board with it? Are they a little bit hesitant with it?

31:18 Uh I get a mix, right? Folks don't want to move their hands from the keyboard, right? Like there's this fear that the code is going to be misaligned, right? And this maybe feeling that it's going to provide me with more work to clean up the mess afterward. But if you invest your time instead to just simply saying I will not accept slop, I don't let the agents produce slop, and I systematize the reduction of slop, you have kind of accomplished the same goal with much fewer synchronous attention cycles from yourself. You want to kind of think like a staff engineer, and you have five or 50 engineers at your disposal. And with that amount of resources, there's a lot of damage you can do in sort of reducing bad code from being produced.

32:03 Yeah.

32:03 Rather than having to shoulder surf along the way.

32:06 Yeah. So, let's talk a little bit about your environment. In terms of when you're doing harness engineering, what does your environment look like? Presumably you're using Codex?

32:13 Yes, using Codex.

32:14 Uh I'm hearing on on Twitter, by the way. I'm hearing so many people go like, "Oh my god, Code Codex is just like unlocking me." And it's like I see so many people jumping into Codex right now. It's it's very interesting to see.

32:25 Codex does the full job. Like I can trust it to go from prompt to merge PR with high quality code reproductions of the work that it has done in the same way I would expect a human, right? Like when I'm working on back-end systems, right? I'm supposed to deploy it to staging. My teammates don't like come over to my computer and like look at my command history to see that I deployed to staging. I'm making an attestation in the PR with some proof that shows that I did this, and they largely accept it as true. And Codex is able to operate in the same way, which gives me high confidence that it's able to do the job.

32:56 I see it executing for 6, 12, 30 hours at this insane pace on Twitter where the laptop is buckled into the back seat of my car.

33:05 Burning tokens as I'm coming to and from work because it is able to do these very complex changes. And the way we do that is Chewie, the web product, the app, which is super super awesome. I love the automations in it. It's constantly cleaning up after itself, you know, every hour making sure my CI is green, making sure that like, you know, we have adhere to our golden principles around what good looks like.

33:28 I have a sticker on my laptop and it says AI codes while I sleep. AI works while I sleep. We're in that space now. capable enough now that that we can do that. Um, we're at AI Engineering as I mentioned. What is the What is the one thing you want to learn from the AI Engineering conference? Is it chatting to people? Is it What's What's What's What's the thing you want to take away?

33:46 The thing I want to learn is where folks are at with these tools. What will it take them to 10x the level of trust and usage of the model to enable them to build more and more products for the world to like solve more and more user problems, right? Like so much software is yet to be produced that like we now have the option to do because implementation is so cheap. So, talking with folks, figuring out where it is they get hung up and how we can bring that to the product is something that I'm super super excited about.

34:14 Amazing. Right, it's been wonderful chatting with you, and I totally look forward to your session at AI Native DevCon, June 1st 2nd in London as well. So,

34:22 I'll be there.

34:23 Excellent. Thanks very much. Hey everyone. Hope you're enjoying the episode so far.

34:28 Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But, for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode.

35:02 Well, we're here still in the in the expo hall at AI engineer and we've we've bumped into Jordan Jerrett who is a an applied AI team lead for the incubator for artificial intelligence project for the government. So, tell us a little bit about your role and what you do.

35:20 Very good to see you. Yeah, so as you

Multimodal agents digitize government planning data

35:22 said, I'm the applied AI team lead in the incubator for AI. We're a team of about 50 technical whole permanent civil servants working in UK government to prove that AI can be used as a tool for the public good.

35:32 Amazing. And so, tell us tell us tell us a little bit about kind of you know, how the government is using AI today.

35:38 Yeah, so there's a lot going on in a lot of different places. But, what we think there's a real need for in government is to have really strong technical capability and like a full-fledged product development applied AI team that can ship things from zero to one and like get things and scale them out across the public sector.

35:55 Yeah.

35:56 Um so, three things that we're working on at the moment which we think are really really exciting. First of which is government consultations. We built the default app for analyzing government consultations that's saving hundreds of thousands of hours a year and many many millions of pounds and is also more quantitative and better than humans analyzing these consultations.

36:17 Okay.

36:17 Rolling that out to look at consultations like digital ID and other sort of ongoing things that are big contentious political issues.

36:23 And for those who aren't in the UK recognizing UK government, what is a consultation?

36:27 Consultation is when government goes out and asks the public what their views are on a major piece of like legislation, how it's going to affect what you can and can't do in the UK. So, it's a really important part of the democratic process. And getting that analysis right is really important. Yeah. One of the things we really have to be able to prove with our AI models is that humans agree with our AI consultation analysis tool more than they agree with each other.

36:50 Right.

36:50 As a result of it. The um second thing that we're doing is digitizing the planning system.

36:55 This is This is super interesting. You showed me this earlier.

36:58 Take a look.

36:58 So, so tell us a little bit about Well, tell us a little bit about Tell us about the problems, I guess, that exist in in that space today and how AI

37:07 Come and get a look at this. This is one of the basements of a council in the UK. And what you can see here are boxes and boxes full of old planning records, which tell you what you can and can't do with the land in the UK.

37:20 This is stressing me out. This is really stressing me out.

37:22 I've had people say, "I hope that the basement floods so that all these records get destroyed and we don't have to go down and look at them every time you apply for a let's say an extension on your house or getting a new asset."

37:32 So, planning permission a planning permission application comes in. Someone physically has to go down. They have to try and identify a card from from the from these places, pull it out, read what it is, and then understand, "Okay, yes, you do have these permissions." or "No, this this this tree has a conservation order on it." or whatever those issues are. It's a very, very manual process.

37:51 Very manual and we've got what is often a very paper-based system. So,

4 Highlight · Multimodal agents digitize paper-based planning archives

37:56 This isn't an AI problem. This is a data problem, as you mentioned earlier. So, what are the What are the stages that need to go through in order to digitalize this? And then how does AI help?

38:05 Cool. So, what we're doing is we're taking these nasty old documents and we've created multimodal agents that can go and take them and turn them into nice geospatial data. So, this is a little like a demo of of of our app here. You You see you can get our local planning officer can go and upload that PDF. We use models like Flooding like Gemini to go through and do structured information extraction. Then do a lot of extra hard work on top of it to go and figure out where in the country this map is, what the correct objects on that map are, and segment them, and then figure out the coordinate transforms from this old pixel coordinate map to this modern geospatial data. You can then see here we're also working with the planning officers to get them to go and tweak and edit it and correct that one will enable them to go and upload their data to

38:54 the um the the planning portal, planning.data.gov. So we start to fill in this map of what you can do with the land in the UK. Um the other thing is we also get to keep all that data for our e-vals. We can create a self-improving system so we can digitize the system even faster.

39:09 The e-vals are so so important for that.

39:10 So crucial. We showed it to the PM and he committed to digitize the UK's planning system by the end of the year. So what we're doing is trying to prove that AI can be used as a tool for the public good.

39:20 Yeah.

39:20 And be like pretty radical about the things that we try and achieve with it.

39:23 And when you say by the end of the year, what scale of data are we talking about here?

39:27 I want to see that map filled in.

39:28 And and and what what does that mean in terms of like this sheer number of of the volume of documents and things like that?

39:34 So we did hundreds of thousands of documents at least and millions and millions of trees and old like fields and lots of different the whole history a whole stack of different like land use that we need to go through and digitize.

39:45 And from a productivity point of view like if we think about a human doing this this Still a human in the loop as we as we mentioned but a human doing this like what kind of a productivity improvement are we talking about?

39:56 So it often takes a planning officer about an hour to digitize one of these records properly. They also don't necessarily always fill in the information in exactly the right way. Now we can get it done in about a minute two minutes for about 10 p.

40:08 Yeah.

40:09 So there's another problem which is there aren't enough planning officers in the UK to be able to do this. Yeah, Hey, we need to kind of like think about how we can use AI to transform like the labor market problem here.

40:20 Yeah.

40:21 Um so there's this thing and then you then you can ask so that that's one layer of efficiency. That's that's that's kind of like one useful thing to do. We save time, we save money. But think about what we can do on top of this, right? So like once we have a digital planning system, what can you do? Well, you can start to do things like this

40:39 Yeah.

40:39 which is build agents on top of that like digital planning stack. So you could say create an assistant for our planning officer. So when you put in your application, the agent can go run, draw down all of the evidence that it needs, suggest the right decision, and get instead of an answer in 13 weeks, get an answer in a week.

40:55 Yeah.

40:55 Like that's the sort of future we want to enable through digitizing the planning system. Um and there's a ton of all like brilliant prop tech companies in the UK that are also pretty desperate to get their hands on this information as well. So

41:06 Yeah, yeah.

41:07 we think there's a role for AI to be used to be able to curate and then publish more open data in the UK for people to build on.

41:13 Amazing. Um I feel like the government the UK government is actually very very progressive with a lot of the digital assets that it's done over the over the last 10 20 years even. In terms of like the AI usage, are there like a number of restrictions based on the sensitivity of the data, there's there's certain models you can't use or or things you need to be extra careful about here?

41:33 So in the incubator for AI, we're fortunate in that we're given quite a lot of trust and quite a lot of control over our own stack. So we're a team like of developers, we're very like engineering first in what we do. So we are able to use pretty much whatever we want. We've got a lot of funding to be able to go and like play around with these models. Um I'll also mention we're hiring for a load of applied AI engineers. So if anyone should come and join us on that that mission at the moment, they definitely should get in touch.

41:58 Where's the best place to get in touch?

41:59 They can go to ai.gov.uk.

42:01 Yep.

42:01 Um and there's a careers page there they can check out. Um and um yeah, but like we also have to think about how we kind of deploy these tools responsibly and

42:10 and safely. Um and and that's a big part of the work is evaluating, like, really, really carefully the quality of the output of our models. So, we're we're both applied AI engineers who want to build stuff, but we also want to really, like, rigorously test our own work. Prove that it's good. One way that we're doing that, if I can shoehorn it in, is the work we're doing in education. Um, so there's for the last sort of 6 months, um, the Department for Education has really had its fingers in the in its ears about the use of AI in education. And kids are already using AI as a tutor.

42:46 Yes.

42:46 But we don't know whether that's safe or effective. We don't know if that's a good way to learn. And as government, we've got a responsibility to make sure that people are learning well. So, what we're doing about it, um, we're constructing a first-of-its-kind benchmark for AI tutoring. And that involves a lot of things, but primarily that's working with teachers, EdTech market, um, and the Department for Education to define what good pedagogy looks like. So, are you giving the right sort of advice to children? Are you doing that in a way that's conducive to them learning? You're not just feeding them the answers, but you're teaching them how to learn.

43:21 And also, um, doing it in a way that's safe. So, if you're creating dependence, for instance, or creating like an emotional attachment, can you like prevent it from doing that? So, coming up with a wider range of, like, uh, metrics that we can use to analyze these long-form student-teacher interactions over here. Um, we're then also working with the Frontier Labs to turn these into benchmarks that they can go and test their models against to see, before these things get into the classrooms, whether they're safe or effective.

43:48 Excellent. Amazing. And it's so wonderful to see like, AI being applied in such a in such a in such a way that is actually, like you said, for the good, for public good. So, thank you so, so much for sharing. Really, really, uh, really, really great to hear. Uh, one last question before we go. We're at AI Engineer. What are you hoping to learn or or share with the audience?

44:09 Cool. So, first thing I'm wanting to share is basically the vision. I want to I want people to know that they can come into government for a tour of duty, come and work with us for a bit, do some of the most like impactful work of their career, affect everyone in the country in the process. And there's a place for you to do it, which is Incubator for AI. Um they should also check out the number 10 Innovation Fellowship Program, which is for like the top tier technical talent to come into government on a tour of duty, too. Really exciting opportunity.

44:34 that you're a a member of as well.

44:36 Yeah, yeah. So, that's how I got into government. I was in academia doing like some computational bioengineering stuff, like in deep biotech. I thought I want to work on stuff with massive social impact. And I find it really gratifying. I've chosen to stay. Um the other thing I'm really excited about in the conference here is just the quality of the people in the room. It's like the most incredible crowd. Um huge credit to like Shawn and the organizers for for pulling this off. I've I've had this thing in the back of my mind, which is like what does the government agent stack look like?

45:04 Yeah.

45:04 We've got this like really gnarly sort of tech setup in the UK, where you've got lots of delegated kind of um different sort of cloud environments with different responsibilities, but it doesn't really reflect how people think about what government is. They think of government as one organization, but we don't operate like that.

45:21 Yeah.

45:21 So, I'm thinking like how could I work with something like Work OS um or some of the other kind of like um uh sort of deployment and like infra platforms that we've got can see in the room here to build agents that could work seamlessly across government.

45:34 Yeah.

45:34 And you have any You give me any pointers on that? I'd be very grateful.

45:36 Yeah, that's really good. Yeah, yeah. Awesome. Thanks so so much. Been great chatting with you and enjoy the rest of your night today.

45:41 Yeah, cheers.

45:42 Cheers. Look who I bumped into, Nipa, who we actually yesterday had a panel at the Tess office for the AI and Native Dev community. So, it's great to see you here at AI Engineer. How are you?

45:54 I'm good. Thanks for inviting me.

Automated code reviews manage generated code quality

45:55 Yesterday was a really nice session.

45:57 It was amazing. And of course, you're a solutions engineer at Cado. Tell us a little bit about Cado, what you do, and and and what you do at Coda, and what Coda does as well.

46:05 Yeah. I'm a solution engineer with Coda, so try to solve problems that AI creates by AI. So, it's like a loop. AI creates problem, and AI is trying to resolve

46:15 right?

46:15 Exactly. So, at Coda, what we are trying to solve is the code. So, when you use AI, you create so much amount of code that it practically becomes impossible to keep track on what exactly is being created, is that a the standards can be actually promoted to production.

46:33 Yeah.

46:33 Uh initially and currently as well, we see every now and and then things, security concerns coming up in the production with major players as well. So, we are trying to make sure we can reduce those gaps as much as we can by providing guardrails, by providing more patterns for the organizations work with, so that the secure code can be promoted to the production.

46:59 Now, when I first heard about Coda, it was Codium back in the day, and then it moved over, and and Codium was built around testing.

47:08 Yes.

47:08 Um and essentially creating unit tests, those types of things, and then rebranded to Coda. Much much more about it, feels like it's much much more about code reviews these days. Is that Is that

47:19 That's true. That's true. So, when we started 3 years back, we started with code gen, we started with testing a part of code reviews. But as we grew, we understand before even testing can come into picture, code reviews is a step where we have a major gap. And that's where the whole team thought like, "Okay, let's try to fill in that gap, and we are good at it." For example, for the last years, we are just trying to make sure we don't do only code reviews, we reduce the noise, which usually nobody is looking at. If you see the code gen as well, it's more the better. But what we are saying is it's not more the better, it's more about more quality, the better it is.

47:56 So, we shifted all our focus to code reviews and then now we do good reviews.

48:00 So, there's two important things I think about code reviews. One is with the with agentic development, we obviously need possibly a human in the loop, possibly some deterministic testing to say is this code reasonable of reasonable quality? The second thing is the sheer volume of code that is being delivered these days and that will be delivered in future. I think Logan Kilpatrick kind of from Google DeepMind said he's expecting 5 years a million times more code that we generate than today.

48:28 With that volume, how do we how important is code review to enable that we deliver at speed while maintaining that we're not just shipping rubbish into production?

48:41 Sure. I think that's the main focus of current code review industry. It's I think this is a problem not only us, anybody in this industry is trying to solve because again, everybody thought process is we have to ship faster, we have to develop more. But at what cost? Do you want to ship faster in a week and spend maybe 2 weeks trying to figure out what went wrong and how to fix it? Or do you want to make sure you can ship faster with guardrails, with making sure the code you are providing to the production is good enough. So, I think we should be a balance where it's not about shipping faster, but it's finding a balance where you can ship faster with the quality code in production.

49:23 And what's more important would you say cuz I like the industry, there are startups flying out everywhere. Everyone's Everyone's going fast. Would you say it's more important to have high speed or would you say it's more important to have like zero issues? Because of course, if we can react to know to to known issues that we that we ship, is that better or worse than shipping perfect code, but it taking longer?

49:44 Uh yeah, I think in the previous days as well, there were senior developers who can actually build their code in a couple of weeks and then ship it. And now as well with the AI, it's the faster. I think it should not be very restricted, otherwise we are restricting the tech or the thing which we are working towards. But we should differentiate between what is critical and what not. Why not experiment with going faster, creating more experimental guardrails on the things which are least important, but still making sure the critical work we are doing around can be saved and can be checked a little bit more better. And I think we are in a place where we we are becoming more and more comfortable with AI doing a lot of automation. So yeah, we can experiment with the non-critical stuff, and once we are comfortable, maybe try it with so we should find a balance not restricting because then the whole point of getting AI and shipping more code goes away.

50:38 Yeah.

50:38 But on the other hand, not let everything done be by AI so that it you have to pay a lot of cost for that. Yeah.

50:46 How do you advise developers use context and skills to create you know, better, higher quality code from scratch that then obviously we still need to validate and test against, but really getting that baseline to be better. How would you How would you encourage people to use like skills and context to to to really enable their agents?

51:06 Yeah, when you say context, context is crazy nowadays, and this question is crazy too.

51:12 Because if you see the context, people don't understand what the context means for them. When you say a developer, okay, the context is important, they think, okay, everything what I have in the organization, the whole code base, you know, the Confluence documents, the Jira documents. Although the context window for our AI models is increasing day by day, but does it actually impact or does it understand the whole context? You can dump the whole code base to the LLMs. Does it understand? Does it actually look into things and say this is important or not? And if you see the current, we are not into that.

51:47 You can dump data on to data, but LLMs cannot. They just choose the first part and the last part and all in between context is lost.

51:54 It's about the right context, the right skills, the right context.

51:57 Exactly. And then the second part is the mature developers thinks like they can build context engine. They can create whole vector DB or the mapping, but is that really important? Is that really worth spending the time on? So, for me it's important to understand what exactly I'm trying to target, what kind of problem I'm trying to achieve and based upon that choose what's important, what kind of context mapping I can do instead of building a whole context engine. Can I do iterative reduction or can I do hierarchical mapping? And then create something to make your results better.

52:32 And let's wrap up with a final question. What do you want to learn? What What do you want to take away from the AI engineering conference?

52:38 Oh, I think the the conversations I have is the most important key for me this conference because when you look when you work with a product or when you work with a specific organization, you tend to start looking in a specific way. If you see AI, it's being used in so many different ways, so many use cases. Talking with people gives you an idea what exactly is happening, what are the problems. And next time maybe I'm solving a problem, I have that I'm a solution architect. That's what I do, right?

53:08 Yeah, yeah.

53:08 So, it gives me a new idea about what actual problems are and how can I help.

53:15 Amazing. But it's been awesome to talk with you again and enjoy the rest of AI engineering.

53:19 Thank you, Lonnie. You're so great.

53:21 Thank you. So, we're on the the Zed booth and joining me is Cameron McLaughlin, an engineer at Zed. Cameron, how are you doing?

53:28 Pretty good. It's been a happy to be here, very excited.

53:31 Were you here last night as well?

53:32 Yeah, last night setting up and meeting a few people, checking out the other booths.

53:36 So, tell me like Zed is an amazing IDE. We've had a couple of people at Tesla play at play with it and and and use it a little bit as well. Um, tell us a little bit about what Zed is and also what's happening in the world of Zed.

53:46 Well, it's a text editor. It's a a generic coding tool. It's, you know,

Standard protocols unify agent experiences across editors

53:50 it's a review tool, I guess, if you want it to be. It's kind of, you know, it's it's what you expect from a text editor and IDE. It's Why why should you use Zed? I mean, I like it cuz it's really fast. It's uh it's not on a browser. It's like an all Electron app. It's just written in Rust from the ground up. We have our own UI framework, our own, you know, our own everything. We built it all from scratch. Um

54:08 What's the reason that someone would use Zed compared to a Cursor or something like that?

54:13 It's It's a lot faster. That's That's something that really affects me, like, you know, I you know, I've used Cursor for a bit and you know, sometimes you press a key and it you know, there's a little delay and you use Zed and you press a key and it's the next frame every time without fail. Uh that's something that's really important to me. I think also, like, we So, we just launched our parallel agents feature. Oh, wait, our our demo's not up, but we

5 Highlight · ACP standardizes agent interaction across IDEs

54:29 have this we have this new sidebar. You can run agents in parallel. You can have all of them going. You can use ACP, so you can have Zed agent or you can have Clock Code, Codex, any of these open code, whatever you want.

54:40 You know, Zed was actually one of the first uh kind of like truly use ACP, right? I

54:43 Oh, we created it.

54:45 It's sort of I I'm not That makes so much sense now. Um Well, cuz I think, like, you know, it you know, we all remember the world before LSP and like how, you know, every editor had to make their own Java extension like C extension and all of this stuff and then LSP was pretty revolutionary in allowing lots of editors to sort of speak the same language and I think it's obvious that to at least it's obvious to us that we need a similar protocol for agents. Um you know, you can enjoy one harness and not particularly like its its UI or you just don't want to keep switching key bindings, you know, you don't have to like shift enter to send a message or just enter in another one, you know, all all of these things. They like the the friction adds up over time and this there's something that I've really come to appreciate with Zed is like it's very optimized for like low friction.

55:24 Yeah.

55:24 It's, you know, everything's fast.

55:26 Everything's how you're used to it. The other thing I really appreciate is the collaboration features. We don't have the demo set up at the moment, but the um it's a bit like a Slack huddle. You can sort of join a huddle, pair with your teammates, you can talk to them. Within Zed, yeah. So, there's like built-in voice chat, screen sharing, uh and then sort of this um it's very very deeply embedded in the editor. So, the the like the every buffer is sort of this um this CRDT, this like replicated data type that um you can send these diffs across across our like protocol, you know.

55:55 There is a server, but it's like effectively peer-to-peer. There's you synchronize like the diffs that you're sending and so I can, you know, hop in a call with Bennett and he can be moving around, I can follow him, I can see where he's looking. Maybe he comes across a function that I don't know, I can do go to definition on that resource, split off, join back, and we can edit the same code, run the same code.

56:14 Do you have other huddles as well that you can do in like Slack and then have that integration? That's so cool.

56:18 It's It's basically, you know, we, you know, we don't use Slack huddle. Like we we use Slack, but we don't use the huddles cuz we just like we just hop into Zed to find people, yeah. And then, you know, if you alt-tab out of Zed, it will switch over to screen sharing, so you can see like, you know, if you're reviewing a PR, you can you can look at the browser, that sort of thing.

56:32 Nice, nice. So, we're here at AI Engineer, um probably, I'd say, arguably, the premier event in the world for for for for developers and AI engineers to to to visit. What are some of the conversations that you're having? What are people interested in? And also, are you kind of like are you happy with the with the level of adoption that people here are are kind of like saying that they that they that they're using a various Zed?

56:52 Yeah, there's a It's been It's been funny. I've been trying to sell Zed to people as they walk past and most of them just say they're already using it. So, it's

56:59 uh it's made my job quite easy. I think the um a lot of interest in ACP. I think, you know, from our point of view, ACP is a way that we can, you know, as a text editor, we can let you use cloud code with your familiar key bindings in your familiar UI, not stuck in a terminal.

57:14 Yeah.

57:14 Um but for, you know, for other projects, it's like a way to abstract over agents. It's a way to, you know, if you want to send a message, if if you want to write some code to send a message to 10 different agents and have them compete to to get to provide the best solution, you know, ACP's is good choice to do that. Um I don't remember the details. I think open source using ACP just for for for this for this like abstraction layer.

57:37 Yeah. And I think I I JetBrains I think is also

57:40 Yeah, they're they're working so they're working with us. Um they're working with us on ACP. It's sort of this big collaborative push cuz like you know you would think JetBrains and Zed would be big competitors not but like it's I think it's in it's in everyone's interest to have this standard like like you know I I think the the LSP parallels are clear like a world without LSP isn't necessarily better for anyone develop like it's

58:04 It's a it's a frenemy frenemy relationship.

58:06 Exactly. Yeah, healthy competition.

58:07 Yeah, absolutely. Um awesome. It's been great talking with you. What what advice would you give to any uh agentic developer out there as maybe an unlock perhaps that that you feel people don't really know about these days.

58:18 I think just just try lots of things. Just like try everything. Set it up. I will say this is a nice thing about ACP. You can just try stuff and you don't have to get used to a new whole new environment. But yeah, just just try lots of things. Things are changing really quickly.

58:30 Yeah. And keep keep trying as well because what you tried one month ago is probably quite different now.

58:33 Yeah, it's you know it's Zed is very different compared to compared to the beginning of January for sure. Um yeah, keep keep trying things. Keep experimenting. See what you like. See what you don't like. Uh yeah, build some stuff.

58:44 Awesome. Cameron, it's been great chatting. Enjoy the rest of the day.

58:47 Nice to meet you.

58:47 Cheers. Hey, joining me is Nick Arcalona. Nick, how are you?

58:52 Uh I'm great. It's great to see you.

58:54 Absolutely. Likewise. Uh you just came out of the keynote as well. Uh tell us a little Actually, first of all, tell us a

Observability tools track effective token usage

58:58 little bit about who you were, what you do, who you work for.

59:01 Uh yeah, I'm so I'm the head of AI and research at Jellyfish. We're Boston-based and uh we're in the AI observability space understanding how companies are using agents to transform their software engineering teams.

59:15 met a few times actually. I think in New York AI DevCon was the was the most recent. We've got another one actually June 1st, 2nd. Hopefully you can join us there maybe in London.

59:23 Yeah, yeah, I would love to come back. Yeah, but that was a great event you had in New York. It was a really fantastic. Yeah. good to do the event there.

59:32 Tell us a little bit about the content here. You just came out of the keynote. What was your biggest takeaway?

59:36 There was some great stuff. The The Google stuff is very fascinating. You know, all the things that they're doing outside of large language models. So, for instance, the omnimodal embeddings thing. Like, I would love to have a model that understands, you know, like documents on a you know, visual and spatial and kind of language level at the same time. The weather stuff was very cool. And I've been a big fan of the world model stuff that they've been doing. So, that was great. And then the open claws.

1:00:04 That was very spicy. Like, that was a lot of fun just to see I mean, it's unprecedented that project, right? So, just to see what actually goes on behind the scenes was very was very interesting.

1:00:15 Amazing. Amazing. I see Peter on Twitter all the time asking for contributions versus just people asking for things. I think absolutely need that. What are you here to learn at AI Engineer?

1:00:30 I mean, the biggest thing and why I keep coming to these events is it's just everything's moving so fast. So, just seeing, you know, the difference between the where things were a couple months ago is the most exciting thing for me. And one of the things I like about coming to the expo, seeing the talks is, you know, people say like, "Oh, I remember coming in like off for MCP like isn't solved." Then you come back and there's entire businesses devoted to that, right? So, just the pace of innovation where there's just there's new products, new open source, new companies popping up to fill all these holes. And you learn how these these gaps you didn't know existed and then you know they exist and then they're being filled. Like, that that flywheel is just so rapid, you know.

1:01:13 we don't need this once a year. We need it every month or every other week, right?

1:01:15 Every every quarter. I mean, I've never had a time in my career where going to multiple conferences a year was so valuable. and every time you're learning all these new things.

1:01:24 know you're just going to hear the same thing.

1:01:25 Yeah, absolutely.

1:01:26 What's more important to you? Is it the Is it the the corridor track here or the sessions that you go to?

1:01:31 Um I mean, you know, the the

1:01:33 say if you came here just to talk to me. That's I'm I'm fine with that, too.

1:01:36 Well, you know, it's the it's the kind of live in-person experience because all this content is available online and it's fantastic and you can get it and I could be watching the live stream at home. So, the things that you can only get here by talking to the people. You go to the after events and actually hearing cuz it's the stuff when you're when you're terminally online, you get a skewed view of what reality is. And when you talk to someone, you're like, What How are you you know, person your personal agents are building them? How's that going? And everyone's like, "It's harder than everyone on Twitter makes it out to be, all right?"

1:02:07 interesting. What do you What are you finding the level of folks who are attending AI engineer? Is it Is it different to that online? Is it Are you getting the elite people who are, you know, on the bleeding edge of technology?

1:02:16 Well, it's more just like the Instagram thing where everybody's putting their best selves online. And when you talk to someone face-to-face over a pint, then you find out like all of these things are harder like learning new stuff, changing what we do, and you know, tinkering with all these things is hard. And we're all working with like piles of broken glass and trying to build useful stuff out of them at the the bleeding edge of this.

1:02:37 Yeah.

1:02:37 So, it's tougher than maybe it seems if you just sit around and and read everyone's post about how how great I you know, I how great my open claw setup is, right? It's harder than that.

1:02:47 Let's Let's Let's end by talking a little bit about Jellyfish as well. What's new in the Jellyfish space?

1:02:52 I mean, we're you know, I'm I'm very much on the data side and we're just excited about all of the new data we're getting, you know, and it's making its way into the product very rapidly where, you know, we used to be focused on understanding AI adoption and then a lot on uh kind of productivity and and transformation. You know, are your teams getting faster? But now we're getting very much into the space of understanding tokens. Um you You what does effective token usage look like? What does effective um kind of agentic engineering look like? Because we're able to get data and monitor usage and understand workflows at the token usage level, at the model level, at the kind of agent, you know, tool usage level.

1:03:31 So, really it's understanding, you know, it's not just about did I use the most tokens? Like, did I use them well, right?

1:03:39 how many companies today are saying, "Well, if you're using more if you're spending more money and using more tokens, you're adopting AI more." That's what they're saying.

1:03:47 Well, who's who's saying that? That the people that are selling us the tokens?

1:03:50 Uh surprisingly, actually no. Surprisingly, there are like people people have targets in organizations to say, "Look, we need a we need to use AI more effectively." Well, not more effectively, we need to grow adoption at any cost almost. And it's like if great token usage is almost showing that more of the business is using it. But the problem is what I think you said effective token usage, right? That's what we're missing.

1:04:11 It makes sense to first order, but you're not going to run an engineering org based on how much electricity you use, right? That we use the most electricity. At At these early stages, it makes sense, but everyone's going to want to know why why you know, if you're not having a good time, why? Why isn't the machine working the way that you want? You know, that's what engineering leaders that's what they do, you know, so.

1:04:33 Nick, it's great to catch up.

1:04:34 Yeah, absolutely. Great to see you.

1:04:35 Yeah, I'm sure yeah, and I'm sure we'll we'll we'll bump into each other again at another conference soon. Enjoy the rest of AI Engineer.

1:04:40 Yeah, you too.

1:04:43 Well, that's it for today's episode. I hope you enjoyed some of the conversations we had here at AI Engineering London. Just an incredible groups of of intelligent smart people that we've been chatting with. I loved how many people really here at this conference to learn and understand what's new from not a year ago, but really a week, a month, a couple of months ago. It's amazing how fast this industry is changing. And and really talking to some of the smart minds here and chatting to people in person is really what makes the difference. So, thanks very much to AI engineer for for having Tess will be here in person and sponsor the the conference. And we're also looking forward to our own conference in just a couple of months, AI Native DevCon in London, June 1st, 2nd. Hope to see you there. Bye for now.

1:05:29 The AI Native Dev is brought to you by Tess. The package manager for skills and context. Your hosts are Guy Podjarny and me, Simon Maple. Our producer is Tom Dawler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tess offices in Central London. Visit tess.io/community to learn more and I hope to see you there.