Ryan Lopopolo: OpenAI's Framework for Shipping Code at 70 PRs/Week

9,987 words · about 40 min to read

0:00 We wanted Codex to be able to connect to our Electron app. Chrome DevTools protocol. The MCP was how we started out. Eventually, I kind of took a peek at the code and it was entirely different. Still, Codex was connecting to the Electron app over Chrome Dev Tools protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of the MCP, because we found that there were only 2 or 3 tool calls that we actually needed.

0:26 So more context efficient was faster, and this had happened underneath me, without me knowing. My workflow was not interrupted. That same dependency edge and entity relationship existed, but I almost didn't have to know how it was implemented. Did you find that cool or worrying? I found it shocking. The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.

0:53 Join your hosts, Guy Podjarny and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.  Hey everyone! Hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests, so we can have the most informative conversations about agentic development, whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

1:26 But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode.  Hello and welcome to another episode of the AI Native Dev. And we're here live at AI Native Dev Con 2026 in London.

1:56 And joining me is Ryan Lopopolo, who is a member of technical staff at OpenAI. So hey folks, Ryan, how are you doing? Great. Excited to be here with Cliff is popping. It's really fun. I go on stage later today. I'm just excited to get after it. Absolutely. Looking forward and tell us a little bit about the session that you're giving later. Yeah. So I'm going to be talking about all things harness engineering, which is sort of, I guess, kind of like some of the discourse that I helped define earlier in the year here, where talk about what it means to kind of set up agents to do highly complex, autonomous software engineering, like work in your code bases with high confidence and almost headless nature if you can get to it.

2:36 Yeah, a lot of fancy context management techniques in order to get these agents to learn from their own mistakes, produce high quality code, and go for a very long time without human intervention. Absolutely. And we're going to talk about harness engineering, and we'll talk about some of the methodologies that you've used internally at OpenAI and talk through maybe some

Harness engineering bridges context and tooling

2:55 some of the bits that have been enjoyable, some of the bits that have been scary. And it's going to be great fun. So so, Ryan, why don't we kick off with harness engineering for for the layman. What is Harness engineering? Yeah. So we have these wonderful technology in these coding agents and these fantastic models that are very smart and able to produce

1 Highlight · Harness engineering structures context and tools for agents

3:16 a ton of code and harness engineering is this idea that to produce code that is acceptable, to produce code that we trust, that we can build confidence and business systems around? There are a bunch of little decisions that go into the process of producing PRS, producing systems that we have to make along the way, and in order to get the agent to know and understand those, we have to write it down. We have to make sure that, like the content of those choices, all those nonfunctional requirements that go into making good code, get surface to the agent at the right time.

3:49 So it's a combination of getting context into the repository in a place where these coding agents can see it, surfacing it to the agent at the right time, and making sure that all the code that we produce kind of like follows the golden thread of what it means for us to produce high quality software. Reviewer agents, tests, lints, big refactoring loops, all sorts of ways to get the agent to close the loop and make sure that it proves to me and the rest of the team that the code it has produced is acceptable and can be merged.

4:21 So we've had prompt engineering, context engineering, and harness engineering. Now, what would you say? The kind of like the main differences or overlap between context engineering and harness engineering? I think really the two only levers you have when interacting with agents is which context you provide and which tools you provide. And harness engineering is kind of a combination of both, with the goal of making sure the context that we give the agent is correct so that it can work its reasoning magic to produce good code.

4:51 And one thing that I think is unique about harness engineering is we're actually using the tool calls to prompt inject the agent to manage its context window. One thing that's pretty neat here is the way that we structure tests and lints and well-formed tests in the code base is actually very different than how I would structure those messages for a human right. I would have an exhaustive list of failures that I give the human, and have them wade through the logs in order to figure out what's wrong.

5:21 Whereas with agents, you know, due to the nature of how they invoke tools, we want those messages to be somewhat compressed, but also semantically meaningful, right? Rather than giving sort of a mechanical eslint failure, for example, I can give a prose error message that says, you screwed it up in this way, in this file, you know, please open it and follow the run book in file X, y, z to go fix it. Because we've seen you make this mistake before, and we know that we can trust you to do the right thing.

5:51 Yeah. And that that level of integration, that deeper integration really unlocks that next step right now. Interesting. I'm like looking at some of the some of your background here. You built stripes efficiency engineering team led developer productivity at Brex for 350 engineers. There's a bunch of kind of like infrastructure and background there. Would you say that helped you or hindered you from the point of view of harness engineering? I definitely helped. It kind of got me used to this idea of working through others.

6:20 You know, when I'm trying to improve productivity for an engineering organization of 350 engineers, I can't be hands on in the weeds of every PR that's being produced, and I can't work directly on the underlying systems. I kind of have to steer from the background, make sure that the right things are happening. By default, the right systems are in place in order to have high trust in the organization to do good work. And that's exactly what it means in order to be high functioning with these agents, in order to move beyond sort of coding agents as pair programing assistance to go massively parallel where I've got the 15 team windows open, I need to be able to take my hands off the wheel to a certain extent.

7:02 So this way of operating this sort of systems thinking mentality kind of comes naturally to me, because that's kind of the expertise I've had to build up over the course of my career. Yeah, yeah. Interesting. Would actually touch a little bit about, I guess, the part of that which is, well, how much is how much trust do you need then to have those 15 open. But we'll cover that in just a second. I'd love to. You know, when you have those 15 open, there's obviously a huge amount of age in written code and various different roles in the in the workflow and the pipeline.

7:30 Now, a really interesting constraint that you imposed on yourself and the team was no human written code.

Banning human-written code exposes agent friction

2 Highlight · Enforcing zero human-written code to build agent tools

7:37 That's right. And I guess, first of all, how how fearful were you at the time when you said, look, no one's to write any code here in this project? How confident were you that that was the right course? It was definitely a bit of a radical idea at the time. Kind of when I started out building in this way was even before we had proper reasoning models. GPT five came out in August of 2025, and I started working in this hands off way in June.

8:09 So this is, you know, oh three era Codex Mini had just came out the very first versions of Codex CLI. And to be honest, back then the models were way less capable. It was very painful in order to operate hands off like this, and sort of by necessity, working with the model, working with the coding agent, I kind of had to stick myself in as a very chunky tool for the model to delegate to if it got stuck, or if it needed to do something.

8:40 The only thing it could do at the beginning was ask Ryan to do something for it, and eventually I get tired of being asked the same thing all the time. Initially, back then it was installing dependencies with cargo. This is just not something that that Codex was able to do reliably back then. So I would type the commands in my terminal and, you know, give the agent some dependencies to work with and sort of automating. That was the first step, and it kind of got me in this mode

9:06 of paying very close attention to where I'm spending my time figuring out ways to build up a tool call that removes the need for me to spend that time. So then I can kind of figure out the other ways that I can help the agent get more and more chunky tools, to give it more capability. And to start with zero like that, you kind of. Viscerally experience the failure of the agent, I think, to kind of have a good sense of the tool calls as they are making the mistakes the agent is making.

9:42 Sort of in the process of writing the code is really, really important in order to build confidence in it, because to see what mistakes it's making, I'm also able to see the parts that it does really good at, and it can start to retool. The way I go about my week to lean into the stuff Codex is really good at. It's really good at following instructions. It's really good at writing tests and calling them. So collapsing a lot of the engineering process into these, like very well paved roads for the agent helps me build confidence in it.

10:14 It's funny how you say it's really good at following instructions and and really good at writing tests. And I'm thinking, wow, the complete opposite of most developers grid is that is that maybe that's wrong to say. But so that's really interesting. And I guess when you think about your team like this is a decision that you made, but when you think about your team, how how comfortable was how on board was everyone in joining this?

Onboarding new hires through shared agent context

10:37 I got very lucky that as we grew the team, we grew it all with new hires to the company, so they can't get dropped into this insane environment and being like, oh, I guess this is just the way we do things now. Let me just go whole hog at it. It usually took about two weeks for folks to ramp up to this way of operating some growing pains very early on in growing the team. Probably a good amount of dog feeding as well. Yeah.

11:03 Of course, back to the team. I mean, the the product we were building to also helped a bunch. We were building sort of a data analyst agent, a data analyst agent using Codex inside in order to write SQL and cold Databricks and all these sort of things means a lot of the techniques that we were using to write the code were techniques we needed to deploy into this agent, to have it be effective of a very cool, virtuous cycle. There took about two weeks for folks to onboard, initially very painful in terms of figuring out how to coach folks to not produce slop.

11:36 A big part of maintaining a high velocity with these tools is just not permitting slop to enter the code base. But once we sort of had figured it out, by the time we got our fifth, sixth, or seventh hire onto the team, within those two weeks, we were actually seeing like velocity and PR throughput go up five, ten, 15%, which is again, not the common path. When you onboard quickly to a team like the the precinct time.

12:02 Right. And the reason for that is because as we accrued more and more context, more and more capability into the code base, sort of everyone invoking Codex as the sole entry point to the code base was getting the best parts of everybody by default, rather than having to spend one, two, three months absorbing the best practices of the team, they're already there, which means new hires to the team were able to very quickly supply their best judgment and context, which means now everybody is being more effective very, very quickly.

12:38 You're effectively onboarding the new hire through the agent already. That's right. Yeah. That's right. Brilliant. So there's a really nice phrase. I'm sure you've heard. I've said it a million times as well. You can go fast. You can only go fast when you have breaks. Right now going in this scenario going fast is using, you know, Codex and AI agents to to generate that code for us at a faster velocity, that then, then we would be able to manually write the breaks. Of course.

13:07 Are those tests, those reviews that that, you know, validate the code that show and highlight where we need to actually spend time and fix and adjust before we go live. Another interesting thing that is the Zero Human review. Now, this is almost you're putting AI to be the brakes now, which is which is a little bit more worrying. In terms of, I guess, you know, an alarming and an alarming proposal.

13:38 How comfortable were the engineers when you said, right, we're now going to do zero human review on this code. We're going to we want, you know, agents to be able to do that review. Yeah. So sort of like the idea of post merge code review if at all sort of thing is, you know, kind of like not not a new thing. Right? When I started my career back in the early 20 tens, like this was this was a normal part of what it meant to write. Software is kind of have a high trust environment with your team, bias to action, bias to throughput, and, you know, have some representative sample of the code that you are looking at, but you know, otherwise have lots of synchronous communication on it as a team to make sure that everyone has the same mental model of the system.

14:19 And it's very cool to kind of like have this come back into vogue with agents, because if the entry point to the code base is always the agent, it's really not required for every engineer on the team to have a full and complete mental model of the system, so long as that they are able to trust that everyone else on the team is enabling Codex to have that level of expertise. Which is not to say that like there is no review at all in the system, right?

14:49 Like to kind of provide a little bit more nuance here. We do have some cases where we require like two party traditional review pre-merger, and most of the time that is centered around things like highly complex plans, sort of phase milestones that would take place over the course of an actual week, these sorts of things. And the reason for that is these plans, these milestones, these exec plans function as the prompt you're going to give the agents.

15:18 And if you are relying on a text document to drive implementation, being crisp around what words are in there and whether or not they fully under or specify the task is quite important because if you specify the task upfront, you are going to get garbage. So that's kind of where we focus the the bulk of the human review. And so the review here like it's different to how we would normally think of a code. Yeah. It's up front right. Yeah.

15:43 It's like first of all a code review. You know we will argue about details and naming and things like, you know, almost more style guide style comments or rather that's what sometimes happens, unfortunately. Whereas now what we're looking at is actually a much more I guess there's probably a couple of reasons. One, because it's actually more interesting to actually, you know, review at that higher level anyway, because they're more meaningful input from, from our, our point of view, where we need to kind of like much more, think about what's the greatest value I can provide here.

16:13 And it is that more meaningful decision making at the top. But also, I guess we're probably more abstracted away from the code. So there is an amount that we just have to trust. Or actually it'll it's almost a waste of our time to go too deep into the implementation because we need that to be abstracted away. And we need mechanisms of where testing just occurs to make sure that that succeeds. That's right. This is sort of again, that sort of like group tech lead sort of mentality. Right.

16:38 Like when when I lead developer productivity at Brex. Right. Like I don't have my hands in the weeds of every bit of Terraform. Every team is writing. But I do care at a high level about like guiding principles. Right? I want the infrastructure to be reproducible. I want it to provision quickly. I want us to have modules in place that solve common developer workflows. But like how you structure your underlying modules, whether you're using for each in an idiomatic way.

17:06 I don't really care too much about that, as long as the code is like, locally coherent and actually solves the business problems that I'm hoping. To do. And ideally that's a decision that maybe even change over time of how something should be structured. So it's not something we should always be so connected to, at all times. Anyway, I think we mentioned kind of like trust a little bit, I can't remember what the, the magic ratio of trust is, where it doesn't matter how many times something works, it's it's how frequently something doesn't work is the key which which breaks the trust.

17:35 But I guess at what stage were you and the team like, truly thinking, oh, actually, do you know what? I completely trust this. Not not, you know, I guess from my point of view. And I'll talk about myself here when I say laziness. But there is a there is a time when I'm using agentic development where I'm just kind of like assuming and just being lazy about not not, you know, reviewing things in enough depth, but in terms of like, true trust is in. Yeah, I'm going to I'm going to hand this off to the agent and believe that this is going to be high quality, that I can actually, you know, push to production.

18:05 At what moment would you say in the, in the during the process, would you say you were truly trusting of what was being delivered? I think there were two sort of phases here. One was, do I trust the agent to be able to produce code? Well, in the magic moment, for that was sort of working on what was essentially a very early version of what the Codex app is today, talking with a teammate around.

18:33 Man, I wish I had voice input here, and then being able to just give a prompt and half an hour later see that in the real world in the app was fantastic. You kind of got this feeling of confidence that it really is down to my capacity for scheduling prompts to go into this magic machine to see my ideas come to life. So that was sort of like the first phase of getting into this mindset of our role as engineers should be to figure out ways to feed the machine, to schedule work, in order to kind of have our vision come to fruition.

19:09 The next sort of phase was, is this code trustworthy? Is it high quality? And this sort of came to us as a team, sort of out of necessity when we onboarded our third engineer to the team,

Asynchronous anti-slop loops maintain code quality

19:25 we significantly increased our PR throughput, but at the same time, we're unable to keep pace with reviews and making sure that slop didn't enter the code base. We kind of started to kind of pump the brakes a bit like you mentioned, and take one day a week, Fridays to do garbage collection, where a lot of long, synchronous stand ups in this way of working. Right, in order to socialize that mental model.

19:50 So we would kind of compile a list over the course of the week of things that we didn't quite like. Right? We want to bias toward throughput, bias toward merging, but still we want to make sure we maintain our quality board. So every Friday was trying to eliminate that slop, but also finding ways to systematically eliminate it going forward. Right. We never wanted to give the same feedback twice in the same way that you want, you know, a new engineer you're onboarding to the team to actually internalize what it means to do a good job.

20:17 So that kind of stacked into sort of these programmatic guardrails and then a very long outer loop of sort of these automated CI jobs that are looking for slop, identifying slop, kind of compiling a list of golden principles for what it means for us to write good code, finding divergences from it in the code base, proposing PRS. And at that point, we were able to see, oh, the agents are able to write code the way we expect.

20:48 We just kind of need to put the right loops and systems in place for it to kind of close that feedback cycle for itself. And presumably that was given to it by context or skills or something like that. Before skills. Were asking. So we did a time. Before, yeah, I know, I know only only what, eight months ago at this point. We. We kind of did a very cheap hack.

21:15 We kind of had just a GitHub issue where engineers and agents would leave comments around sort of underlying principles of good software development. Right. How to properly make a react snapshot, test what it means to write reliable network code, how to think about structuring a lint. And, you know, you end up with this big issue with 100 150 comments on it.

3 Highlight · Ingesting human feedback to automate anti-slop loops

21:43 And that was sort of the seed for what it meant for the agent to go sort of spider through the code base stack, rank a set of violations, proposed PRS. And then one interesting thing there is we had like an on call whose job it was to supervise all the agent produced stuff, all the headless stuff, because we we didn't quite trust it yet. We needed eyes on it in order to build that confidence.

22:08 Human with thumbs up, thumbs down, merge or not, leave review feedback. Which means the next time that anti-SLAPP ification loop spins up, it looks at all the PR it had produced. It ingests that human feedback, compares it with the session logs from the last time it ran. Says which mistakes did I make? Which priorities did I miss? Did I produce misaligned code and why? Let me figure out how not to do that going forward.

22:35 Produce some markdown. Save it as an artifact in the GitHub run. So the next time I run, I have this extra context that I have learned from the actual human feedback in the system. And is that the correct way? Is that the way you would recommend others set up their agents to, to almost like iterate through? So it's almost like build the code and then do these almost like audits or reviews in post, and then raise PR where you're almost, you know, you're outside of the guardrails.

23:02 Or would you today look at adding that into

23:07 the agent at code creation time to make some of it's a mix of both. Yeah. We we do think sort of these asynchronous loops are a big part of what it means to be effective. Early versions of Codex Security via Aardvark also worked in the same way, right where I'm getting a security review of my code after his ended up in main. It is maybe finding a way in which it is vulnerable. Producing a patch. We put up a PR, we review it and it merges.

23:34 And I think this sort of flow is really, really nice because it allows mistakes to actually make it to main, which means that it can be learned from anytime we are able to come up with a correction, one, we want to fix it, but also repeating patterns sort of thing, which we identify with agents and loops and distillation and all of that. That's the stuff we want to pull forward earlier into the pipeline.

23:59 Yeah. And when you when you do push with mistakes or things that you want to correct, are you less fearful of pushing those mistakes to production these days because you know how fast those can be reviewed and fixed by an agent? It's definitely a spectrum, right? I will say that for this project, we didn't have continuous deployment, right? We're building a native app. We are manually cutting a release branch and doing smoke testing.

24:26 That smoke testing is initially all by humans, where we look at where we're spending the time and try and get agents to do some of it right. But it is important, I think, to have some level of human supervision over the release process in this world. But if I were to kind of like put my group tech lead hat on again, right? Yeah. Some mistakes are are able to be made like the consequences of them are low. And much like I want engineers who are becoming more senior to learn from their own mistakes, I also kind of want to see mistakes that the agents make so that we, as their supervisors, can figure out how to get them to learn.

25:02 As well. Because if it feels like in this kind of scenario, debt and fix is cheap because it's about, well, we'll run an audit. And as long as you have those guardrails and rules in place to say these are these are some of our best practices, I can put, you know, something in which isn't ideal. And no, it's not going to take six months for a human to come and review this code. It can be done in a more regular time frame.

25:28 Yeah. Like with this sort of like new world where the production of code is so cheap. I actually think that you can at least working with agents flip sort of like the DevOps shift left thing onto its head a bit, where the cheapest thing I can do is literally produce a change to my prompt, whether that is like one of those golden principles that I'm eating into that GitHub issue, or, you know, a prompt that I socialize with my teammates.

25:58 Right. That's the cheapest thing I can do in order to fix the code that is produced. And, you know, if we find that, like the mistakes are a bit more systematic, I can kind of think about pulling that further left. I can add some docs, or I can operationalize a review agent that is going to judge every PR according to those docs. Or I can do more shifting left by making deterministic tests that suss out this behavior and on and on earlier in the process.

26:27 But the cheapest amount of effort I can spend to shake out that misbehavior is kind of where I want to start, because these things are magic reasoners that most of the time they do pretty well with very little effort on my. Part is. Super interesting, and I'd I'd almost love to jump ahead a couple of questions, because I kind of want to ask the question when the the way you talk about that, whereby, you know, you make minor changes over here to a prompt and then, you know, implementations and things can just change.

26:55 And it kind of makes me feel that when we talk about the thing that persists, it's less about the code and it's more about our prompt and our context and our functional requirements. And it brings me to something called Symphony, which is a ghost library because it's spec. It's all about specs and context versus versus implementation or code. And I guess when we think about when we think about that, the code just becomes a throwaway artifact.

27:25 The implementation is just it's there today. It might change tomorrow. But the thing that persists, the thing that we care about and we should put most time in, like you say, lifting up those reviews more to the functional requirements and things like that is the specification and the prompts. Is this a is this a future? Is this a is this like when we when we think about the relationship between the specs and the code, how much is that changing today?

27:51 Yeah. So it's kind of funny.

Deriving specifications from working prototypes

4 Highlight · Prototyping implementation first to distill robust specs

27:55 Where you think traditionally inspec driven development. Right. You start with a spec and then you kind of refine it as the code is produced sort of thing, to make it more concrete to suss out those hidden decisions and ambiguity in the spec, and for things like Symphony, for all these lists that we're producing, for things like documentation on critical user journeys or product features or click flows and screens, I've actually found it is much easier to produce the code first to have that code sort of like present a straw man for what the world could look like, engage with that as a team to refine it, and then distill the spec out of that, because sort of produced artifacts, whether they are code, a spreadsheet, a Google doc, right.

28:47 These things are very information dense in terms of the decisions that were made to go into producing something that we have accepted as good. So for Symphony, the thing that we shipped is very much a spec, but the thing that we started with was kind of a vibed implementation of Symphony in TypeScript in our mono repo, where when we thought it was good and it was solving the problems that we wanted it to, we worked on producing a spec out of that to share with the world, and it was this very, very cool process where we had this three phase pipeline.

29:17 We started with Symphony as it was, and we gave it to Codex to say, produce a spec markdown file that is able to reproduce this system. We end up with a spec, give that to a new agent that says without access to the original, that says you have a spec, please implement this system. You know, it goes for a while with goal in order to produce this thing. When it says it's done, we give that system the spec and the original implementation to a third agent is going to act as the judge to say, we are attempting to replicate the original version of Symphony.

29:55 This was the spec that was given to an agent to produce it. And this is what they came up with. Where are there misalignments in the derived artifact versus the original system, and proposed changes to the spec in order to make the next attempt better? That's like a very token intensive process. But, you. Know, working at the token. Factory three. But I suspect you get a discount. But what you. End up with is this really refined spec that is reliably able to produce the system as it is written, while still providing space and ambiguity for consumers of this ghost library to adapt it to their actual business context, their actual repository and toolchain and issue tracker and all these other sorts of things, which I think is pretty cool because it means that we end up specifying in very tight, high detail the parts that actually matter, the the business logic, critical parts, while still leaving the flexibility for folks to evolve and change it as they see fit. Yeah.

30:55 And what I love about that is the process that you just described.

30:58 And I think we need to learn from everything that we've done in software engineering for many, many years, going into a development. But the process that you just described is a very pure prototype. It's us rapidly developing our application. So we make the learnings we need to be able to say, oh, this work that didn't actually I'd prefer it to be like this, or let's change the UI, which ultimately, you know, breaks a number of flows. So we make those learnings.

31:24 But the difference is because development is cheap, now we are able to say, I can just throw this away. I'm not tempted to to use that, but I'm going to try and distill everything that I have learned and the the functional requirements, etc. into that spec. And I love that it's iterable kind of iterations through to make to, to really, you know, I've seen implementations of spec gaps and this is, this is really good to be able to really fill that in to make sure that the spec captures everything that we need.

31:52 Yeah. It's not like this process is new, right? Like if I think about other areas in my career where I've seen this, right, like partnering with data science, they want to cobble together a model in a Jupyter notebook to prove out the idea first. And once they get something that they are confident in, then it sort of gets handed off to engineering in order to production it, to actually deploy it into the MapReduce pipeline, to operate at scale sort of thing. If we're thinking about doing rapid prototype of a design like in Figma, and then handing it off in order to be production is also the same sort of context. Right?

32:29 But now, because like you said, production of code is cheap, we can actually do that in our production systems. One neat way that we made that idea real in this app we are building is in Dev Electron app. We had a window, a screen that the agent could launch, which would basically give it the native rendering canvas that we're shipping, as well as our full design system and component library, which means it's actually able to prototype new screens directly in the surface that we're deploying, produce screenshots of them, give them to our designer, and now it's a super, super tight loop on does our vision match the reality?

33:11 Are we able to produce this experience in the surface that we're going to ship, which is just wild and super, super cool? Yeah. I'd love to talk about capabilities, and I'll take a slight deviation here. And I'd love to talk about Codex and, the industry perception of, of of Codex on socials, in community discussions and all kinds of things, particularly over the last month, I'm hearing people rave about Codex and moving from, you know, other well-known agents, Claude, and things like that, over to over to Codex.

33:48 And they are they're stating all of a sudden I'm more productive, more productive on Codex or I'm actually it's actually more accurate in terms of doing these various things internally. I guess you're probably maybe seeing this and like, you know, has there been a has there been a moment or has there been a feature or something whereby, you know, Codex leveled up for it because of a number of things,

Rapid model iterations unlock massive PR throughput

34:12 or is it just, you know, the gradual hard work of a of of a group of, of of teams? Yeah. So the, the, the product velocity on Codex is fantastically quick. And we just recently celebrated a milestone of having 5 million weekly active users. Congratulations. Teboe just did weekly resets. You know, go fast go go burn those limits down. I will say that there is a very tight and virtuous cycle between the development of the codex itself and our research team, which means there is this fantastic flywheel on every point.

34:48 Revision of the GPT five series, where the capability just improves in leaps and bounds. I would say big lifts from five 2 to 5 three. Background tool calling. Parallel tool calling means that Codex was able to do more work all the time in order to make more progress more quickly on more complex changes. The unification of the normal GPT five model and the Codex model in 5.4 means that you were not only getting a super powered code producing agent, but one that is also incredibly, generally intelligent a good writer, which means I'm able to start using Codex for all other parts of the software development loop other than producing code.

35:33 The slide deck that I'm going to be presenting later today is 100% written by Codex using the the app connectors in Google Sheets, which is not something I would have imagined to be possible last year, and some super powered stuff that came out in 55 with computer use the browser embedded in the app. These things make it so much easier in order to close the loop where a ton of the jank that we had to put in place in the 5152 era just doesn't need to exist because we are giving more, more chunky, powerful tools.

36:07 Because there is this virtuous loop between the harness and the app that we are deploying, and sort of our research environment, and what it means to get the model to be effective at using these things. It's just a constant march of capability improving. And one thing that I love about Codex is it'll just do the full job. It won't give me much nonsense about it. I can treat it like I would another member of the team, right?

36:33 I'm not shoulder surfing all seven of my teammates in order to bop them on the head. If they do a bad thing, I give them a task. I check in maybe occasionally at stand up, and I trust that I will get a produced PR or several that does the job. And I have the same level of trust in Codex and its ability to be autonomous and complete. And you mentioned Codex, and then you mentioned models and model versions. What what would you say moves the needle most as it updates to the agent or updates to the model?

37:00 It's a mix of both, but I have to say the the new model releases I think are the most exciting bit to me. The, the most sort of powerful levers we have in terms of increasing capability are continuing to train the model. And this is a thing that goes faster and faster over time. We're super excited every time a new model comes out. Getting to test them internally is also fun.

37:26 To be able to kind of see the leaps and bounds of new progress we are able to make. Each revision of the five series from five two onward. I think five two, I think is singularity era. Here with these coding agents has seen increases in PR throughput time and time again. You know, whereas at the beginning of sort of the five two era, we were looking at maybe three and a half PRS per engineer per week on my team.

37:56 Now with five five, we're looking at 70, which is just fantastic. Like that's more than than linear scaling sort of thing. It's it's truly amazing. And is that people changing or is that capabilities a better job a bit or a mix of both? Mix of both. I would say there, as the model capability improves and leaps and bounds like this, there are certainly less effort that needs to go into the enablement step.

38:25 Yeah, the harness engineering gets easier. The same techniques we would use, but the need to build chunky tools in order to attach the agent to the outside world is just things that we need to do less and less. Yeah. To give you an example, in order to get Codex to drive our electron app before computer use was a thing, we needed to boot up a graphical Ubuntu in a Docker container with a virtual display driver and X server have engineers install XQuartz on their Mac in order to proxy over to that headless host and like wire up FFmpeg to this gross contraption in order to record videos of what the app was doing, how the agent had changed it.

39:09 And this is like entirely obsoleted and like a single toggle in the app to turn on computer use. Like so, so much easier. And I expect this to only get better and better over time. Yeah, it kind of reminds me of thinking back to, you know, when I, when I've been in in the industry 25 years now, but when I, when I think back to wow, people actually used to write like bytecode or machine code or binary and it's like, this is the same thing, right? It's like, wow, people used to actually do that and connect.

39:35 I don't need to write TCL in order to test my use UI's anymore. That is definitely an improvement. So let's jump back into into harness engineering then. And let's specifically look at a good use case, the Frontier Project. Tell us a little bit or walk me through what the harness actually looks like for that, including like how I guess, what gets updated, what's in it, but also ownership, who owns that and who changes them.

40:03 Yeah. So a little bit of context. OpenAI frontier is sort of our enterprise platform for agents, and it covers everything from how to build agents with the API agents SDK, the Codex harness, to how you would sort of observe and govern agents running around your enterprise. And it's very, very cool to kind of see all of the harness aligning on Codex for all the things that we do, which means there's a single, highly leveraged interface between the product we are building and research, which means sort of all the post training improvements that we do on the model accrue leverage to every product that we build.

40:41 And right now there's very, very clear interface in order to inject new capability into Codex via plugins. These things are skills and scripts. Same thing as always. I'm talking about context and tools. Yeah, very, very cool that we have ended up sort of in this like IO, CTL sort of extensibility where really the biggest levers we as agent builders, as product builders can give the model give Codex is to give it chunkier and chunkier tools that connect to the actual business problems that we're trying to solve and to do context shaping and management for it.

41:21 Another exciting part of frontier that me and my team have been working on is sort of like the organizational context management pieces to be able to figure out what work is actually done in your enterprise, or what the data ontology is of your data warehouse, and how that is actually used in order to answer metrics questions. And these things have been doing some really, really interesting experimentation on what it means to potentially give each agent in your enterprise sort of a sidecar that is continuously managing context for it, this idea that all agents are coding agents, so we should give them things that are native to that little git repositories that they can grep and search over.

42:01 And all of that is to say, the techniques that we're using to write the code are things that are going to translate really naturally to building agents in the enterprise. And all of that comes from using that base codex harness in order to build these agents. Yeah, yeah. And I guess when we think about, you know, a minute ago, we were just talking about bytecode and machine code and things like that.

42:26 The code that gets created or let's, let's go back then the code that we used to write years and years ago when we did assembly and so forth, that it's really machine readable, less than, less than human readable. Then we kind of moved into the next, next generation of programing languages. And actually we were writing more for, for us as humans, sometimes we actually took the decision to make to write more maintainable, human maintainable code versus most performant code because it mattered.

42:54 It was, you know, less likely that we would add bugs and things like that when we make changes today. And I'll ask two questions today and in maybe a couple of years time. I hate asking questions for a couple of years time when we can barely, barely think about this week. But today, in a couple of years time, is it more important that code is legible to agents or humans?

43:19 I would say if I had to pick only one thing that was legible to me, it would be the reference documentation of the system, the interfaces, and the rendering of all of that into mermaid entity relationship diagrams, system diagrams and sequence diagrams. And that's today. That's today. If I could only pick one right? Still, of course, like I am peering down into the machine code, right?

43:45 I am looking at the diffs as the agent is producing them, but I am able to do that less and less as I gain trust in it, and I am able to do that. I'm able to ignore the code as it is produced for more and more complex tasks. Not all of them, but for increasingly many of them, the things where I find that me and the team provide the most outsized value are on defining the interfaces, defining what the components of the system are, how each component should be structured, and how the code relates to each other.

44:20 Things like wiring up all the system dependencies together and using like Scala cake pattern style like dependency wiring. I find to be very useful to pay attention to, because it actually means that I am able to like draw the lines on that system diagram in my own head and in the reference documentation. Now, you know, sort of the concrete implementation of those things.

44:45 Like, to be honest, I might not even know what language they are in. Very, very funny story to think about those system diagrams of lines. Looking at the local development process, We wanted Codex to be able to connect to our Electron app. Chrome DevTools protocol. The MCP was how we started out. Eventually, I kind of took a peek at the code and it was entirely different. Still, Codex was connecting to the Electron app over Chrome Dev Tools protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of the MCP, because we found that there were only 2 or 3 tool calls that we actually needed.

45:26 So more context efficient was faster, and this had happened underneath me, without me knowing. My workflow was not interrupted. That same dependency edge and entity relationship existed, but I almost didn't have to know how it was implemented. Did you find that cool or worrying? I found it shocking. I'd find it cool. I found it shocking. But also. I found it great, right? It basically means I lived that high trust relationship in a very, very real way.

45:56 Someone on my team imagined a way the world could be better, went after it, and did it in such a way that disrupted literally no one on the team. And because it's the agents that are writing code, they have no opinions around the tools or their structure. As long as we give them things that work and permit them to do their job. Yeah. Super interesting. And I think it's, it's definitely a, it feels to me like humans are going to be the slowest, the slowest here in this in adapting to this kind of level of level of trust.

46:29 Yeah. It's a consistent challenge. My attention my ability to focus across all these threads. Yeah it's it's still something we are contending with. So when do you feel like, you know, in your usage with these product projects, when do you what are the circumstances that you see today where you actually want to go in and look at the code? Sort of if we were to kind of put a sort of like two axis graph together here of sort of low and high ambiguity and low and high complexity in this sort of high, high bucket here.

47:07 I think that's largely two shapes of projects or tickets or epics, whatever you want to call them. Things where I am literally starting from zero to do a net new thing, where maybe the shape of the interfaces, where it lives, what the experience is, I don't know yet, or sort of the most difficult refractors that I want to do, where I either need to break interfaces or redefine them, maybe go backwards and delete code in order to achieve this thing.

47:35 But I don't know what the end shape that I want is. These are the places where I pay the most attention, and this is honestly where I want to be, right? Because this is what it means to kind of like peer six months into the future to unblock my team. It's also, you know, you know, code is cheap to produce here, right? Like, I find myself operating in this space differently than I did a year ago, where I am willing to vibe out a 50,000 line of diff PR and throw it away, only to learn the areas where the agent is likely to fail, where it is going to struggle, putting those interfaces into place that I like.

48:12 And usually I'll actually push up those press, trash them immediately, and then crank out 15 PRS out of that to kind of like do the prep work or do the staging in order to prepare myself for the actual cut over that I want to do. Yeah, yeah. Let's, let's, let's jump to a quote that you, that you made a while back, which is, you call it borderline negligent not to use a billion tokens a day.

Consuming a billion tokens a day demands async workflows

5 Highlight · Consuming a billion tokens a day requires parallel workflows

48:38 It's a strong claim. Who are you aiming to to shake the ladder of the. This idea that the amount of intelligence we are able to extract from the models is in a way, linear with token consumption, I think at the time was quite controversial, but we see it being increasingly true. Right. This is ultimately why test time compute exists, right?

49:05 In order to get the models to be smarter and have more and more rich side effects into the world, we want to move workflows to high token consumption use cases. And I think in order to get to a billion tokens a day, you must be thinking in a grander world than pair programing with the model, you must find ways to be in parallel. You must find ways to put these asynchronous loops into place.

49:31 You must find ways to build agents that are able to side effect onto organizations and teams, rather than just yourself. And in order to do that, we have kind of had to invent a bunch of patterns on what it means to do that, to have automations that are seated from the repository context that everyone in the team can use, rather than sort of single player mode sort of things. And it's definitely not the case that everyone today can say, I'm going to consume a billion tokens a day and poof, magic.

50:01 It happens. There is work that has to be done in order to make that safe, in order to make the output aligned, to make sure those tokens are burned in ways that are productive, that make it to main and improve the end experience that I am shipping to users these sorts of things. And I think at this point in the industry with this technology, we don't really have a durable, scalable, locked in sense of what those patterns are.

50:32 So, you know, to kind of poke aggressively like that, I get to a billion tokens a day is really to kind of challenge people to figure out what it is that's going to work, sort of feels like we have just invented the concept of CI CD, right? And like people are kind of scrambling to figure out what that means. And only 15 years later did this become a thing that's like fairly standardized.

50:57 That is table stakes. And that's, I think, what needs to happen right now with agents and producing software. We need to figure out those patterns. So let's continue this flow then harness engineering. If we if we follow that to its logical conclusion, engineers and

Engineering shifts from code production to systems thinking

51:15 harness engineering is part of every project. Every team developers on that team are using a billion tokens every day. Yeah. Open AI very happy with that. What is an engineering team look like in that scenario? I have found it very valuable to have a diverse team in terms of perspective, experience and expertise.

51:40 I know myself I probably bias more toward back end and infrastructure sort of work you probably see in my sort of profile here, which means that when it was just me on the team, I was producing quite terrible reaction code. Right? We had we had some components that were 6000 lines long, a ton of really bad renderings. These use effect hooks that were like four overlapping closures, like very hard to reason about and to bring folks onto the team that actually were able to say, Ryan, this is garbage really helped.

52:18 And to be able to kind of like have these these full stack teams I find really, really useful. The other sort of thing is I think we no longer need to focus so much as software engineers on developing code production as a skill. Really, what I want to see engineers sort of nurture in their own careers is this sort of systems thinking mentality, what it means to set up teams for success?

52:46 How far can I peer into the future to solve problems, to increase the throughput at which we are producing code and shipping to customers? Probably for me, my time has been split 30, 30, 30 between sort of the hardest refactor and 0 to 1 product ideation, talking to customers and prioritizing scheduling and staffing work.

53:12 Whereas before I was probably spending closer to 50 or 70% of my time producing code. So I am able to step back a bit and focus on these other highly cross-functional, high priority work streams in order to unblock teams of agents from going and doing the code production pieces. And the most important piece, they're talking to users and understanding the requirements and actually mapping those back to the functional requirements of the app, versus thinking too deeply about how an app should be built and the implementation and not thinking enough about, okay, what does the actual user want?

53:47 Rather than building this, this, this back end to be absolutely perfect in its design and so forth? Yeah. And one of the things that's part of that is deciding what not to build. Yes, very, very common trap I see people falling into these days with agents. And particularly because, yeah, coding's cheap. We can build everything. But we need to kind of like stop and ask ourselves, should we build this? Because I guess it's still us as humans that are consuming in the majority in the most part.

54:14 Well, I suppose that there will be these middleware things where I guess other agents and other apps can consume. But when we think about humans and users actually using this, we need to develop at the pace that they're comfortable consuming as well. So it's super interesting, super interesting balance. Ryan, this has been absolutely fascinating and really enlightening to chat with you. So a massive thank you for spending the time to to chat with us today and really looking forward to your session later today, which is going to be live on the main stage and also recorded for, for our listeners to to watch back in their own time as well.

54:53 So Ryan massive thank you. Appreciate it. Amazing. Thank you for having me. This is such a fun conversation. And I will catch you later at the conference and on the live stream. Amazing. Thanks very much for tuning in and hope to catch you on the next episode by For Now. Bye, folks. The AI Native Dev is brought to you by Tessl, the package manager for skills and context. Your hosts are Guy Podjarny and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community.

55:18 And we host monthly meetups at the Tessl offices in central London. Visit tessl.io/community to learn more and I hope to see you there.