AI Agents Still Can't Maintain Your Codebase
While coding agents accelerate feature delivery, unattended models degrade codebase quality over time, requiring structured planning, execution backpressure, and collaborative control planes.
Highlights
3 clips · 4 min of 16 min-
- Unattended models fail to improve or maintain codebase maintainability over time.
- Slop Code Bench shows current models struggle significantly with iterative, long-term code evolution.
- Maintainability lacks a fast oracle, making reinforcement learning difficult for architecture.
-
- Spending 20-30 minutes planning upfront saves hours during review.
- Balancing spec depth helps minimize expected pain and costly rework later.
- Avoid arguing over trivial UI details during planning phases.
-
- Provide agents with browsers, curl, and video recording to test their own work.
- Use curated repo-level anti-slop rules to prevent common agent anti-patterns.
- Feed user feedback and pager alerts directly into agentic PR generation.
That’s all the highlights.
Questions this video answers
6 · every answer cites the moment it was said01 Why do unattended coding agents degrade codebase quality over time? 3 sources · 5:13
Unattended coding agents often focus purely on passing immediate tests without regard for good architecture or maintainability. Reinforcement learning relies on fast oracles, but code maintainability has no fast oracle to catch bad design decisions made by models. Consequently, codebases accumulate technical debt and slop that requires human intervention to clean up.
02 What does Slop Code Bench measure about modern AI models? 3 sources · 6:30
Slop Code Bench evaluates models by disclosing challenges in incremental bits and making the model write code iteratively over multiple features rather than solving the entire problem all at once. This mimics real-world software evolution and reveals that top models score relatively low on maintaining clean codebases over time. The benchmark demonstrates that unsupervised coding workflows still struggle significantly with long-term code quality.
03 How can planning beforehand reduce rework when using coding agents? 3 sources · 9:17
Spending 20 to 30 minutes up front on planning and architecture drastically cuts down the time and friction required during code review. YOLO prompting with a two-sentence instruction usually results in a high probability of having to fix or rework code later. Writing a more detailed spec reduces the expected pain and likelihood of major agentic hallucinations.
04 What practical methods give AI agents backpressure during development? 3 sources · 11:15
Developers can provide agents with execution backpressure by giving them browsers, shell access with curl, or recording tools so they can independently test their changes. Additionally, adopting strict linter rules cataloged specifically against known AI anti-patterns prevents models from introducing sloppy syntax. Providing immediate operational feedback loops lets agents hill climb toward correct functionality.
05 How can engineering teams integrate AI directly into incident response? 3 sources · 11:50
Teams can streamline incident management by configuring pager alerts to automatically generate pull requests that the agent believes will fix the issue. Even if engineers can only successfully merge a portion of these automated PRs, it substantially increases development throughput and system reliability. This shifts the burden of debugging out-of-hours issues toward collaborative machine assistance.
06 What is the role of a modern control plane in an AI infrastructure stack? 3 sources · 13:48
A control plane acts as the centralized system where teams dispatch work, collaborate on active sessions, and share architectural plans. Instead of passing ad-hoc patches through chat tools, a dedicated control plane stores session traces, prompts, and streaming diffs. This setup provides visibility into what agents are doing across the organization in real-time.
Section-by-section summary
3 parts
Unattended agents accumulate codebase slop
Coding agents excel at solving isolated problems but fail to preserve long-term codebase maintainability. Data shows that relying entirely on unattended models leads to degraded code review quality, increased bugs, and architectural regression over time. Standard benchmarks often fail to capture this because maintainability lacks a fast verification oracle.
Best practices for agentic software factories
Successful AI-assisted development requires disciplined workflows like upfront planning, visual alignment, and robust testing harnesses. Giving agents tools to execute code and test their work provides essential backpressure. Additionally, adopting repo-level anti-slop rules helps prevent models from introducing common code anti-patterns.
The modern infrastructure stack for AI agents
Building an agentic software factory involves managing compute, developer environments, custom harnesses, and control planes. Centralizing session traces, plans, and real-time diffs enables seamless team collaboration. This unified workspace approach helps engineers coordinate effectively on complex agent-driven tasks.
Connects to your library
by shared topics
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said
-
Context engineering1× · first 0:43
The practice of structuring prompts and context for coding agents to improve generation performance.
-
Slop Code Bench2× · first 6:21
A benchmark designed to evaluate how well coding models maintain clean codebases over iterative feature additions.
-
Human Layer3× · first 0:49
A multiplayer coding agent workspace and control plane built for engineering team collaboration.
-
12-factor agents2× · first 0:43
A set of design principles and best practices for building and running agentic coding systems.
-
Show Me1× · first 10:50
An open source skill that prompts agents to present information visually rather than in walls of prose.
3,662 words · about 15 min to read · YouTube’s automatic captions
Unattended agents accumulate codebase slop
Uh, it's funny, this is actually uh, this talk was going to have a much longer title, uh, but then I found out it was 10 minutes and not 25, so uh, yes. State of the software factory. What's going on? What's happening? Uh, raise your hand if you're writing 100% of your code with AI. Okay, good. Some of you got the memo, the rest of you, you know, you'll get there. Leave your hand up if you're trying pretty hard to read all of that code. Maybe not every single line, but you're trying. Okay, good.
Um, all right, my least favorite part. Engine did a good job. Uh, I yap about coding agents on YouTube a lot. I think we have over like a million views, probably 1.5 now. Uh, I'll post a link at the end. You don't have to screenshot this. I'll put it up for longer later, but we have a list of everything we've ever written or said about coding agents and agentic AI and 12-factor agents and context engineering and all that stuff. Uh, I am the CEO and co-founder of a company called Human Layer. We build a multi multi-agent coding sorry, we build a multiplayer coding agent workspace. Uh, and our goal is to help everyone be awesome at agentic coding and make it accessible to everybody. Uh, we are building building blocks for software factory builders.
Uh, and we're really focused on an area that I think is underserved is hard problems in complex code bases. You have hundreds of engineers, you have hundreds of repos. A lot of out-of-the-box coding systems, you have to do a lot of stuff to actually get them working in that environment. Um, and I think like anyone can really build it's it's not trivial, but you can build a ticket to PR or slack message to PR pipeline for the small stuff. Our our focus is really like, how do we as engineers and teams collaborate meaningfully on the big stuff? Cuz we're all trying to get this AI coding thing working.
Uh, and there's a ton of hype and noise and FOMO on the internet that we have to cut through. There was this whole loops thing last summer of like just build loops, don't even prompt anymore. Uh, at the same time there was a report that came out in May that you know, since uh, January when we all picked up Opus 4.5 and decided this stuff was good enough, uh, code review uh, quality is way down. A lot of PRs just skipping review, longer comments, more comments, uh incidents per PR is way up. Bugs for developers way up. Um a lot of people will say it's Do you think this is a skill issue? Raise your hand if you think that's a skill issue.
Okay, we've got a couple. Nice. Uh I think the data says otherwise, that data and more data. Um I want to do We talk about software factory a lot. I want to do like a quick little history here. Uh cuz software factory is not an AI thing. This has been around since 1968, uh the same the same year that we invented the word software engineering. Um I'm going to zoom to 2022 just to show you that most of what you see about software factories in AI is actually kind of like the same thing we've been doing as engineers for decades.
Um so, your standard circa 2022 software factory. You've got a team. You've got some engineers. You probably got some product managers. You have GMs or a CEO or someone who's kind of like driving the vision. And you come up with stuff to build, and you put it in a tracker, whether that's linear or Jira or beads or whatever it is. Uh And then someone pulls things off the list, and they go build the thing. And they maybe do some automated testing and unit tests and integration testing. They might pull it up in a browser or poke it with curl or whatever the manual testing steps are.
Make a pull request. And then we run this battery of checks, you know, the the the top-of-the-line software factory was doing, you know, hundreds of CI checks and security scans and all of this. Maybe a human pulls it down and tests it. Probably a human reviews the code before it goes in. If anything is wrong, we loop it back to the builder. If it's ready to go, we send it to prod. We have some fancy rollouts and feature flags and canary systems to like safely get it out to all our users. At which point they use it, and they do what users love to do. Uh they file feature requests. Uh they complain. They tell us what's broken.
And then that goes back to the team, back onto the queue. Uh at a certain point we decide, you know what would be great? What if we had a monitoring system that could wake up our engineers at 3:00 in the morning when something breaks? Uh the getting woken up part sucks if you're the person, but technically it is nice to be able to fix your site cuz it's down at 3:00 in the morning. Um but this is our software factory. Um And then we get to the agentic software factory. Not much changes here. We take this thing we had before.
Um and then we do this hype cycle on it and everyone has one and yes, yeah, I know you did Who has a little ticket to PR agent in their company that they're super proud of? Um no. You should get one. They're cool. Everyone Everyone has one. You're falling behind. Uh no. Uh Okay, so you take this thing and you do The person building the thing becomes agent builds the thing. Um so instead of a human going and doing the code, the agent does the thing. And this part now takes minutes or hours, but this part over here still takes hours or days.
Uh and so we bring in more agentic stuff. We have, you know, agentic code review, agentic regression testing, and that makes the review part faster. You catch all the small stuff, uh but it may still be the bottleneck. Um which brings us to uh this funny idea called the Lights Out Software Factory, uh where we just say, "You know what? I don't I'm tired of reviewing code. This too much. I'm burned out. Uh all the senior engineers hate it. What if we just didn't review the code anymore? What if we just let it go in?"
Uh and like theoretically, if you invest into all this other stuff, you could like probably get uh uh close to a place where where this would actually work.
1 Highlight · Unattended coding agents degrade codebase quality
Um there's some challenges though. I think my core thesis here, and like I'll share some data on this, but like unattended models will not improve or maintain your code base quality over time. And whenever seen like slop in a pull request that someone tried to merge, someone tried to get back, yeah. A lot of slop. Um models got really, really good in 2026 at like solving problems and doing interesting things. I don't think they got very good at improving code base quality. You could tell them to go improve it, but then you have to know what to tell them to do, and that means you have to understand the code base, which means it's not unattended. You're paying attention.
Um there's at least one good benchmark that proves this, um but you probably have this vibe, and if you go catch this talk from June, I gave sort of a intuition-based explanation of like how we do RL and how how the benchmarks are shaped that might incentivize sort of cheating. Uh just trying to get the test to pass without really like rewarding good architecture. Um because the cost of bad architecture is measured measured in months and years. And to do reinforcement learning, you need an oracle. You need a verifier.
You need something to say, "Is this code maintainable or not?" And maintainability has no fast oracle. There's no way to know We don't find out that the code was ugly until 2 months later when someone again is paged at 3:00 in the morning trying to figure out how this made it into the codebase. Um which brings me to a really cool benchmark called Slop Code Bench, um published out of a lab in the University of Wisconsin, uh in the US. And uh they basically, instead of telling the model the whole problem to solve up front, it gives the model the challenges. It discloses like parts of the challenge in bits and makes it write the code and then do another feature and then do another feature, which maps much more closely to how
everyone in this room builds software. You don't design the whole thing up front, ship it, and never never touch it again. You're constantly iterating and evolving and adding things. Uh when they launched this, GPT-5.5 was the best model on the market. It got a 14.8%. So, when you see terminal bench scores and everyone's getting 95% of this saying coding is solved, there is a small lab in the US came up with a university came up with a benchmark that is full very much unsaturated. Um there's a post I can share that is uh a bunch of data. You can go check it out.
Um the labs actually admitted this. Um uh Anthropic did a postmortem. They're like, "Yeah, we had a security issue uh cuz we had a bunch of uh like messy code that accumulated over time." To which the benchmark creator, of course, pointed out that there was a benchmark that would prove that models create sloppy code over time. Uh and even Astra and Fable, you know, we in I remember in the Opus 4 days, uh or maybe even Sonic 3.5, you would write the code and it'd be a little sloppy and you would say, "Uh you know what? The code is slop, but that'll be GPT-7's problem."
Folks, we're at GPT-6, and I don't think we've gotten that much better at this. Um I think there's a lot of reports that like they're getting better at random things like video game slop and and computer use and like uh blender demos or whatever, but that the coding workflows, there's like we're people are online begging like, "Please, someone produce RL tasks about a abstraction and centralization cuz these models still can't do it. Do you remember this guy?
Steve, what is it What is Steve famous for? Gas town, uh which apparently is only good for building other gas towns. Uh but only did he build gas town, he went around the internet telling people for a year that if you're still trying to read the code, you are going to fall behind. You're not going to make it. Uh and here he is it last week. Um yeah, if you don't pay attention to the models, they're going to write slop code. They're going to build something so complex that they themselves can't debug it. Uh so even Steve's come around.
Um and here's the numbers, by the way, on Astra and uh Sol X high. Astra is one percentage point or 1.5 percentage points higher than GPT 5.5. So we are getting better at writing maintainable code or being able to maintain the code we wrote, but not as fast as some people would
Best practices for agentic software factories
have you have you believe. Um so if unattended model code gets worse over time, then for now we should probably read the code, which leads us to what I like to call uh well, they told me I couldn't call it this, but 12-factor factories. Uh I will come up with a better title than that. Uh but we got 12-ish great practices for software factories.
2 Highlight · Plan thoroughly before coding to reduce rework
Um number one is plan before you build. Uh 20 minutes to 30 minutes up front can save you hours during review. Um so we take our software factory. This is a team This is the thing teams figured out before AI. It's if it takes hours or days to build something, and if it takes hours or days to review something, then we better get together as a team and talk about it before we start building, and talk about the architecture, and do what's some kind of sprint planning or refinement or whatever it is, talk to people on the team, and figure it out in the hopes in the hopes that we can carve off some time at review time, reduce the chance we have to rework all this stuff.
Um don't overplan, though. You can easily put too much time planning. I think of this is like if you YOLO a two-sentence prompt, there's probably a 50% chance you're going to be doing more prompting later. You're going to have to like rework some stuff. If you handwrite a detailed spec for 10 for 5 hours, maybe there's only a 10% chance that you have You you do a little bit of polish at the end. Uh or you can go write all the code by hand for 20 hours Uh uh and then there's a 0% chance well there's a 0% chance you have to rework it because of something the agent did.
Uh you might still have to fix stuff. Uh this is why we had code review before. Uh and I think of this in terms of expected pain, right? How how likely will you have to change something and how painful is that change? I'm not arguing with a with with my coding agent in the in the middle of building. Uh sorry, in the middle of planning uh about what color the button's going to be cuz I know I can go change that later. That's not going to be hard. As you got to find your way up and down this curve depending on the task and depending on the code base and depending on the problem you're trying to solve, but planning is about eliminating expected pain.
Um you should align visually. Um we published a skill called Show Me. Uh
there's a bunch of examples of like this is built into Human Layer, but you can go get the open source as well. Uh find ways to let the model give you information in a way that doesn't make you have to read like walls and walls of like agentic paragraphs and prose. Uh talk about how you're going to break down your stack PR's and what how what order you're going to make the changes in. Uh this is open source, you can go get
3 Highlight · Give agents execution backpressure and anti-slop rules
it today. It's probably up to like 20k installs at this point. Um help agents test their work. This one's easy. Give them a browser, let them run the app, let them use curl, let them record videos, whatever it is, like give them back pressure. Um if you are anyone here writing in TypeScript? Yeah, okay, cool. Uh go get this repo right now. This is Dylan Mulroy. There's about 100 anti-slop rules. He literally just like went through and cataloged every anti-pattern that agents put in the TypeScript code base and built excellent rules for them.
Um there's also a bunch of stuff in the Slop Code Bench paper that you can steal. Um you should feed user feedback into the factory instead of like reading support tickets and then turning them into Jira tickets, you should just uh read the support ticket and the PR that the agent made for you. For incidents, when you get paged at 3:00 in the morning, like have have the pager alert come with the PR the agent thinks will fix it. Even if you can only merge 50% of those, that's a huge difference in your throughput and you know, you wake up, you merge the PR, you go back to bed.
Um let models experiment. Uh give them feature flagging tooling so that they can experiment and say, "Hey, here's three versions of this feature." Uh models of course they're not good at UI. They I mean they can make beautiful UIs, but they can't really see them in the way that you and I see them. But if you want to test UIs, give three UIs to all of your users and figure out which one people can figure out. This is how you give people back your models back pressure for UI. Um I think loops are basically a
combination of forward pressure and back pressure. This idea of loops was a little mystical. I think if you disentangle it into like how do you kick off new work? What triggers cause new work to happen? And then how do you give the agent like feedback while it's working so it can hill climb an outcome without having to go to a human? That's like the core of loops, I think. It's how do you get more stuff to start whether it's schedules or automations or web hooks. And then you should compound over time. I think it's only nine. It's not even close to 12. But anyways, compound over time, learn from past sessions, go read what your engineer go analyze the sessions of your team, figure out what they're struggling with, figure out how to get that into your skills or your memory or whatever it is.
Um I think this has some interesting
The modern infrastructure stack for AI agents
implications for tooling. I think the new software forge. You know, has anyone Does anyone remember the days when we used to send get patches around on like email lists? I think I think Linus still does this to build the Linux kernel, uh but almost everybody now uses GitHub. Uh and so we have a centralized the the protocol is is decentralized, but we have a centralized place to do this. Um and now there's other parts of the SDLC that I think are first class. So you have session traces, you have architecture docs, the the AI native version of this get mailing list thing is like I'm talking to my agent and I copy something out and I paste it into slack or teams. My my teammate reads it, they paste it into their agent, they get an answer, they paste it back to me. I paste that into my agent. Hopefully we only have to do one round of that.
Uh but this is the idea of like we need a place where sessions and plans and prompts and all this stuff kind of lives in in a centralized sort of system of record. This is kind of my vision. I think this ends up I'm about to speed up a little bit, but uh
[sighs]
I know I know it's already really fast. Uh we'll I guess it only got a bunch of resources. We go deeper on this in some days like an hour long deep dive on just this slide. Angie says take my time. Uh Um so you have the compute layer, which you can buy or build. Um where, you know, you can Oh, we have some spare pods in our Kubernetes cluster. And uh and and we're just going to run agents there, or you can go buy Daytona or E2B or whatever it is. Um you have your dev environment. How do you get your language set up, your runtime, your preview environments? Do I need to connect to internal services in our VPC in order to even run a dev environment so an agent can test its work? You have, of course, the harness, which you can buy or build. There's a ton of options here.
Um and then there's a control plane uh where you like, how do I dispatch work? How do I collaborate on sessions and plans? Each of these layers you can buy or build. Um Human Layer we're kind of working very closely in this control plane. We're going to open source We also have a custom harness that we really like. We're going to open source it in the next couple weeks. Um Uh basically, Human Layer is a shared collaborative workspace. I can see everything my team is working on. I can jump into a teammate session. I can invite people to code prompt to my session. Not even in the cloud, like it can be running on my laptop and I can pull somebody into my cloud code session. It's got sort of a Google Docs notion-shaped uh artifact collaboration built into the session view.
Uh and then every agent running anywhere, as it's writing code, the diffs are streaming to the cloud. So, we can do code review uh without waiting for pull request. And it's been architected like this. We've been building this for a year, basically, uh mostly in secret uh so that we can do these really interesting building blocks stuff. This cool architecture uh Again, come find me in the hallway. I'll I'll we can go deeper on this. Um come say hi. I'm Dex. Uh There is a list uh humanlayer.dev/resources for everything. It's kind of chronological uh and broken into eras of like, hey, we learned a new thing and we kind of like revised our thinking on some stuff. Um so, thank you. If you want to meet this week, shoot me an email. Uh go check out the resources. Thank you all so much for your energy.