Transforming Debugging and Root Cause Detection with Sentry | David Cramer

11,187 words · about 45 min to read · YouTube’s automatic captions

Vibe coding is a myth

0:00 I think the concept of vibe coding makes no sense. I'm just going to type stuff into an agent and what comes out is like what comes out. It's still software engineering. You're still designing systems, writing specs, making sure the tests are good, code reviewing, so our customers are dev. So we think a lot about what's changing in how they build software. Are there new developers? Do the new developers have different sort of needs? Can we use new technology? This isn't new to AI. It's like any kind of technology. And so Centry has a ton of data. And what we do is we effectively compress and summarize a lot of that data, feed it into an LLM. that OL has a bunch of tools similar to all these agents like read source code and then can it come up with an analysis of this using those tools and that data we're able to feed it in there. I would say historically we have a moat in the sense of we have so much interconnected data about the application itself whereas a lot of other vendors that do application monitoring they operate at a different level.

0:44 You haven't written code by hand in the last 6 weeks but you're shipping production systems on it. What is your development process look like right now? So first I want to start with a very important concept. Hello everyone. Welcome back to the AI native dev. Thanks for tuning back in. Uh today we'll have a fun conversation that talks about developing with AI from these sort of two angles. One is from a um slightly I don't know if it's contrarian or you know pioneering perspective of how to build with AI. uh and the other is from building tools into AI which I guess is also a pioneering frontier in its own right and to cover all of those or talk about those uh we have with us David Kramer who's probably best known as the co-founder of Sentry which is tool I've always uh very much sort of appreciate a lot of the Devex sensibilities and usefulness of it uh and with some sort of deep developer background prior to that. Uh David thanks for coming on to the show. Yeah. Yeah. Thanks for having me. Looking forward just u.

1:45 So maybe just to get us started, can you give us a little bit of background about uh about yourself and then maybe about Sentry just mostly from from the context of this is the world in which you're experimenting with AI and so um you know how uh what is it that you're uh learning all around you?

2:01 Yeah, we've got a I and and we as in Century have a pretty wide breath of concern. So like obviously there's um century is a dev tool and so our customers are devs so we think a lot about you know what's changing in how they build software are there new developers do the new developers have different sort of needs um right now I would say not really um but then there's also the angle of like can we use new technology this isn't new to AI it's like any kind of technology ship but like can we use it to build new or different software and there's also sort of like the green field of like should we be building something completely new that's not just like sort of incrementally adding to century software and then I think like the last piece which has probably been the more interesting topical piece as of you know the recent months which is like how does it actually change your job like how can you leverage a technology to say you know higher shipping velocity or something of that nature and I personally spent a lot of time on that front of like um sort of exploring the tools like I'm not an academic by any

3:02 means I'm like a self-taught software engineer but I think of myself as a really good applied engineer in the sense of like I don't know anything about the academics. I roughly know like what a markoff chain is. Um but I'm very very good at like debugging software and like figuring out how things work and and so I've just spent a lot of time like really in the weeds of like okay like

3:22 how can you push this and use it to the the maximum effectiveness which to be fair is like the same way you would learn how you might apply it to like say your product or a new product or something of that nature.

3:32 Yeah. Yeah. Which is it's the practitioner view, right? you're sort of learning by doing and uh uh and you're bringing in the the competency, right? Like I think in software we often times um there's a blending of the world of like I'm building dev tools and I'm using dev tools which is actually like a privilege that we have that we often forget is a privilege. Uh you know when you're whatever building tools for lawyers you know often times people building it are not lawyers themselves and so there's this split here. So it sort of things like you're you're coming in from the practitioner perspective, you have some subject matter, maybe you have a little bit less familiarity with the underpinnings of how the uh how the uh the AI math uh operates. Uh but you see it in practice. So I guess I guess in that in that context um uh we have this two topics here to to cover. one is uh just sort of understand a little bit with that sort of sentry lens uh what it is that you are building uh as you're kind of experimenting with bringing value uh to your user base you know or like using LLM to do so and then the other lens is just you've been doing some really interesting things around

4:32 how you build software yourself and how you uh minimize uh the uh the manual typing of uh of that coding um maybe maybe we start a little bit with the sentry side just so that we can relate then your practices as to maybe things that you've built. Can you tell us a couple of interesting things you've built?

Embeddings improve error deduplication

4:51 Appreciating these are all kind of early stage or like you know a fastmoving uh changes but that seem useful around how do you apply LLM to Sentry to make it better for its users.

5:04 Yeah. Yeah. So I think the first thing we built which is actually time is fuzzy. First we had co and now we have like this AI thing where like it's moved pretty fast. Um, so I don't know, call it a year ago, we took embeddings. And so Century's core technology, it isn't much. It's like we're a big big enough company these days, so it's not that simple, but if you try to simplify as much as possible, we capture errors. We

1 Highlight · Using embeddings to solve error deduplication

5:29 put them in a dashboard, and then we do a bunch of like sort of processing in between those phases. And one of the things we do that's really critical is dduplication. And that comes from what we call fingerprinting which generally is like a lot of regular expressions or something equivalent to regular expressions historically. And one thing we found pretty early with embeddings you know so you know adjacent was we could actually improve our dduplication with embeddings quite drastically. And so for example, the way a lot of this works is like you'll you'll go into like a programming language. Take Ruby and Ruby will have anonymous functions in the programming language and they have like random integers inserted into the function names or something like this.

6:08 It's been a while so it might have changed but this is what I recall. And so you end up writing a lot of these rules that parse those things out so they actually match together because they have to be static. embeddings actually let you get much closer, you know, from uh uh I guess lower total work. Um and they're not enough on their own for what it's worth, but um so we applied that and and saw like massive improvements early on and we're like okay that's cool and that's how we think about it for what it's worth. It's not like oh what's the cool gimmick we're like we are a real business that is like going to sustain and we want like real applications and so that was a great early application more recently and the thing that actually impressed me

6:43 and so we're like okay how do we apply LM and in our space generally speaking

Traces combined with LLMs enable root cause analysis

6:46 in dev tools I think the most exciting thing about LLMs is just like code generation right because like if you can generate code and understand code there's a lot of really interesting things and that's one area that is like basically not approachable before whereas like fingerprinting yeah we could solve it forever it's just a lot of tedious work. Code generation, not a chance, you know. Um, and so early on we're like, okay, what's adjacent to Century that we could apply this tech to? And we were like, okay, what if we could actually sort of automatically fix bugs or automatically generate tests or something along those lines, right? And let me preface this with you can't reliably do that today. So, this is more of like what if we can get there, which is really important. I think that's the way the and so we started building something in the space and the thesis was Century has a lot of data. I would say historically we have a moat in the sense of like we have so much interconnected data about the application itself whereas a lot of other vendors that do sort of application monitoring they operate like at a different level they operate like at the system level or something it's just not

7:43 the data is just kind of junk if you ever looked at like like something like Splunk or like a log aggregator and you look through the logs you understand what I mean dump all the data in as opposed to some form of qualification proper curation of it. Yeah. And ours is like highly structured, highly semantic. And we're like, you know what? Let's try like interlinking all this data, which this is prel to be fair. And so we we made this trace connected data set. And then we're like, you know what, I wonder like if we could throw this into LMS because they're it's effectively pattern matching um and they do a really good job at large scales of data. What would come out of it? Like how well could it help us diagnose things? And we're building this actually for probably the better part of the last year we've been working on this. And this is our thing called Seir. S E R if you've seen it.

8:27 But

8:28 I remember a few months back I'm like, can we ship this yet? It's been so long. And I'm like, it probably doesn't work still because if you've used any of these tools, they like just make up like they hallucinate, if you will. Um because they just don't have the right information. And so I remember going in and there was a JavaScript bug on one of my side projects. And I'm like, look, let me show you why this technology is a problem. It's not going to be able to understand what's going on here no matter what. Like I can't understand it. I'm an expert in this and I run the thing. It gives me some like really wacky root cause and that's the main thing it does is root cause analysis, not patch generation. The patch generation is whatever. Um, but it comes up with root cause. The root cause makes no sense to me. And I show it to the team. I'm like, look, look, this is completely wrong. This is why it's not ready. The next day, somebody at our company is like, okay, I hit this bug.

9:13 And they show me like the actual error. And I don't remember exactly the details, but I'm like, oh, that's like that's what it said the problem was. And like it was actually right. And I was genuinely impressed because what I will tell my my singular belief is the reason an engineer is valuable. Like go to like a big company like a Google and you have these staff principal engineers that get paid lots of money. A lot of times they're not doing that much in terms of actual outputs. And for a long time in my career I'm like well what's the like why? It doesn't make any sense you know.

9:42 Um and I have a lot of friends that had these jobs and they're very cushy. And then I I eventually came to reason about like the value of that is their knowledge of the system and their ability to like theory craft around what's going on and thus to debug the system very very fast. Um and this can actually do that pretty well. I was like genuinely shocked. And so we spent a bunch of time building this. We're still working on it. Um it's still super early. Like just like all these things, it doesn't work that well. Um but when it works it, you know,

10:09 I I I think uh so so I love the the notion of like uh digesting substantial volumes of data. uh especially when you have it and you have it well structured on it is is is I think like a superpower that's sort of been acknowledged uh and you're right that one problem is hallucinations like it will give you an answer even if it is making it up uh pretty uh creatively um another problem is uh is kind of managing the context window right of like how which bits of the data do you deliver right to uh to uh uh to be analyzed on it um I don't know like was was that problem an issue?

10:46 I guess in general what were the obstacles to getting it to to work? And do you have any insights on almost like um uh the uh the the practices or the methodologies that you think it's the same skills that sort of help to build or make this product uh actually effective uh versus which are new skills that you know people on the team needed to to build up. Honestly, I think a lot of the skills are the same. Like there there was this era early LLM.

11:15 So the industry has always been this way. There's academia and then there's the real world is how I would describe it. And I remember uh the era of NoSQL databases where academia was always present. It's like oh maybe we found a new way to scale databases and it would never work. It could never work in practice because systems are really large and really complex and infrastructure is hard etc. Um that's the same right now. And so when this all started it was mostly like data scientists like building with LMS and there were DAGs and all these complicated workflows and then very quickly it it started to evolve and now we for example the common pattern today is an agent which is just a service we just renamed service to agent for some reason um and it has a bunch of tool calls which are just RPC calls you know to something whether it's internal RPC or external RPC and I actually think we are still a little bit legacy with our workflow for this. it might still be

2 Highlight · Applying map-reduce principles to LLM data compression

12:07 like a an effectively like a pipeline or whatever you whatever these early terms were looks more like a a DAG uh a graph call um but generally speaking it's a lot of like the way I always articulate the best way you can leverage technology today and I think this is important because like if you listen to this a year from now who knows

12:25 it's like map reduce so it's like you have a lot of information can you compress it in a way that isn't losing the most vital parts

12:32 and then can you repeat that process until you get the outcome you need and so century has a ton of data and What we do is we effectively compress and summarize a lot of that data, feed it into an LLM. That LM has a bunch of tools similar to all these agents, right? Like read source code. I don't actually remember what else is in there, but that was really important. Um, and then can it come up with an analysis of this? You know, using those tools and and sort of that data, we're able to feed it in there. And from there, and I think this is true with everything you try to do these days, it becomes like questionably scientific in the sense of you're like, does this kind of look better? Does this seem like it would be better? like it it's there's just it's it's so non-deterministic that it's actually really painful to work with when you're trying to improve it. And obviously we have things like eval that help, but it's it's still I feel like it's still just like you're mostly banking on model improvements at the end of the day and and the tool chain being effective of course like that is really really important and the data being like

13:24 high quality and um high signal and stuff but but it is a lot on the model ultimately

13:30 on the yeah I do find it um entertaining where on one hand like what has evolved substantially is these calculations of massive numbers and math to really really encode data and a very

Surviving eight weeks without writing manual code

13:42 mathematical, very precise way to create these massive models and like spend billions of dollars training them. On the other side, you have vibe coding uh and h yeah it looks right. I'm going to ship it. I can build this feature. So maybe maybe this is a good time to to shift a little bit. So thanks for the taste of it of some of the uh learnings and opportunities that you're seeing in Sentry to uh bring kind of LLM powers to the forefront. I'm sure there there's many more. Let's talk a little bit about your uh development flow. So when we spoke before you kind of gave me the statement that says uh that you haven't written code by hand in the last 6 weeks but you're shipping production uh systems on it. Um maybe take us a bit into this world like what uh what does your development process look like right now and maybe maybe some of the early learnings uh from uh um how it changed.

14:30 First I want to start with a a very important concept. I think the concept of vibe coding makes no sense. It's like this idea of like yolo, if you will. Like there's this whole thing of like I'm just going to type stuff into an agent and like what comes out is like what comes out. It's still software engineering. You're still designing systems, writing specs, making sure the tests are good, code reviewing all this stuff. So I think that's really really important to understand. But um and unfortunately I did have to write code like what day of the week is a couple days ago because I couldn't get it to do the thing I needed. Yeah.

14:59 Version now of uh how long how many days have you gone without writing code? This was uh about eight weeks before I had to do it. And I spent like four days and and just so everybody understands my full-time job is not really writing code even though I I actually do a little bit more for the company now. It's more like passion projects. So even on MCP service, which is a thing that I built and maintaining, it's very valuable to the company right now, but it is effectively a side project. Um and it's a side project I'm helping maintain. Um but that the the MCP service I wrote it by hand on that particularly I like systems don't change software doesn't really change at the end of the day and so if you can reason about how just software works everything looks pretty similar so MCP I'm like okay it's a plug-in architecture it's how you integrate it's a little bit of a flipped funnel it's like I can build the integration and stick it anywhere or a customer can stick it anywhere which is like not typical when you think about cloud service integrations and things like that it to be clear it has a lot of use cases I'm very focused on how it applies to Sentry Um but and so as soon as they announced like the oath spec I'm

16:00 like okay I could extract value out of this now I can insert century in the cursor in cloud code or something like that and it's very very high value and so as soon as that happened I I quickly whipped up a prototype that cloud flare was was uh working on some MVP stuff as well. We pulled that in and I I don't know within a couple days we had something production ready or not. It's it's questionable these days. What is production ready? But it was working totally fine. And so I spent a bunch of time working on that. And and if you use that too, definitely check it out. Um it is you can just drop in a link to a century issue. It'll pull in the right context. If you use Seir, it'll pull in Sears context as well. And the workflow is like it's really really really good.

16:37 And so it's a great experience. But when we think about it that way, it's not like we built an a random Sentry everywhere integration. It's like no, I built a thing that makes it highly functional to use Sentry inside of coding agents, right?

16:51 And it just happens to be an abstract protocol. And so that's how we think about the product. Um

16:54 it's effectively middleware, right? Sort of it's it's middleware.

16:58 It's kind of like middleware. Yeah. because you needed the API you needed the sentry APIs to begin with uh to provide multiple services and then you have the client that already runs and you're creating an intim layer that that does the LLM world translation of here are here's some information client about when would you use me and here's a a means of converting that into the proper API call uh and maybe in the back end

17:24 I would say that's high level but it goes deeper like I'll give you an analogy um MCP actually has two agents embedded inside of it. And so a bunch of the MCP call tool calls are just like roughly translate this input to one or more API calls and then translate that output into one response. Um but now there's a couple search endpoints that exist inside of it. So like find errors or or run um so Century does more than errors. It does logs and traces and all this stuff. And within logs and traces particularly you have a bunch of attributes. You might want to run functions on those aggregations ask questions. Oh, here's a really good analogy in the MC in in claude or uh cursor. I can say what was the total token consumption of X service yesterday and it can answer that through centuries MCP call and that's not just like mapping that to like an API call. That's actually mapping that to a natural language input that then gets thrown inside of another agent that then gets mapped to API calls if that makes sense.

18:20 So it's I would say yes it's middleware. I mean everything's middleware. Um, but you can actually go a lot deeper if you really think about the product experience. And so that's where it gets interesting. And honestly, I think the future of a lot of this is just this like um it's like the embedded agents or agent to agent communication because it's still a lot of natural language stuff. It's still non-deterministic. You can't expect it to randomly write the write API calls, you know. So, um, but we spent a lot of time on that and particularly this is why I ran the experiment of like could I not write code by hand ever again? And that was a lot of time like can I like colllocate the right documentation? Can I write the right context prompts? Can I can I get all all this stuff that will probably improve the system based on how the technology works? The answer is like yes and no. Um sometimes it'll impress you and then other times you're like dude just ignore everything. Um

19:09 and then so like it was a really fun experiment and

19:13 we're we're not there yet is what I would say. But but the techn is really interesting. I I would I would encourage everyone to try an exercise like this if they can. Like one thing I will say is like it will slow you down a lot of times doing it this way and you should not think about doing it this way in like real world critical software like for centuries core codebased you do not write code through agents and only agents kind of thing like you might run an agent to generate a test or an API and then going to go review tweak that thing and that's that's still a big performance improvement or to debug something big performance improvement. M this is like I can take it off, go to a meeting, come back,

19:48 maybe I've got a starting thing to run with, you know. So I I don't know. But but either way, it's a useful exercise to learn the limitations of where we are today and to learn how you might improve upon those limitations because that latter part that's where I would absolutely bet is going to stay true again because like software is pretty reliable. There's no magic in any of this. Like the the way systems improve generally stays the same over time. um they just get easier to improve or maybe need less of this kind of behavior. And so um but it has been a good learning experience.

20:18 Yeah. And first of all, it sounds we've actually done a couple of those versions

The high cost of agent prompt refinement

20:22 uh here in Tesla. We sort of picked a couple of topics in which we build entirely agentic workflows, not to create the code, but actually systems that run entirely agentically uh as we assess software to create specs from it and and and the likes. Uh and it's been it's been, you know, it's bittersweet, you know, you you learn it. Um I guess what are some of the key um uh maybe I don't know if it's a learnings or sort of insights right you're coming into this experiment coming out eight weeks later like uh there's sort of two questions in my mind one is what do you believe now that you didn't believe before right some conviction you've acquired uh and two is uh what what did your job look like uh during that time what were the primary things you did create by hand that you got better at uh over the course these weeks.

21:10 I think importantly like the workflow just to understand what this looks like is you kind of start like I want to solve a problem. Maybe you've got a ticket or something like this and you really need to kind of get it down to like a really lightweight spec of some sorts. I I I don't write anything to be clear. Like this is all generated 100% of this. So it's like even the spec is me describing and refining it. Like actually think of like uh center. It's like, you know, we're like uh I forget what they call themselves, but you're sitting here just

3 Highlight · Why pattern matching dictates agent success on complex tasks

21:37 prompting over and over trying to refine the thing, right? But doing it through human language, which sometimes is ineffective. And so I do that first and then I get into implementation. Um and then usually what you find a lot of the time is it will do okay. And it really depends on the task for what it's worth. And it's not even about the size of the task. And this is you find a lot of like just FUD on the internet. It's just like people who don't actually build software, don't know what the they're talking about, frankly. It's not about the size of the task or any of this It's about sort of the complexity in the task based on the commonality of the task. And so like again, it's a it's an effectively it's pattern matching. And so it's like if if you're doing something that can't easily find like common patterns where this lives next to this all the time, it just doesn't actually know what to do. So it's going to reach for another common pattern that looks like that. And and so you'll find like a really good example where it went off the rails is when I started doing these embedded agents inside of the tools. It's an MCP tool.

22:34 Calling an agent, calling other tools. Good luck ever getting on the guard rails, you know, because it's just it gets so lost in

22:41 words, right? Which it is a confusing thing. I'm like, which tool is which, you know, right?

22:46 And no matter how you try to steer it, it always ends up getting lost again because it it's it's it's a very lossy compression at the end of the day.

22:52 Yeah. Which interestingly is like it's a human problem as well, right? If you were to say you're calling this thing and you're using the word agent, you know, in

22:58 uh the newest fashion of it, you don't know what's going to happen.

23:02 Yeah. And it doesn't help. And and and what I would tell you is again, it's not simple versus complex. The implementation is unbelievably simple. And I can give it a full spec of exactly the architecture of how this works. It's really easy to reason about. It's it's not it's just a chain of function calls

23:17 and it would just still go way off the rails. And then and then you get in these cycles where you try to get it to fix itself. And then my my favorite analogy I give people and this is a very good example of how you cannot control it with context management is immediately when something goes wrong it will try to insert the any type in Typescript all the time. No matter how much you guard rail against using the any type and so I have it in like in the cloud envy at the top of it in the prompts like critical don't use this.

23:44 And if you if you send that in every prompt, it's probably okay, but you don't that's not how you work, right?

23:49 Um, and that's that I think that happens because of how common that is in the training data set. But again, I'm not an academic, so I don't know. Um, but either way, it goes off the rails and then you sit here fighting with it. And this is the problem. This is where you waste time and you shouldn't rely on like pure agent development because like you are going to spend way more time trying to refine that thing through prompts than you would just opening the editor. And so I think alian that's not like a it's kind of obvious that would be true, but you don't really know how frequently it's going to break. I would bet sub 10% of the time was I able to get like a one shot fix in, you know, without like serious refinement. And and even sub 10% where like one or two prompts got it correct. It was it's usually like 20 prompts to get something even remotely good. And again, this is this is real world software. I'm not just generating junk. I needed to be maintainable, testable. Um,

24:39 and then if you don't review it, it's even worse because it will just in like I there was one I didn't review and I

24:44 thought I I did cuz I usually skim this stuff whereas having it generated on side projects. So there isn't uh there isn't like a mandate of uh or like this like this a specific project and so it's not a uh the team won't review it if you don't necessarily. kind of yeah we don't have the same controls on this so we don't require the team to review it

25:04 but even so humans make mistake too right like so I actually review the code but you know it's like I review the code and I'm not going line by line of everything um and there was one where I had to build an integration test and I thought I had reviewed it and made sure it was doing what I wanted to and I went back to it and I found it was like inserting something random that was basically it was not testing what I wanted it to test at all you know and it was fine like the the there were other things that checked this But that's that's one of the fundamental problems.

25:33 And so what I learned through this was and again this is like what you should know but there is like a very real thing like don't be a pessimist is I think a

Cursor beats terminal interfaces for human-in-the-loop workflows

4 Highlight · Editor UX vs terminal UIs for AI-assisted development

25:40 really good rule. You need to kind of like don't just assume truth in things but you need to live some of these experiences. And so like part of this was me. I'm like I'm going to try out cloud code. You know it's a terminal based UI. I don't know how well it's going to work. It's very impressive to be to be clear. Cloud code is like phenomenal as software.

25:57 Yeah. But I think my biggest learning and something that a lot of the internet will be like no no no you don't know what you're doing is like cloud code is actually not a great like this is the wrong user interface for where we are with this stage of technology and what I mean by this is if I use cursor ignoring the agent differences because at the end of the day the model is the main thing that's making the biggest impact. So the agents are close enough to the same. If I use cursor and I run through their agent and I I generate a change set in real time I'm getting the diffs in my editor. I'm able to approve blocks of code really seamlessly. I'm able to jump between these files really seamlessly.

26:28 If I'm using cloud code, I've got this like endless terminal slog. Yes, I can manually approve every single change and review it right there, but it is not easy to review it right there. You're viewing like code editions, not not the actual source code and things like this. And and so my biggest I think point of view is like that is that is wrong. That's not the right customer exper like that is not a good product right now in terms of how it needs to be used by developers in real world software. I think cursor is a much better experience for this kind of thing.

26:56 So what's the I I love the perspective of it and I I don't necessarily disagree like I find this is like a learning uh exercise right now. I guess if you cast your eyes further there's a claim that says well cloud code needs to get better at explaining the changes that it has done because of this constraint of working through the terminal. And so, you know, is it is it in a place in which it would uh uh improve its ability to uh engage with you without needing to review the code changes because of this interface or or does it I guess another lens is that it just commoditizes the idea. You don't really care if it's cursor or the other. You just need some sort of lightweight integrations into the surrounding. Do those resonate or do these come back to like no no this this like there's a couple of years here at least or a few years in which you're going to need to look at all the changes?

27:45 I I think you're going to look at all the changes forever. I don't think this technology is enabling AGI or any of these these things that people are spewing on the internet. They're not removing engineers. I spend as much if not more time doing actual engineering work than ever before with this technology because I am still designing a system like these things aren't intelligent. they're just spitting out patterns, right? And ignoring the academics because who cares? It's about like real life what these are and what they're producing.

28:12 And so, and again, this is why I find the cursor flow so much more superior because I do actually have to review the code. And if I can incrementally review it as it's going through and I can interrupt and do all these things, I think that's important. Like I'll tell you, we when when I was hiring our CTO, one of these things you do when you hire executives is you usually have them give some kind of like pitch or like like go through a deck of some random shenanigans. probably done this with with folks you've hired.

28:34 And I'm like, I hate this because they're always so just

28:38 useless basically. And so I'm like, you know what I want them to do? Because I still want to understand, can they storytell? Can can they engage in stuff? And and so these things you test, I'm like, I want them to just give a pitch on something that they're interested in, like a short presentation, a Q&A and stuff. And our CTO previously worked at Apple for a period of time on the autonomous car division. And and he was interest or at least wanted to do it on on just self-driving cars. And I'm like, that's cool. nothing new entry. I don't care.

29:03 Yeah, definitely cool though. And one sav I super agree with is the problem right now is everybody and and sort of maybe his take was Apple did this wrong is like they just wanted to go full self-driving all the way to automation which you see this in the LM too like everybody's like wants to believe that agents can just run on their own and the right answer is like humans should be in the loop and I think cursor's user experience for example is a much better version of humans in the loop because I incrementally review things so the context for me is smaller you know I have it in the surrounding context I can see the source code I understand. I don't build software through a blog in the terminal. Like it just it just doesn't make a lot of sense to use things that way except if you had full autonomy, which I just don't see.

29:46 Yeah.

29:47 Like I don't see the technology we have today getting to full autonomy by any and it could. Who knows? Like that's the academic part. Maybe it can get there. I don't think they know either to be clear. But um

29:57 yeah,

29:57 and I think that's really important though. It's like you you can't pretend these things can do things they can't do and you need a human in the loop and it's like well what's the best experience if you have to have a human in the loop to me it's not generate a bunch of stuff then go to GitHub and review it and then go back and refine it over and over and to me it's not have it running in a terminal over here and have the editor over here and then click around file you know I just they're not they're high friction experiences

30:20 I I uh super interesting and I guess again like maybe back to the uh we're all experimenting here nobody knows on

Simple architectures and documentation empower both humans and AI

30:26 it I do wonder about uh the definition of the guardrails. So I guess in uh in in in Tesla like there's a bunch of words you were doing that made me tempted to sort of mention Tesla uh uh in those where you're saying you know you write a spec at the beginning on it like clearly that's like our core premise is that should be your sort of the the primary artifact uh uh in your uh in your project and actually we have these uh kind of levels of autonomy levels of a unit of software uh and modeled after self-driving cars and and thinking about sort of the five levels of of autonomy U so we'll kind of write that up and and publish it soon enough.

31:02 But within within that context, one of the premises is well one of the things that would make you not need to review it uh is if you have good tests. And so if your definitions for the things that you've built uh are uh covered or sort of you know backed by tests and you trust the tests which by the way don't get in the way of the IDE commentary. That's a different place of a human in the loop. Uh but once you've defined those then to those you don't really need to get back. Um and I guess I'm I'm curious on you know whether that resonates and whether uh I guess how how did your kind of relationship with tests uh uh change or play a role as you were

5 Highlight · Simplifying tech stacks makes code AI-friendly and human-friendly

31:40 sort of avoiding the coding? Was it like I'm trying to get you to understand something that I'm doing here? Uh and the way to convey that is this test needs to pass.

31:50 Yeah. I mean my rule was I cannot use I cannot edit the code myself. So it's quite complicated. I'm a big test guy. I'm not I don't do TDD if you will. That is like I don't write the test before writing the code, but I don't test the code manually ever if I can avoid it. And this is my entire career. I I refuse to manually test code if I can avoid it. And then we are in the reality where UI is ever present and UI is more code than ever and good luck kind of thing. It's like a real problem. If you've ever worked in JavaScript like you cannot test things like end to end is impossible. It's like really really difficult to get it working. Test harnesses are hard. Most people don't actually test things in an effective way. Um, and so I do have a bunch of tests. I I've even written like um if you use TDScript in V test um I wrote an evals adapter that just implements an eval spec but does it through the test because I need CI to do CI like that again technology doesn't change. We still need the same validation layer but I'm like okay I need it to generate the test and what I will tell you what's really impressive if you can get a really simple system. And so I'm using this technology called ORPC which is

32:51 like um it's like a JavaScript API binding layer like a server side thing that also has a client side binding but that doesn't matter but the server side is so easy to reason about as a human and as a machine of course because if it's easy for a human it's easy for a machine to pattern match against it. Um

33:06 I was able to code generate an entire migration from one technology to the other because the API is so simple to reason about

33:13 and I was able to generate tests for that and it was able to validate everything because everything was simple. it was clear. It was not a lot of boiler plate. And I think that's a really important concept because what I think happens over the next few years, fingers crossed because we need this is a lot of this complexity goes away and we we return to spending time solving real problems because a lot of what's happened in my industry, which I live in this JavaScript world these days, Yeah.

33:36 it's just infinite complexity. I don't see the value gain we've gotten as a as an industry out of it. I don't see better software, more reliable software, faster to ship software. I see more complicated software, things that are not common place being like front and center, right? And I I see people not writing tests. I see people not understanding core systems design. And I got to believe this is going to reconcile that because again, there's no magic in these. The simpler the sort of technology stack that they're working with, the better it is for humans, the easier for us to write code, the easier it's going to be for them to debug, to to build all this stuff. And so I I'm very hopeful that this just implicitly creates more value in that just like it's implicitly creating more value in specs in documentation and stuff which despite what people would like to believe are actually not that valuable to humans because we we learn context and retain it really really well. So you go to a new codebase. Yeah. You might read some onboarding docs. You're not going back to the docs ever. You don't need to read them ever again. You just look at the code and you know what's going on. Yeah. Yeah. But like the docs are actually super valuable to the LMS because they don't retain context, you

34:38 know, and and so it it's just really interesting how the the same things that have always been sort of important to software design or at least as important now, if not more important,

34:48 right? Yeah. No, I love that. I I guess I love both of those points. One is this notion of retaining context versus not like agents know they they retain context. They're also more disciplined in that sense, which is, you know, they uh there's all these sort of running jokes of like, you know, we wouldn't write docs for our teammates, but we do for the agents. And, you know, the obvious reply being, "Yeah, because the agents actually read them." uh and but but uh but beyond the sort of the snarky uh version of it, you make a good point which is like as humans we uh were maybe better at remembering the things that are salient uh while thems and the other the other is this notion of adaptability you know I think a lot about adaptable software uh and I think your point around JavaScript and the mess is like much of that is because of the interweaving of the implementation details and know browsers changing over time and all these subtleties and supporting the sort of infinite sort of uh stream of of historical ones. Um, and and you know, over overcoming the natural way that browsers evolve is they see these quirky patterns and then they create a new way to make it simple and make it clean. But then you have all

35:48 these historical websites that uh that do not have that. And so um it it it's nice to think that it did that higher level abstraction uh into natural language will will reduce that because LLMs can provide adaptability. I'd love to uh take it a little bit into uh into the the the broader team and maybe ecosystem you know in our sort of last 10ish minutes here um and just talk so so you you know you've you've you've taken the plunge right you've sort of uh uh sub subjected yourself to these sort of uh eight weeks of uh non-coding and um and uh and building with it uh and you know you have a bunch of these learnings around you know how do you interact with them what what is it if you think uh today and tomorrow which of these practices would you recommend and you're probably not just recommending but like actually driving the broader Sentry development team uh would use and I'm using that really as a proxy of course for dev team today uh to build

Companies must drop IP restrictions to let developers use AI

6 Highlight · Removing IP compliance barriers to accelerate AI adoption

36:43 what should they do let's let's split those up is like what should uh do you want to see the entire team do today and then we'll talk about you know what do you think tomorrow

36:53 yeah well okay so I think the first problem is this is unrelated to century because we've solved this you need to able to use these tools and you really need your workplace to enable consumption of these tools. Like a month or something ago, I took away compliance on IP at Century in the sense of

37:10 we no longer consider source code protected.

37:12 Like that doesn't mean we won't enforce our license. I I will come after you if you steal our code to be clear. Um but what I mean is like

37:20 I don't care if say cursor accidentally up and leaks our source code. Well, what it's already public whatever we don't care. But I'm also not worried about I'm not saying cursor would ever do this to be clear, but but

37:32 some people will as we've already seen this in the industry. I'm not worried about them training on our data. I'm not gonna necessarily say they can, but if they do the shenanigan that some of the model providers are doing where they're like, we're just going to ask for forgiveness later, we don't consider that a risk to the business anymore.

37:46 And that was a 24-hour decision that I just made that was really, really important to the company because instantly it allowed us to adopt a bunch of tools.

37:54 Not everybody can do that. So that's hard, but you need to really have a workplace that can enable these or you're paying out of pocket quite a lot, which I also do um to do these on your side projects, but you need to be able to use them. I think and that's the first and foremost thing. And so,

38:06 and that is what I'm pushing essentially. I like I did this and I'm like, folks, tell me what tools you want. I'm not going to give most of them to you because they're garbage and they don't solve problems and you can't just have infinite tools and customer data is still a separate concern. Um, but you need to use the tools. And so we gave access the cloud code and cursor to everybody in the company. we give chat to everybody in the company. There's probably some other valuable tools, but it's about like most bang for your buck, but what's most important to start learning? Um, and so I think that's number one. Beyond that, this has been hard. So, the biggest challenge you'll

38:37 face is this archetype of like um a senior engineer that's maybe not pessimistic, but roughly pessimistic that tries one of the Yeah, cynical. Cynical is the better word. And I am this to be clear. They open a a prompt, they type something, and it produces garbage. And they're like, "See, it doesn't work, you know,

38:56 and yes, it doesn't work."

38:57 Yeah.

38:57 And and despite what anybody wants to believe, it will not work that way. And so, you got to get past and you got to force people past that point. Be like, "Yes, it's not going to work." This is how you make it work. And that needs to be like a self-feeding cycle of everybody teaching each other because the name of the game is just to give it all the information it needs to succeed. And I like to explain this to people, which is like if you don't tell it the facts, it will not be able to answer the question. You have to give it the answer right then and there. and and then you'll look at you like that makes no sense then why don't I do the work myself and it's like well what you're giving it is like the page from the docs or something it's like you're not actually writing that or giving it that but like here's the UI go fetch it and then don't do the thing right you have to still feed it all of this information to set it up for success and that's something yes that's obvious if you just reason about how the thing works but also you just like like very quickly come to understand as you try to use it too and then you see wildly different results and so I think it's really like use it as much as possible, really learn how like what kinds of context it needs

39:57 and how to pass it in and then find ways to do that. So a really cool experiment we've been uh running down is like can we expose all our docs in a useful way. So there's this MCP provider that I think is really interesting called context 7. I haven't used it but what it does is take a bunch of like I think roughly open source products and stuff and expose the docs over MCP because you can't just like these things don't know software. They don't know the random things you're using. and they don't know the APIs. They're just guessing and that's what people don't understand.

40:24 It's just pattern matching and you actually have to feed it the right docs for the right library. And sometimes it'll look in the source code or whatever. But um so we've we've taken our public doc. So century docs.entry.io. We've generated markdown files. They're sort of um cleaned up. And so it's like we generate a markdown file for every single page that's at like a dedicated URL that gets rid of a lot of the layout stuff. It gets rid of some other elements. And you can think of this like loom.ext, text, but that's not really a thing that matters in the world. We took this and then we exposed it to our MCP via some tools. One is a search docs tool and one is a git docs tool that basically allows you to find the relevant documentation and pull it down.

41:01 And we did this in the context of like, oh, we actually want to build some agents that will help you instrument code and it doesn't really really if you can make docs work. Um, but we're also running an experiment like could we do this for internal development where we can make our docs available and it can fetch the right information at the right time with slightly less input from a human. And I actually think those investments are going to be very very good

41:25 in terms of like again these going back to the core things that make development better is like we're building better talks or better specs or something and we actually have a reason to consume them now. And so I would I would definitely invest in those kinds of things. But again, I think the your your results are going to be super anecdotal and and not scientific and you know, if you will, you like vibe check these things. I I hate that terminology for words, but like there is no better analogy. Like you actually don't know if it's better or worse on vibes. Yeah, it is. It's not a great term, but it's actually like a fairly great term and like it's descriptive uh to what you're doing. I guess I I guess kind of on that uh on that note that's what I was going to ask which is like how much uh do you

Boilerplate generation and re-engaging senior builders provide real ROI

42:06 expect to see results and kind of real world impact uh today versus uh how much are we are you doing this out of conviction of I'm building a muscle that I think will be impactful not in three years time but maybe in six months time or 12 months time. So I think it spoke. So there there

7 Highlight · Where AI coding assistants deliver immediate productivity gains

42:25 I I'll give you a real quick anecdote of like why I think this is impactful today. It has been for months. I started off like when cursor had just started and it was mostly just type ahead. I'm like folks it's just type. not there's no magic here. And I tried to and I think that like composer, which is their agent, was still very new. And I'm like, okay, look, I'm going to try to not be cynical. And I'm cynical by default because like crypto bros and like the whole internet is full of people grifting all the time, right? And so cynical by default is actually a totally reasonable way to be in life. Um, but I'm like, okay, I'll try it. I tried cursor

43:00 and I went through a normal workflow. a normal workflow in any in any company in the world that if you are serious is like I have an API or something like that that looks the same as all the other APIs I need to build another one or I need to like I need to build another thing that looks exactly the same as all the other things right like that is like software engineering in a nutshell and so I'm like okay I did that but I I tried to do it using composer and stuff so I'm like okay I need to build this here's here's an existing one that you can model it off of and it roughly did it and then I'm like okay and I need to generate a test here's an existing one and it roughly did it in fact the test it did better than I would do because it just it added additional branches that I probably would not cover just cuz I don't care because they're like not you know exactly

43:40 less lazy than the developers.

43:41 Yeah. And but it was successful and it was basically what I would have done. It copy pasted the file. It changed a bunch of stuff to match a new thing. That would have been my flow and I'd roughly time myself like how long this would take me versus how long it took me to do almost an identical task through this and it was significantly faster. Like I was able to get like two to three times more output from that. That is not the same as prompting cloud code to build stuff for you to be clear.

44:03 That was me like using the agent to do boiler plate working that and this is months ago. That was already super effective. The type ahead super effective. So I think value's been there already but it's a different thing. Um but what I will tell you also is there is a an sort of an implicit value that

44:18 is happening right now that is not talked about as much. people like me, uh, a couple founders that we acquired the companies of that, uh, one one's newer, but he's the same persona, I'm sure, but, um, this one founder of Century, his name's Inji, got interested in this for a long time at Century, he's been kind of, um, managing teams, if you will, like kind of like me at Century, not really writing code for the company.

44:43 And this piqu his interest, as a lot of us it is, and now he's back in the weeds. He's building stuff and and whatnot. that is not talked about as much but that is like so unbelievably important.

44:54 Yeah.

44:54 And you see this persona of people that are builders that are curious about technology like the core people in my opinion that drive this industry are now like backed in the weeds. They are excited about change because for a long time this industry has kind of been stagnant. It's like the big companies out in the world. There's not much you can do about it. We solve most of the hard problems in in all honesty. And so I think that's like a really really important thing. And a lot of this comes from like this excitement and this passion and whatnot which leads to like genuine ROI in businesses because it leads to thought I guess if you will. um that that's that's like immeasurable in my opinion and that's a really important thing that you just got to enable people to to dig into the technology and and what it translates to for people like me is like I will happily

45:35 my wife less happy about this for but I will happily spend my evenings working on random things because I'm like actually excited about technology aid you know and and again immeasurable but like unbelievably important and and so I think you get all of these things some practical value right now today if people understand how to use the tech Okay.

45:53 Right.

45:53 And some sort of value that's hard to really analyze from people sort of how they think, how they make decisions.

46:01 Time more on the uh the deep thinking on it that you might produce the same lines of code like it's going to be hard to doing it. They would just sort of be better and maybe you also just energized. Uh uh I think energized

46:14 would otherwise like uh be sort of a bit tired of the grunt work. Uh

46:19 yeah, but I think it's also like you have people in your company that make decisions. Doesn't matter where they are in the stack of your company, the more they can see what is happening in the world of like technology trends and stuff, the better they're going to make decisions period. That's just a source of data, right? And this has always been my strength at Century is like I am always in the weeds. I will always have strong signal out of data. Data is not surveys and by the way. Data is like me living the world. Um, and that that's the value I see is like people are like in the weeds again, which just means they have better opinions. They have taste. They have all these things that are just like so so so important to building like great products and great companies. And so that is like the most important thing about getting people involved in using the technology. But again, you can also get speed ups if you just use it in a way that's rational. So

47:04 yeah. Yeah. Yeah. And you build those out. I think uh well, there's like uh a dozen more questions I want to ask you about, but I think we're out of time. I I'll ask one more sort of quick one uh uh for the broader audience here. um you know we've kind of gone from your practices to sort of development teams as a whole and how do you build them for the individual right if you're if you're a developer out there uh and you're you know you see the transformation to your own profession happening I guess what's the what's the primary tip that you would uh give someone uh in this state right like what's a what's to uh to like do or don'ts uh uh uh to uh adapt.

47:49 I would just say embrace it. Embrace it. It's fair to be skeptical. Don't listen to everything on the internet. Don't listen to any CEOs of of big companies that are like spewing out nonsense. Um but yeah, just embrace it. Like it's not it's not removing engineering or any hogwash that's going on in the world. Like I do just as much engineering now as before, if not more. But just embrace it. Like you can't ignore it. If you ignore it, you're you're dead. like dead in the water from a career point of view, from a company point of view. You can't ignore anything that's happening right now. I think that's evident with how every single company in the world has shifted in this direction and they are very smart people running those companies. So, but yeah, just embrace it. Embrace it. Find a way to like spend time using the technology. Ideally, you can do it for work. If not, maybe find a job where you can. I'm not sure. So, but yeah.

48:34 Yeah. No, good uh good tip. Very aligned with my uh view on it. you know, you can't deny it and uh you need to build up the muscle even if you're not feeling the return of it uh uh immediately. Um hey David, thanks uh thanks again for all the uh the insights, the learnings uh of it. Uh looking forward to uh continuing to hear your opinions. I love the sort of the uh the sound bites and such uh you put out on uh on the social media and uh and the conversations of it. Uh and yeah, there's a you know pioneering space. So keep on doing it and uh looking forward to to hearing more learnings.

49:08 Yeah. Awesome.

49:09 And thanks everybody for tuning in and I hope you join us for the next one.