Oxide and Friends 1/12/2026 -- Engineering Rigor in the LLM Age

19,217 words · about 77 min to read

Software engineering faces an existential dread over LLMs

0:00 Last week he made us wait for like four minutes. Well, my man in the office says, "Hello, Adam." Says, "Brian's here." Hey, Brian. How are you? I am doing well. How are you? I'm doing very well. And we've got all the oxide friends here. We've got David and Rain. And Rain. Great. Um, you know, if our predictions episode was only a week ago, and yet it already feels like at least one of our predictions already feels like such a lock. It's amazing that it was even even considered a prediction as little as a week ago. Uh I think this oni the software engineer on and then your absolutely brilliant naming of deep blue for this sense of software engineer on uh wondering what the real purpose of anything is that the LLM could just do everything for them.

0:49 Uh it feels like this has already taken root in the last week. Is that my imagination? Feels like this has been

0:54 I think it uh I think I don't think it's just your imagination. Uh, I think it may also be my imagination, but when I saw someone uh not even tag us, but but just describe the feeling as deep blue, I was like, "Wow, this is this is really getting we've really made it." Yeah,

1:11 we made it. We We've definitely arrived. I you know how ironic would it be if that cease and desist came from IBM for for naming for for sullying the good brand of Deep Blue into a kind of like

1:25 I'll tell you the predictions market does not have that one coming. So

1:29 exactly let's see Deep Blue disambiguation on the Wikipedia page when they actually need to

1:34 clarify that we're not talking about the the the the software engineering neo depression LLM based depression.

1:39 Yeah. If we see the uh if we see the the poly market uh on that spiking, we know that there's a CND coming and some insiders are profiting on it.

1:47 Well, it's my understanding those insiders are supposed to be us, right? Isn't that the way isn't poly isn't that what poly market isn't that sorry, isn't that who they serve? Isn't

1:54 I think in this case it would be it would be the folks at IBM about to sue us. But yeah, I mean that's that's basically

1:59 Yeah. Can we take out a position on getting a CND? Is it they've got CN over the course there going to be Oxide and Friends? Uh I mean old conventional wisdom oxide friends bingo car. New conventional wisdom oxide friends poly market.

2:12 Yeah.

2:13 Um there's got to be a good hedge, right?

2:15 Yeah.

2:15 Yeah, it should be. Uh but it it feels like this has really been uh I mean we knew this last week, but it just the presence of LLMs. What does what do LLM mean for software engineering? I feel like I've seen like six different pieces a day on right about talking about like what does this mean? What does it not mean? And I feel like

2:39 there's there's quite a bit of of noise out there. Um I well noise. Um

2:47 there's a lot of consternation out there. That is for sure. Uh it this is a uh this is an issue that has a lot of people thinking about it one way or the other for sure. I definitely learned that the I mean there is a demographic and if you it's hard to say because like these all right look if you're in this demographic you are going to think that we are belittling you by making other people aware of this. I

3:14 I just want to pause right now and all listeners please write down what large group of people Brian's about to alienate.

3:22 Excuse me. I'm being handed this folded piece of paper. Listen, I just have I I I've st the mortgage payment on a CND. So, I'm really I need that. I really need to get the goose this thing along. So, really trying to after our you know, we we we tried to get a a CND from the Republic of Germany last year after offending all of our German listeners, but we uh that nothing came of that. So, um, there are there's there's a virulently anti LLM demographic out there

3:51 and and I like I at like I I get it. And uh, but that's what not what we're going to talk about. I guess we already are. Sorry. Whoops. Um,

4:02 I was like, wait, those folks? You already alienated all those. Like,

4:05 all those folks are they are that Hornets you've already kicked many times. Like do you not follow yourself on blue sky because

4:12 you maybe you should.

4:13 I know I I also feel like I'm doing you know I had a boss who would do this once who'd be like listen we're going to go into this meeting and I want no one to mention insert name of former customer like we're just not going to talk about them. I'm like I wasn't going to bring them up and then like the very first I want to explain to you why former customer is no longer a customer. I'm like okay didn't Okay. So oh I get it. When we were in the car ride down you weren't talking to us. you were talking to you like the part of your brain that tries to not screw everything up was trying to talk to the part of your brain that actually in fact screws everything up and that part of the brain wasn't listening as it turns out.

4:50 So I kind of feel like same thing for me here on like

4:54 nobody bring up the fact that there's a demographic that believes that LLM use

The false dichotomy of vibe coding versus total rejection

4:59 is immoral. I will do that from the top. So no I'm sorry.

5:03 All right. Well um this is a hash. We can cut all this out, right? This is

5:06 Yeah. Yeah. Sure. Sure. That's all.

5:10 Um but we are to the contrary what we want to talk about today is um the what we have seen is that we want to split this kind there's this false dichotomy out there that you are either vibe coding a term that I again I believe is not going to survive the year a prediction that may not be not be fairing that well in its first week evaluating our predictions one weekend. um the uh that it where you have a fully closed loop and an LLM is just simply creating software of its own valition. Uh that is kind of like that that is one poll and then the other poll of course is like no no no these things are like you should never use them they shouldn't be used for anything they're you know etc etc etc

5:53 and correct me if I'm wrong but I feel like a shiblith of vibe coding is this idea of like you just do it and if you don't like the results you do it again and if there's a bug you do it again and you're never sort of cracking open that nut and like seeing what all the gooey middle is you're just like just go for it like kind of a lack a a toological lack of curiosity and about what's going on inside.

6:18 Yes. Well, this which is part of the reason I think that the term will die with this because I think the term is going to be associated with that lack of curiosity. But yes, absolutely. And there are um domains in which that lack of curiosity may be okay but other domains which it's not. So that's kind of the the the kind of the two polls and I think what we have we what we believe what we have already seen is that there is a big big big middle ground and in particular what we have seen is LLMs are can actually be used to result in more rigorous engineering uh and it's

AI excels at structured, tedious kernel refactoring tasks

6:52 actually not even that hard. Um, I think that there there's and I've got

6:57 I've got some specific I my some specific and recent experience. Um, Adam, maybe I I could lead off with that before I introduce our colleagues. Um, so I have been exploring uh using cloud code to do kernel work to do um we've got our host operating system is Helios. Uh it's an alum derivative and I had some like what I thought was a good you know you always have to want to have like a good first task for these things just like when I picked up Rust I wanted to find like the good like what is the right thing to try Rust on um my first thought of a doubly linked list ended up being that was the

7:33 that was the wrong idea that was the worst thing so okay let's let's not do the worst thing let's do a different thing um and I mean just and actually you know you kind of had the same experience with Russ of like picking up not a great first thing although not deliberately right I cuz you did a Sudoku solver.

7:47 Yeah. And a grammar. Yeah.

7:48 And And I mean, how And how was that as a good as a first Rust project?

7:56 Um it was like

7:57 I think it was very early for Rust.

7:58 It was early for Rust. It was early for Rust and just the work that has gone in the intervening like 10 plus years.

8:06 Yeah. uh to making it approachable and the error messages um sort of like convergent rather than divergent. Like I think my big frustration was it was like go go try this and it's like oh wow that's much wronger. Like don't like who told you to do that? You told me to do that. What are you talking about?

8:23 I didn't tell you. I don't even know I don't know what you're talking about. Yeah. Exactly.

8:27 Um so but you but you also so I mean in hindsight would you today would that that would be a fine first Rust project. There was nothing about the project itself. Yeah. Yeah. Right.

8:38 Yeah. Yeah. I mean it was it was it was even simpler than the thing you that ended up being your first Rush project,

8:45 right? Um so you always want to have a good kind of first thing for these things and I've been kind of waiting for a good like what is a good thing to use cloud code on because I just want to like see how it does basically on this stuff. Um and I had some like some relatively straightforward scalability work that needed to be in a lock that needed to be broken up. I knew I how I wanted to do it. Uh it was going to be a little bit tedious. Um but I was just kind of curious to see how it did. Um and

9:13 and it should be said that the idea here also was like you you're breaking up this lock in a way that many locks before it have been broken up. Is that fair to say?

9:21 Yes, absolutely. that there's actually like what needs to be done here is really quite straightforward and I can describe it pretty pretty completely um to to cla um and I I'll I'll drop a link to to the actual uh the the actual bug itself um alumos 17816 um so I'll drop a link in for that and so you can you can see exactly what the problem was at hand pretty straightforward Now, I was going to use and like very deliberately not using it.

9:56 I'm definitely not closing the loop, not

1 Highlight · Using LLMs for complex kernel refactoring

9:58 vibe coding it, not oneshotting it, but really um because in particular like I am not we're not even I'm not even going to let it build anything, right? I'm going to let it we're going to go into the source base and I just wanted to see how it did and uh really did like it did remarkably well. Um, one thing that was really interesting and I would and but not I mean not definitely not perfectly and had some subtle issues that needed to be resolved but we got those resolved pretty quickly. Um, and I think I would say like it had two subtle issues but it had also did not have a subtle issue that it could it made a subtle discovery as well. And the thing that was really interesting to me about it is I was unleashing it on like a a pretty big source base in terms of Lumos and it was really interesting to watch it effectively read block comments to understand how subsystems worked. um and to understand it and so reading not just code but also comments um and all in all it was really pretty impressive I you

10:59 know it it definitely understood I mean it's we're talking Lumos here so it's like this is not like anything that you have trained on that is the Linux kernel or the BSD kernel is like literally not going to apply it would be very easy for you to create arguments to to functions that didn't exist I'm talking about the kstat facility which is a facility that doesn't exist in so it's like you cannot rely on on something that you've really trained on. You're going to have to kind of look at this. Um, but it was good and I would say like net net it probably saved me probably about in terms of the like the actual um time to implement this. It probably saved me like half the time. I spent about two hours on it to have something that was I was pretty confident would work and did work. Um, versus I think it' probably be about four hours on it. Um, and someone's suggesting well how many LM train.

11:50 Yes, but what it was do you the the way it was iterating uh and if folks haven't used cloud code it is really um it's worth experimenting with especially on an established source base and so one of the things that I would just like to throw out there as like a first way that these things can help increase your rigor is by asking questions about a source base. Um and clearly like you know all of the caveats apply that you can get the wrong answers and so on. you need to verify these things. But um it it was really made it much more made the t it figured out a lot of what needed to be done um surprisingly quickly. So um I will absolutely be using it again for other kernel projects if only to as a starting point to uh and I I I you know one of one thing it did it was funny is Adam it needed to add a field to a structure and this is the actual structure itself. None of the fields is commented. You know how we like best practice would be to comment every

12:51 structure member. And in this particular source file, none of the structure members are commented. And the what its proposal was was to actually comment the structure member. I'm like, uh, for bad reasons, like we're not going to do that. We're going to be consistent with what's there by not commenting the new member that you just added. Like the code you want to actually write is actually cleaner than what's there. Um, but it does think it the other kind of thing that it it brought to mind is like, boy, there's so much like technical debt kind of things. And one thing I think would be interesting that surely we're going to see is people going into existing source code and

13:23 commenting it better using Claude that just have better comments then obviously validating all of its work and and you know not allowing. Anyway, that's that that's kind of my my story um from my experiment over the weekend doing a Lumus kernel work and uh came away pretty impressed.

13:39 Awesome. That and did when you started that project, did you have a sense of what the code was probably going to look like?

13:46 Yes, I definitely Yeah, I mean like this is one of these where I in in many ways I had biased it for maximal success. I knew I had a pretty good idea of what it was going to look like. Um, but there's also some fiddly bits that people, you know, look at the uh, and I actually I'll put a a a link to the diff into the actual bug. It's like there are some fiddly bits to get right. Actually, there's a little bit of math that needs to be that you need to do correctly. It's not um, but yes, I definitely knew what the code was going to look like.

14:14 And this is it doesn't span multiple files. We're not introducing a new subsystem. Like this is pretty straightforward as it goes. So this is I would say in a a relatively a case that is that I really picked because it's kind of biased for success. Also picked it because we need to do it by the way. I mean that's the other thing. It's like this is like

14:34 this was not a yak shape. This was like you were you were doing it in four hours or you're doing it in two hours. Either way

14:39 either way it had to be done. That's exactly right. I would say the other thing is that the the 4 hours versus 2 hours ends up being really uh actionable because I started this at 10:00 at night and it was like there's a pretty big difference between going to bed at midnight and go to bed at 2 in the morning. You know what I mean? in terms of so you know sometimes like that that difference can be um so anyway it was uh pretty impressive um and gave me uh the belief that we could actually use this in lots of other places but that is my limited experience I I want to I definitely want to so we've got two of

LLMs act as pattern amplification machines for repetitive APIs

15:16 our colleagues here um we've got we've got David and Rain here and both of you have used LLM quite a bit and have discovered I would say new vistas of of rigger. Um the I Rain, do you want to kick us off on on some of the the stuff that you uh that that you've done where you found this to be useful?

15:39 Uh sure. Um so there's a couple of different things I can talk about here. Uh one of them is kind of the first work that I did. I was around May of last year and then the other one is like the work I did around December with like reorganizing types and stuff. which one should I go with?

15:57 Let's let's actually start chronologically because let's start as you're kind of getting into this stuff. Um

16:02 yeah.

16:03 Yeah. Um yeah. So I guess you know like as as you pointed out the a lot of the memes around uh you know LLM based coding are uh you know vibe coding right you don't pay attention to the code you just like let yourself in the flow or whatever right um that is uh I I have to say personally speaking that is kind of uh exactly the opposite of the way I want to build software

16:29 um and and and you know for me like I want software to kind of you aim towards correctness. I really want high degree of rigor in my software. So when I came into LLM, I came in with like a huge amount of trepidation like I was like really worried about like you know I was just kind of trying it out right and and I was like okay you know I want to make sure that everything looks good and so on. So um the first use that I found that I thought you know was kind of really impactful was um so I wrote this uh we were having a bunch of issues at work around like you know how do we store u keys and values in maps and so uh I'd kind of on the side around like April or so I kind of started prototyping this u this approach uh which lets you store u which basically lets you store keys and values side by side next to each other. And um I spent a few weeks, you know, kind of trying that out, right?

17:27 Like and and you know, I I did a bunch of prototyping. I I did I did a bunch of work. Um and then, you know, um and then uh it was it was like an interesting experience because that was all handwritten, right? So it was like 3 weeks of like around 2,000 lines of code like carefully handwritten like there's a lot of unsafe code and um and you know it was it was like pretty challenging. But then um I I realized that one of the things I needed to do was that if you define a map in Rust, there is like a lot of extra there's a lot of things you need to add to that map in order to make that like a functional API. So if you look at Rust like hashmap or B3 map or whatever, [snorts] uh there's like you know there's a ton of different APIs that are all like some of them are syntactic sugar, some of them are more primitive. An example is like say the entry API which if you've if you use Rust maps you might be familiar with the entry API. So that's an API that lets you kind of say whether an item is occupied or not and it lets you insert

18:27 an item. I I think it's a it's a beautiful design but it is a very uh verbose design

18:33 and

18:34 and so the map library I was writing and I'll just drop a link to it. It's called uh IDQD. Uh this map library had four different maps, right? And so one of the things I was dreading was, okay, oh my god, I need to write like all of these map APIs four different types, right?

18:54 And that is just like terrifying. So, so it's like okay, you know, you have a prototype and maybe you have like one of those types, but then you have like these, you know, three other things and for each thing you need to go in and like, you know, update the the map type and it's just like it is um it would be like a couple weeks of work at least and it would be like pretty hard for me to justify that work uh as opposed to kind of, you know, just like amling along with the default maps. Then I really also wanted to get this in my in the hands of my co-workers because I actually really excited about this pattern. Um so what I ended up doing was that I ended up handwriting one of the maps and then I told uh I think back in the day it was like sonnet 4.1 or something. Right. So this was you know we were like couple couple generations before. Right.

19:43 Back in the day of like eight months ago.

19:45 Yeah. Right. Right. And and so I just told it to kind of replicate, you know, the same APIs across all of the other maps right?

19:54 Yeah.

19:55 And it just nailed it, right? It just

2 Highlight · Amplifying code patterns and doc tests

19:58 like um it it just like it it like, you know, it like there were like local differences to things. It kind of adapted the map types to those differences. Uh this was like I want to say a total of around 20,000 lines of code. Um then I asked it to generate doc tests and and you know like one of the one of the things you should do for and if you look at say the rust core types like you will see that like every method has a dock test associated with it right and so you know I wanted to get that kind of rigor right where like every method has a doc test associated with it and I I don't know about you but like I hate writing 5,000 lines of doc tests right and I [snorts] just told the LLM to do that right I kind of you know I gave it a couple of examples to start with and I just told Sonnet 4.1 I think to do that and you know it just kind of replicated that the things it it wrote like thousands of lines of doc tests and you know this work that I'd been dreading because it would be like weeks of work it took me like I want to say like less than a day to get like the

20:59 whole thing ready right so it was 3 weeks of careful deep analysis and work and like thinking about unsafe and so on and then like one day of um some I was talking to someone on blue sky about this and um I think they described it as like a pattern amplification machine where

21:18 interesting right

21:19 right and so so you give it a pattern and it just kind of amplifies that pattern into the rest into you know whatever like degree you want right there's like you know I I spent like the thing is that before LLMs I would have probably like I would have like investigated like a code generation library I would have like tried out macros or whatever and all of them have like some downsides uh the the kind of the LLM kind of doing things and like tweaking things locally as it went along and like you know things like for a B tree map it'll say like ordered and for a hashmap it won't say that just like you know making sure that the documentation is all aligned and everything uh it was like that was my first experience and it was like a great experience where like it wasn't a oneshot but it was like I want to say like maybe like five or six prompts

22:09 total and it just kind of just nailed it. And so that was my first experience.

22:14 So a b yeah yeah a bunch of followup questions. So that's really interesting. So one I mean this is I mean this is the kind of tedium that you do kind of like just like you say about the doc tests. We all know the doc tests are great as a user of something you really appreciate them

22:28 just takes a lot it takes a lot of time to like to get that working correctly. It's really easy when you as a as a human are I mean like bluntly cutting and pasting right as when you are cutting and pasting

22:43 it's super easy to make a mistake where it's like oh that doc test by the way have you looked at the doc test like that actually you just cut and pasted you you changed it in two places but not the third and so now like what you have is kind of nonsense in the test like well that's that's not very good. um like or the test is testing the wrong thing, right? Like they're testing the wrong method or testing the wrong strct or whatever. Like it's so easy to make mistakes here.

23:05 So easy to make that mistake. Yeah.

Rigorous code review requires a heightened state of alert

23:07 Yeah.

23:08 Um it's okay. So another question I have for you because the other thing is that when you are I mean as you say it's I've got the pattern I want you to replicate it. It also makes for a code that's pretty easy for you to review. Are you like this is kind of reminds me of my experience like I I I pretty much know exactly what I'm expecting here and I'm going to be able to review this pretty quickly. Rain, one question I've gotten for you because one thing that was super surprising for me is and like look maybe hopefully I'm in a safe space here.

23:39 Like you you I've got the the brain that I engage when I'm writing my own software and I struggle to engage that when I'm reviewing someone else's software. you know, I try to and and the best reviewers, I think, are able to review code as if they themselves are writing it. And I think I I I but to me, like I really have to work on that. And I definitely know when I'm in the like, yeah, yeah, this probably works mode, my brain

24:07 versus the like, no, no, wait a minute, like this, like I I I need to like I'm in like doing my checklist before takeoff and like I'm going to die in this airplane if I don't get the flaps down correctly. So I'm like and the thing that was super surprising to me is that when I was reviewing Claude's work I was in that mode of like I'm writing this myself and like very a heightened state of alert really reviewing things closely finding some subtle things that had script. Did you find the same when you when you were reviewing the code that that it had written? Um, in this case, I think so I had I have I have the same struggle that that you do, right?

24:44 Like where I'm like, you know, when I'm reviewing code, especially when I'm on like look on github.com. Uh, the I'm sure we all have our complaints about the GitHub, you know, I'm sure. Right.

24:56 Yeah. Like, oh, um, by the way, like here, let me show you all the trivial stuff. The non-trivial stuff. I don't know. That's a lot of file. That's a lot of lines to render. Let's not review that.

25:04 Too big. Why Why bother?

25:06 Why bother, right? Um, so I I I had a bit of the same exper I I I feel like I was kind of somewhere in between here where um I think much of this depends on how or at least for me depended on how intensely you and the LLM were pairing with each other, right? So I've had experiences with an LLM like so for this for this one of an LLM, I just like, you know, it just was doing its thing and I was not paying a huge amount of attention. Uh and then I ended up like reviewing it and and you know it it like made like maybe two or three mistakes, right? Um but like also like I feel I felt like you know I was pretty assured by the fact that uh all the hard bits were kind of handwritten and then you know the LLM was just like wrapping those hard bits right um so it was like it it was doing like relatively easy things. Um, there have been other things that I've used LLM uh for and especially like Opus 4.5 over the holidays and that's uh for those ones like I ended up having it

26:06 like this very intense like mindmeld pairing session and like that felt like you know I knew every single line of code and what it was doing right and so I was like you know carefully kind of working through things and that was like a wild time but like I felt like it depends on kind of the mode I end up using And so so you know it depends but I do you know like even like the current LLMs and and again this can change because I know I know things have advanced so quickly but even current LLMs have um they they they do get things wrong or they do things suboptimally or do they do they think did do things in a way that's unmaintainable and you do have to pay attention to that right and that is part of the rigor which is like okay like I feel like I have built up some muscles around this from having used it, right?

26:56 And so I think part of the rigor is also like getting some practice with like looking at LLM code and reviewing it.

27:04 Yeah. Interesting. So the so so in this first use case

27:08 you I've got like I've got a lot of just TDM that needs to be done. And I the thing that I think is really interesting about about this case is you're doing something that we do a lot which is like okay I've got this problem. I kind of want to solve it in a way that's a little more generic where I where my my my colleagues can use it and so on. But we always have the tension. We always on the one hand we always encourage ourselves to hey this is a good opportunity to build a new abstraction to if you think this

27:34 but we're also all kind of realist like yeah but like we can't like not ship the next release or what have you because we're kind of focused on you know and and there's always that balance and to take this thing that like oh this to so to reduce the amount of work involved in this by a factor of four.

27:53 Yeah. Maybe the difference between doing it and not, you know, where it's like

27:56 just straight up, right? Yeah. Right.

27:58 I think I I actually suspect David has a

Autonomous agents transform workflows by pulling local repository context

28:01 few things to say because I know David and I have some have had some chats about this, but like for me like there are like new vistas that open up and I think that's the way I think David put it, right? So there there are things that were simply not feasible to do given you know company priorities and like personal life stuff going on and like all the different things right that are involved in you know a human's life that I feel like have opened up right and so for me like IDQG actually like the goal of this library was to increase the amount of rigor in our software so I think it is very cool that you know is able to kind of work on this right so this is a way you increase rigor is you build an abstraction that increases rigor even if it is tedious, right? That that is an increase in rigor, right, in the overall system.

28:48 Totally. Yeah. So, David, I mean, you you were, as Raen points out, like you were among the earliest adopters at Oxide. I think you've really shown the light for a lot of us and and you know, showing what these things can and can't do. Do you want to talk a little bit

Agentic tools pull in repository context

28:59 about your experience kind of getting into this?

29:03 Uh, yeah. Yeah. Um, yeah. I I mean for a long time I think until this year really when claud code took off I was using LMS as kind of like a fancy search even before they were really well even before they were actually search engines and you know everyone was like it's not a search engine because you're getting this very lossy picture of what's in the model weights even then on things that they were trained very well on which is like what I work on webdev they were great even you know just as just for retrieval so I was using them a lot for that or you know small snippets um this year I think is when it really um took off that the models could really do more complex autonomous um things based on a very small description. Um and more importantly, I think pull in like what you were talking about where when when the cloud code is looking at the Luminos code that you have on disk,

29:49 it's pulling in context that it doesn't have. And that's very different from

29:54 you know, it's not so much, you know, the typical use case. The typical use is, you know, you ask it a one-s sentence question and there's only so much detail that you can get back out of it because there's just not enough texture in the question um to tell it what to tell you back. And so like when you know I gave that the talk about LLMs at at Oxon in September, um a lot of what I stressed was like the the way to set up the problem for yourself is like you want to give it enough so that the answer is in some sense contained in what you give it. And what these agent tools do by just living in a in a repo and pulling in whatever context they want is like that they give themselves that that texture and context. So that's that's really what's changed um this year from the way I was using it a really long time ago. I was like I was trying to you know I wrote a CLI that lets you pass stuff on standard in and you can dump files into it. Um but you know giving the things the ability to just do that stuff on their own it makes things so much easier because you don't have to you know manually select a list of files to that's worth looking at. Um,

30:51 and so what kinds of things were were you kind of where were you first really beginning to use this to beyond just search or what have you really beginning to like okay I can actually use this to I can pair with it as Rain was saying

31:04 yeah the early things this earlier this year were things like stubbing out like I would stub out a test this was this was before they've got good enough at to really like you know you can tell it the kind of the shape of the set of tests that you want and it'll write 50 tests before that it was more like you would write the title of the test and maybe five comments saying the steps of the test and it would fill in, you know, it would still feel great because you'd be saving all this typing of the most tedious kind. Uh, you know, make this request, check this, you know, this on the response. Um, that was where it started to feel like it was really helpful. And I think I gave some some demos of that kind of thing where it's like, you know, what you want and you can tell it piece by piece and it'll fill it in and it would do a good job.

31:41 This is kind of what people are talking about, you know, in examples where they it can it can follow a pattern really well. like if you give in one example, you do this thing yourself once and you need to do it five more times, it can follow that pretty well. Um, but more recently, it seems like it's it's, you know, with Opus 4.5, it's been able to um figure that stuff out on its own, even without the stubbing out of all the details. Um, one example, the thing that really impressed me when Opus first came out was something quite different from

Debugging unfamiliar codebases using stack traces

3 Highlight · Finding real bugs in unfamiliar languages with LLMs

32:10 from you guys' examples, cuz it was an example where I'm not an expert. It was specifically something where like it was kind of a pure test of the thing's ability because I didn't know anything about what I was doing and I was still able to get to a surprisingly good result. And this was uh debugging crashes in the Ghosty terminal. So I ran into a couple of

32:28 crashes. I've never written a line of Zigg. I don't know anything about the code the the Ghosty codebase. I've never

32:34 looked at a crash dump to my shame as an O-ite employee. Um, so, um, but I, you know, a few crashes that I wanted to investigate that, you know, there were things that I couldn't find anybody talking about them on the ghosty GitHub. So, I figured they were pretty rare. So, I I looked into them and I just have Opus essentially figure out I, you know, the only thing I really had to do was find the rust port of Mini Dump Stackwalk to like look at the at the crash dump and um, point it at the problem and and and I knew where the crash dump was located on my disc. And then from there it basically was able to like look at the code, statically analyze it and find the source of these of I found three different bugs this way um real and then these I was able to write up the bug reports and they were confirmed to be real bugs and and and fixed and I so that was what really unsettled me was that this was an area where I really knew nothing and just using my sort of like sense of what sounds like it makes sense to validate

33:26 that I wasn't going to be posting AI slop on the ghosty GitHub um I was able to come to you know three real uh bug reports without really putting very much into the process.

33:38 Yeah, that is wild. And so they are were they primarily operating on the stack back trace or were they stack back trace plus was it actually walking data structures and was it actually like meaning?

33:48 There was no live debugging. I think it was looking at the stack trace and then looking at the code that and you know the the error that came up and then just sort of thinking about what the what could have happened in the code to cause the error.

33:58 Interesting. Yeah, that that is interesting. I want to be using them as de debugging tools a lot more. Um, and I'm I'm very curious about this use case. Um, so that is I and that is wild. So when you um you submitted the uh I mean this great thing about Eosti being open source um and Mitchimoto's project it's like you know I mean I would just like say good on Mitchell like as obviously does not need to work for the rest of his life has made generational money and he's writing a TTY emulator. I just think that that's you know that's pretty great. I think that is every software engineer dream. Um but and then making it open source. Um what was the reception to the act? But you said these were bug these are bugs confirmed to be bugs. So it sounds like uh what it found was legit.

34:48 Yeah. Um you know part of it was that you know the bug report itself was even to some extent out of my depth. Like a couple of them I was really confident and then one of them I was like it sounds really good but I just wasn't able to you know I didn't know enough about how ghosty worked or how Zig worked to to really evaluate. So, I was nervous, but I was, you know, upfront with a lot of humility of like, I'm really not sure about this, but I but it sounds so good that I cannot hold it back.

35:11 All right. So, let's actually talk about the first two where you're like, okay, I don't know any, but like I'm a software engineer. I' I've I know many other programming languages um where you were like, okay, I'm pretty sure that I just based on the its description and me looking at this code, I'm pretty sure I got a legit bug here. Could you describe kind of those first two a little bit in terms of like what did you what I mean you you had confidence you could like I I can actually not knowing very much zig or knowing only the zig I've learned I think I've got a legit bug here.

35:45 Yeah. One of them was very simple because it was like a copy paste error where they were just referring to the wrong variable and you could tell

35:51 uh you know it was supposed to be graphing bytes and it was hyperlink bytes you know and and you could tell that that was uh so it was like okay that sounds pretty pretty straightforward. Um, another one, this was like two months ago, so I can't

36:03 Yeah. Yeah. I'm so sorry. The two months ago being several eons ago, especially in in

36:08 the really complicated one was that, you know, something was it was a mutex uh lock that was like not being taken at the right time. And so there was like a conflict

36:15 in um and and so you know, reasoning about that was pretty tough for me not understanding how the code worked. Um so but it was pretty impressive that the the model was able to see it, you know, like this is where you should have taken a log and you didn't.

36:27 Yeah. Okay. So, another thing that I think is really interesting is the And then so Mitchell himself replied to uh on you you've linked all the issues in the chat. Um so obviously people can go get those. Um, I think one of the things that I really like about this, David, is that like you lead off by saying like, look, this is I've been using cloud code with Opus45 to investigate. You're very upfront with, hey, I this an LLM has done the work here as a way of like saying I'm not I like someone is else is going to need to look at this who's got greater domain expertise. Um, which I think is worth

37:01 Yeah,

37:02 it's worth looking at. In GOCI specifically, they have a very they have a quite clear LLM disclosure policy. So Mitchell has been pretty open that he uses LLM's uh tooling, but he also has they really want upfront disclosure.

37:13 Um

37:13 yeah,

37:14 so they made it easy by telling me exactly uh what to do.

37:18 I was more worried about sort of the the embarrassment if my issue was fake for me to be like one of those guys posting issues that are fake uh that the LM told them, you know, was a bug.

37:27 Yeah. Well, and the um but you know, you you obviously quadruple checked all this stuff and it looks like you had So all right. So, so this experience, as you said, was I mean, I like the the way you say it was like unsettling. Um, when you describe it as unsettling, why why was it unsettling?

37:44 Yeah. Well, I thought this was such a clear-cut case where it was obviously not my expertise that was operative here

37:50 because I didn't have any,

37:52 you know, I had like there was some highlevel like I could tell that it it felt legitimate. Um, and there I think there may have been one or two things where it came up with something that that I that I was like that doesn't sound real, but you know, the amount of guidance that I actually provided in the process was was a very small proportion of of what actually took place. That was what I think felt um unsettling about it. And you know, the the guidance that I did provide also didn't feel, you know, that ineffable human taste that that people love to attribute to themselves. It really wasn't that. It was like finding [snorts] the the rust port of the of the stack trace, you know.

38:27 symbolic or whatever.

38:30 Right. Right. But I do love the fact you're like I'm actually even a little bit embarrassed. You say in your this these issues that you're like I'm I uh but it's also like what am I what else I supposed to do? Like this thing crashed. Like I I am I just supposed to like not give someone the feedback that this thing has crashed and I've got like or am I supposed to just like sling an issue in there with I mean it's like it just feels like you're being actually helpful to the project. Um,

38:56 right. Well, if it had turned out to be fake, I wouldn't have been.

38:59 I guess if my diagnosis was wrong, then that was just creating work for them. Um, so it hinges uh pretty tightly on on the fact that they were legit.

39:08 Yeah. Yeah. And that it's it was okay. So I think it goes to and you know we've we've talked a bit about RD536 where we kind of talk about our own LLM thinking at oxide and it just goes to that like having empathy for the the person that's going to read this and the the making sure that and in this case really contextualizing it but also like it sounds like you're you're doing your own checking to make sure that the degree that you can. So um yeah,

39:36 you know it's interesting a lot of these uh sort of attributes or these qualities that we attribute to LL generated code are all things that as we're talking

Channeling rigor into code review scrutiny

39:46 about like I've associated with other colleagues I I just I'll I'll provide an example. I just mean when you're doing a code review, Brian, I think my guess is like the degree of scrutiny that you feel yourself applying may change depending on where that code came from. Like it certainly does for me and not so much at oxide but like when I was at Sun there were sometimes I' get a code review I'm like I really need to imagine what I would been been like to write this so that I know what I'm looking for. In other cases you're like well there's some code and there's some tests and I'll I'll look around. But um you know a lot of the thinking has probably already been done. Well, you know, on that like for my You're exactly right.

40:26 And like I mean, oh, look, I'm just ashamed to say it, but I'm going to say it. like the the way I would review code from like a nemesis, you know, a nemesis integrates code and you're like, I am going to I'm going to get my ne and I'm like one of the things I realized I needed to do was for my own self-re and for reviewing people that were not my nemesis. I needed to like channel that dark part of my brain that's like I'm gonna I I I'm gonna find the this thing in here. And that's like I mean it's embarrassing to say, but it's definitely true.

40:58 Yeah. Well, I I do that because uh for you know when I'm reviewing someone who I consider a friend and I want to do them the service of helping them with their code, but I guess we're just motivated differently and that's fine.

41:07 Yeah. Okay. Okay. Are you It feels like you're just trying to explain away a whole bunch of code review comments. Very good code review comments you give I mean feels like comments you'd give a nemesis, but that's right. Okay.

41:17 Uh one man's nemesis, another man's friend. Um

41:19 there you go.

41:20 Oh. Uh but and then you know David as you're as you're describing you know like you don't want to file a crap bug report like man have I seen some crap bug reports where you know people take you on this wild ride through a core file and you end up just nowhere. You're like I okay I'm following but like all of this is just blather. Like you don't need an LLM to hallucinate. Like we've been doing that.

41:46 Yeah.

41:47 Uh and we've seen these bug reports where you're like okay like you have there's certainly a lot of information here but you've actually not contributed. Uh so that same that empathy you're talking about is is so at the core of of of engineering full stop irrespective of the tools we're using.

42:04 Yeah. No that's a [snorts] very good point.

42:06 It is really infuriating to see like a a bad you know AI bug report. I'm probably more optimistic than most people about LLMs and I think part of that is just like working at Oxide and I don't really see anybody doing the pathological things that I that we hear about online. You know, everybody's so um careful and serious at Oxide. So, I wonder I worry that I'm biased um toward optimism because I'm not seeing the like the median user of these tools. Um but then, you know, I see one example. I get one bug report on a on a repo that I'm actually familiar with. I'm like, forget it. Throw these out. We're done.

42:39 Yeah. Yeah, but I but to Adam's point, like I don't like that when you get bogus bug reports without LLMs either. I mean, when you get or you get like bogus PRs.

42:47 Yeah, but it's harder to write off all of humanity. Like you can write you can write off

42:50 I guess that's true. I guess that's just

42:52 it's more it's more limiting if you

42:55 Yeah.

42:55 Yeah. Absolutely. Because but I do think that you know you get people like and we've definitely had this happen where we will make things that are that we are open sourcing not making a big deal out of it. We're not trying to create a community out of it. we're just open sourcing it kind of hygienically and then someone will come along with a kind of like spirious PR to change things like no sorry no this is like not LLM assistant this is in the prelim age like no no this is sorry this is actually not helpful so Adam just to your point that like the the lack of empathy is uh is is definitely drive by PRing is not new as as someone was pointing out in the chat um

The maintenance burden of automated contributions

43:32 I think the difference though is that you know like LLMs will like do amplify that problem, right? Like you know, you can kind of get I was talking about this with someone like you can get something that is not great by in like a few minutes as opposed to maybe a few hours.

43:52 You're you're so spot on and I I think we my experience has been like a [clears throat] unproductive uh like not empathetic uh colleague like that's fine. like if I I can run faster than you, I can keep up. Like I'm gonna like you're not gonna outrun me. I don't need to like worry about kind of diverting you in the wrong places. A highly productive, unempathetic, uh careless colleague, like that's takes way take 150% of my effort just to like keep them from doing harm. And you're right, Rain, that like it takes that formerly plotting uh you know, colleague who you had or or or collaborator who you had to like keep on the rails, it keeps makes it much harder to steer them.

44:36 Yep. Yep. Um it's like a gish gallop almost like I feel like that's that's how I think about it, right? Where it's like a gish gallop for issues. Um I've luckily not faced um too many like crap bug reports. I've seen some AI bug reports, but they've all been like very high quality. So, you know, kind of at the standard that like I think I would expect myself to write a bug report. So, again, like I am biased towards optimism here, but it is something I'm worried about. Like I do look at people just, you know, putting up garbage and it's like, okay, well, uh, it's it's now harder to filter out garbage or, you know, I have to say on the flip side, a thing I've done is I've used Opus 4.5 and fed it a bug report and told it to tell me whether this bug report is real or not.

45:22 Um, so I um, yeah, so that's that's you know, maybe that's the way to keep up. It's just like all them

45:28 stuff. It's like some open-source Jeffans paradox or whatever. Like there's no money involved here, but I just mean the the cost of creating PRs and projects and all of these things has dropped so much that the volume has just accelerated. Well, I think I also do think that with these open source projects, especially I mean, you know, God bless small communities where you've got I mean, it's like I I mean I I would be almost like intrigued by someone who's like I'm going to use an LLM to file a bunch of bugs against Alumos.

45:56 You're like that's weird. I mean that's like that's not a versus like

46:01 talk to someone about that. Yeah.

46:02 Talk to someone Yeah. I mean like I'm almost like I'm almost like not opposed. That's a that's a very Okay. Um versus like a project. I mean and certainly we saw this with node where you know I've been in very very large projects with many many many contributors and very small projects and there's a lot to be said about being in a small project uh and a lot to be said about a project that doesn't attract as much attention because it doesn't attract as much of that kind of negative attention either. Uh so there's uh there I I think this problem is I'm sure there are uh there there are some high-profile repos for whom this problem is really really acute and you know maybe that was that way with with Ghosty and Mitchell but for a lot of the stuff at least I work in it's it's not really an acute problem.

46:45 Yeah I've been surprised that the tooling for maintainers hasn't been able to keep up. I mean, you expect some lag, right? There's the you the volume of of garbage has to balloon for a bit before it becomes such a big problem that people are incentivized to put some work into solving it. But, um, I think one of the things we're going to see in the next few months is maintainers more openly using LLM tooling to like cut through that uh, morass of of of AI bug reports and uh, AI

47:14 and for and for code review too. I mean I think just like it is the code review is I mean honestly like my eye opening moment with respect to loss and software engineering was on oxide and friends when we had a listener who had access to GPT4 when I did not have it and I Adam for some reason I can't even remember and you know this you have to figure out exactly when this was I guess it's a little hard to ring the chime for an episode that I can't I can't recall more more just ring it exactly give the people on YouTube something to complain Um, but we um, and I'll go back and find the episode, but the thing that was really interesting is I had had like a PR that day that I was I linked to and someone dropped in a GBT4 code review of that PR and I'm like, wow, this is not all wrong. Like this is not also not great, but this is definitely not like garbage the the comments that it has.

48:08 And that was a long time ago with respect to LLMs. Um, and I I mean code review it just feels like the opportunity for code review is really rich and Dave to your point of like not really giving like why don't maintainers not have uh you would you would and maybe they do and I'm missing it about like it just doesn't feel like GitHub is providing it anyway. Why am I doing this? Of course too large to show.

48:34 Yeah, you definitely expect it to be built into GitHub pretty soon. I mean there are tools like Graphite, Code Rabbit. I mean that that's kind of what started me on this. saw someone, you know, praising this this tool, Graphite, which does look really nice, um, online, and it's like 20 bucks a month a seat. I was like, wait, 20 bucks a month a seat? I can write that in a script. So, it's like, so I wrote a script that pulls the diff, the comments, um, the PR body, um, and just feeds that into an LM just says, review this. And, you know, you you have downsides there where you're, you know, a lot of times there's additional context that's not in the diff. Like, if you're using something that is imported in the code already, the import is not in the diff. So, it's going to say, "Are you sure you're importing this?" It doesn't know that the test passed. Doesn't know the CI passes. Um, but you can get quite a lot that way. You know, it can find, you know, a mismatch between your SQL migration and your um main DBNIT SQL. I can find inconsistencies really well and inconsistency even between like your human readable stuff like your PR description and the actual code. Um, there's a lot you can do there. um even without what we now have which is like

49:36 the tools that can go and vacuum up anything that they need to and to to um validate their hypothesis about why the PR is broken with cloud code it can write a new test that validates that the code as written doesn't do something um you said and there's a lot of lowhanging fruit there that we're not we're not really touching at all

49:54 on the topic of low hanging fruit I think my nemesis on GitHub is stalebot right like I I hit some real bug Oh, Stalebot. I

50:05 I feel like LLMs could slay Stalebot.

4 Highlight · AI volume and the failure of Stalebot

50:07 You know, the like, well, this bug is 6 weeks old, so I guess nobody really has it or whatever. It's like, nope, there's a crash dump and a and a stack trace and a bunch of information.

50:17 Yes. and at least helping to weed through that so that you could sort of deter maybe neuter stalebot a little bit or or make sure you keep around the things that like refer to real problems with sufficient data to diagnose them or maybe diagnose them autonomously but um you know a lot of the work that you know I think there's a theme here but a lot of work that just doesn't get done that could get done uh and in some cases maybe doesn't need the highest level of sophistication to complete complete. You know, those are great tasks.

50:49 God, Adam, you are so right. And and God, I hate Stalebot with the white hot passion of 10,000 sons. I Stalebot is such an indictment. Uh I mean like we don't talk about Stalebot enough. We don't we like for all of you decrying the future, look into the past. Stalebot is I mean Stalebot is everything wrong because I mean it is just like oh well no one has seen this issue in six weeks so we're closing it.

51:19 You're like how does what how does that make sense? That doesn't make sense.

51:23 That comes from Facebook by the way. I know Facebook had an internal scale bot and then someone built an external version and and I mean it's the Facebook culture like it was exactly the kind of thing that you would expect Facebook to make and so you know

51:36 move fast and leave broken things around.

51:39 Yeah.

51:40 Move fast and close out bugs that haven't had any activity in the last 48 hours. Okay then. Yeah. But it is also so gross because like but no no I made look I made the dashboard green. It's like [groaning]

51:54 um

51:55 so Adam, you are right. I hadn't even thought about what this means for

51:58 Stalebot. I Stalebot, you are a marked bot. I I hope loss because that's a it's a great point of just like if nothing else. I'm sorry if like if we get rid of Stalebot, it was worth it all. I I got to say,

52:11 I mean, you're going to be up till like 4 in the morning vibe coding like the

Lowering friction for complex infrastructure fixes

52:15 Joker of of Stalebot,

52:18 you know? just trail snailbot around like reopening and fixing bugs that it's trying to close.

52:25 I actually have an example of of a bug that I feel like I would have just ignored um in the past but had a much better time with Opus 4.5. And so uh this is a bug on you know on cargo next test which is a personal project and and the bug is titled sig ttoou when test spawns interactive shell. Now, if if you've spent any time in this stuff, you're like eyes my eyes glazed over pretty much, right? And so, it's like, okay, you know, this person actually uh did a nice investigation with Claude um and kind of posted this and said that um they I've worked with Claude to get good attribution for this and reproduction.

53:03 It wrote words below, but I stand behind them, right? And so it was like a pretty well-written issue, but you know, it's the sort of thing that I really would want to dive in to like, you know, it kind of gave an example of like, you know, this is used by all these other projects and so you should do this thing as well. And it is like it is one of those um you know things where like okay, you got to spend like a whole day like investigating what the other projects do and how it fits and like you know really getting to the root of the problem, right? And I I'm like pretty lazy generally and I'm like I don't want to do that. And so you know I'm like I would either do a halfass thing and honestly in the past I would just do what you know what the suggested fix was right. Uh it turns out that the suggested fix is actually like woefully incomplete which is you know which is where like I feel like you know I I kind of gave this to Opus4.5 and so you know one of the things it said is that like less and vim and a few other projects follow this pattern right so I actually gave it the less source code and I gave

54:04 it the Vim source code and I gave it the source code to like a bunch of other things and I was like okay like dig into this like what do these projects do and so this kind of comes back to the you know asking questions of code bases you're unfamiliar with And so, you know, I did that, right? I had no idea about the less codebase. I had no idea about the Vim code base or anything. And so, it spent like 10 minutes or so and it actually like, you know, wrote up a nice summary of like here's what all the projects do and so on, right? And then I, you know, I kind of, you know, I was like, okay, you know, this makes sense.

54:34 And then, you know, I tried that. And, uh, and so, you know, it was like an interesting, like, it took me like maybe two or three hours to do. And the final fix for that was like pretty small. It was like 130 lines of code or so. But like it was great because like you know we we tried the first thing right. We tried the suggested fix. I you know the LLM did the work and the LLM kind of you know wrote the test which is its own annoying and janky in its own way. Um and then you know I kind of tried that.

55:02 Um I I like dog fooded it a bit. I found that okay this isn't complete in various ways and then you know we iterate a few times and so there are so many places along this path where prelims I would have just dropped out and being like ah this sucks I don't want to deal with this you know I'm done for the day or something right

55:21 well and and totally like when you do that like you're going to kind of take one of two things it's going to be like h we'll just take this kind of like mediocre fix

55:29 or it's going to be the like h maybe I'll just let Stalebot finish this one off maybe I'll

55:34 Exactly. Right.

55:35 Right. I don't have to kill it. I'll just But I mean, even this title reign like sig tou. Okay, fine. Ty out.

55:44 Well, no. You got to like the cue like antiques road show. You got to be like, okay, like I'm now going to go into like pix signal semantics from I mean,

55:52 well, and and then you're like, when test spawns interactive shell, it's like, well, here's a thought. Don't do that.

55:58 Don't do that. Don't do that. I mean, yeah. No, seriously.

56:04 Yeah. But but the Yeah, it makes it attainable and you're makes makes you get past the like, have you tried not doing that? I don't know. That sounds like a dumb test.

56:11 Yeah.

56:12 Okay. And so but and then this the thing and I do think like this is a really important point because then Okay. So you you pick this up now properly because it's easier. We've lowered the the friction. You actually get this completely fixed.

56:25 Getting this fixed makes next test more robust. It makes it more rigor. like you've actually like I mean on the one hand it's like okay really I mean as you say Adam like maybe don't spot in action but like hey actually no now you can though you know what I mean and I think that like I just see this in lots and lots of places where we are going to make our infrastructure actually more robust because we can now go pick up a bunch of work that we just weren't going to get to realistically we the people who work on this lower level infrastructure we're just not going to

Using AI completion to build previously avoided tools

56:57 get to

56:58 yeah so I have an example of some some work that I got to finally. Um I I I mean Rain described herself as lazy. Rain, I offer this counter like I like I think you're kind of bringing a knife to a gunfight here.

57:11 Um but uh out lazy Adam, will you?

57:16 Yeah. Cuz And you know this like I have been wanting to do an open API diff library since before you joined. I'm sure I've been like talking up this vaporware of like

57:27 and and I I made multiple earnest attempts at starting it

57:32 and it was just it's just one of these pieces of code that's like there's no good way to do it. All the ways to do it are gross and boring and stupid and actually it doesn't even it this is not like your kstat code running in privilege mode or whatever. This is some code that like if it segies like if it reboots the machine somehow like it's fine like I don't know like

5 Highlight · Using LLMs for rapid CLI tool prototyping

57:55 it's just not that high stakes very low stakes. Um

57:58 yeah,

57:59 and the thing that got me across the line was uh you know I started using uh some of the open API open API uh excuse me open AI models in uh VS Code and mostly using it just through the lens of uh of very smart completion and it allowed me to kind of repeat this pattern and that that I was that I wanted to use to make sure I wasn't forgetting to compare certain things. Uh and as Rain was saying, absent this, I would have like written some code to write code. I would have, you know, written some Earl script or something stupid to output a bunch of code or or use a procer or something like that. But my my real so that was great and I actually got the thing working uh and it was really fun to build. My real breakthrough was then it was coming to demo day and I wanted to show it off and this is a library so there's not like a front end to this thing. So I was like, okay, I'll write a little CI tool. And literally all I wrote was function main open a comment said parse the first two command line arguments and literally the

59:00 rest of the program I just had tab completed. It's like I think this is it figured out this is probably the program you want to write. I had to fix a couple little things here and there but uh it was very eye opening. Uh, and and then that became my demo in addition to the actual library I built, but was like the the live coding, the live I guess coding of just hitting tab and watching it do the thing that it inferred I wanted to do.

59:23 So I would argue, Adam, that you have embodied all three of Larry Wall's famous virtues of a [snorts] programmer

Shifting expectations on code repetition and type safety

59:31 that your you've shown your laziness, your impatience, and your hubris in a stroke. Um the but I actually but this point of laziness is really important because we all know and we we kind of speak of euphemistically as laziness. But we all know that like a hallmark of of good software engineering is coming up with powerful abstractions. And when you are kind of repeating code multiple times that part of your brain is like h this is not the right abstraction. And because Adam, both you and Rain mentioned like, ah, I would have made this a proc macro or I would have done something.

1:00:07 Yeah. where because like I I think we like overindex on that where we're like then this this whole like the dry thing the do not repeat yourself where you become so overindexed on it that you then do things that are actually either generating suboptimal artifacts or it's like there are times where it's just

6 Highlight · Overcoming DRY over-indexing with LLMs

1:00:26 like actually it's just not that big of a deal to have code that is like similar but slightly different in three places like it's okay we're all going to live but we really resist doing that and LM's make it easier to kind of do that.

1:00:38 One of the thing that annoys me the most and you know I'm very very grateful to the Rust open source community right and and this is just to qualify that this is nothing it's not meant to be negative at all right the open source community but one of the things that really annoys me is like you know in like the Rust talk you click on the source link right and the source link leads you to like a macro definition

1:01:01 yeah you see that for like um even for like the in the standard library there's a few examples with like the integer types So you cannot click like for example you want to see like the next power of two implementation which is like I mean you know it's a bit manipulation thing I want to look at that right you click on it and it doesn't show you that and it's like really sucks and I hate it. Um, so I kind of have made it a point in my libraries that I would much rather copy paste code just so that you can click through the source link and you can get it right and so you know and so macros are just like you know they don't work with that and but the DR the don't repeat yourself is to use a macro. So it's like okay well LMS actually do provide a of a better solution to that.

1:01:41 M

1:01:42 I've been thinking a lot about this because like if you think about where our intuitions come from about what is worth abstracting or like what is too much repetition they're so tied up with the expected reader of the code whether that's like ourselves or the people that we know like they're they're they're tied in very very tightly with our intuitions about what people can handle and what's reasonable to expect of other people and what is reasonable to expect of an LLM is radically different from what is reasonable to expect of other people. Um, and so like the amount of repetition that that is tolerable in a codebase or is manageable is is seems to me like what may way higher and it's not just in a codebase like we were thinking about um API design API response shapes earlier and we uh with I was working with Adam on this that we we aired on the side of making a response shape like more flat and less nested even though we lost a little bit of type information that way just because the flat one was a little easier to read and the type information we decided was not really worth it keeping in that particular

1:02:43 case. But I think when you assume that future developers will have LMS at their disposal, I think that it must tilt the calculus toward encoding more type information at the expense of readability because that type information is what is going to keep LMS on the rails um in the future.

1:03:01 It's like you know all the good practices like make invalid states unrepresentful and all of those things. Um it turns out that all of that actually helps LMS a lot too. So cool.

1:03:11 Yeah. Is there anything like does is do Rust do LLMs like Rust? I know that's a weird kind of question, but I I I've wondered that if the the the things that we appreciate about it in terms of like not being able to like represent invalid states and so forth, uh if that is a useful property when LLMs are constructing code.

1:03:34 Yeah, I I 100%. I mean, I feel we said this when we again ringing the chime for unknown episode, but I feel we we said this when we first started talking about LLMs and Rust that like actually Rust is going to be a really good fit for these things because you get the it mean Rust I something I've said from the beginning that Rust shifts the cognitive load to the developer in development and it forces the developer in development to consider a lot of issues that historically you wouldn't see until some code is deployed into production. And I loved that shift. I think that shift is really important. I think that like that tacks right into what LLM can do. And I think that it's that it they reinforce one another. So I think like I LLMs I think are and Rust are a very good fit for one another. It's what which I don't think is that hot to take. I don't think that that that's that spicy.

1:04:25 and just say like more, you know, if a more elaborate type system lets lets you put more of that work in upfront to sort of constrain the program further. You could say that LM's like allow you to tolerate an even more elaborate type type system like maybe now dependent types are going to be uh feasible to for people to learn and and work with. Maybe you don't see interest like you know take like if there's a diesel uh error like you'll be able to understand what it means.

1:04:51 Easy easy. Like listen, we're not living in that kind of a future yet pal. That's

1:04:58 Yeah.

1:04:59 Um

1:04:59 artificial super intelligence required for that.

1:05:02 No, I think the ASI is going to be like I I've actually I actually don't know what this message I can't make sense of this thing.

1:05:08 Um it's like it's like 2K log. Um yeah, that that is really interesting when I just think just in general having the um great type information. I mean the the code that I would be scar that to me would scare me the most would be just like pure JavaScript. I know it generates a lot of it but we use TypeScript. I mean Dave correct me if I'm wrong. We we use TypeScript for more or less anything everything. It would really terrify me to use because it's just so easy to have a an issue that doesn't show up until you get into runtime. So my blog is uh uses a static site generator written in JavaScript and I and I don't really know JavaScript. I mean like I I know I could hum a few bars but but that's kind of it. Uh and I used LLMs a lot to sort of get things the way I wanted them. And part of it is like I don't give a right? Like it's a static site generator. Sure.

1:06:01 It's gonna it's going to generate it statically and there isn't some like runtime edge condition I need to consider. So it's like eh go for it. Uh so I just be in in some cases like it's going to you know depending on the context you know I think that uh and I think that may be true in many JavaScript contexts and that's why in the cases where people are writing front-end code and they have additional rigor they want to apply they're using TypeScript or more robust languages.

1:06:30 Yeah.

1:06:30 Yeah.

1:06:31 Yeah. Just to defend the JavaScript world a little bit, I I think it's the spectrum of um you know rigor that you might need it applies in a lot of different situations. Like you might make a one-off rust CLI as a debugging tool in that same situation like you can tell if it works by running it. You know the sort of the depth of static analysis you you don't really need that because you run the thing and it does what you want and you can tell that it worked you know. So there there's a lot of situations I think that's kind of an underrated um point that you know people assume it's all or nothing that the code needs to be perfect or or it doesn't work at all which is like ridiculous.

1:07:03 You know our most rigorously engineered

Migrating complex APIs with precise natural language guides

1:07:05 code is still going to have some bugs in it. So obviously there's sort of a spectrum of of amount of um bugs that we can tolerate or amount you know of leeway that we have.

1:07:13 Yeah.

1:07:14 Yeah. That's fair. Um, Ren, do you want to talk about some of your more recent experiments with LLMs? you've really kind of gone nonlinear with some of the the things that you've been and I think in particular well because like I mean getting past the like okay these things are kind of experimental and getting into the like no no like actually we're going to at some level we're going to I mean I I I I don't want to say assume them because we're not really assuming them but we're um but we we kind of

7 Highlight · Executing large-scale type migrations via guides

1:07:44 acknowledge that like these things can actually um it can be used as part of software engineering. What do you want to describe some of the things that you've done recently? Yeah. So, um this is kind of a project that I, you know, kind of a bunch of us were discussing and uh I decided to take on sometime around early December. And and so the project here is that you know we um as as uh some of I'm sure some listeners may have heard of like we have done a lot of work in building automated update for our system, right? So we have you know the self-service update uh thing now. Um, and one of the things it has to cope with is the fact that, you know, you're not going to be able to update everything atomically, right? You're not taking the entire system down and back up again. And so you need to deal with um you you need to deal with like how do you or kind of you know like manage this and like this kind of skew while an update is happening. So you know uh like my colleague uh Dave Pacho kind of has

1:08:45 done this like genuinely brilliant uh design where like you know there's kind of the server side APIs which is the idea is that this is an API that can talk to multiple versions of clients and so you know you update the server first and you have like this DAG of dependencies that you update. It's just like this you know really well constructed system. It's pretty great right. Um so one of the issues we ran into is that as we gained experience with the system we were having trouble figuring out how like you know you have all these different versions and so you have like a type right and that type and like has the same name but it has different like fields for example or like you know maybe maybe one of the sub fields is different and so um how do you actually like store those in the repo right like and and it sounds like a simple problem, but this actually blows up and becomes like this incredibly complicated problem with many many many uh different uh factors involved. So um you know kind of I I did

1:09:46 again like this is one of those things where it's like this combination of human and LLM work where like I kind of you know I I I spent a bunch of time um you know prototyping a bunch of things and like you know kind of coming up with a with with like you know an approach that works and that uh satisfies all of the hard constraints and like also as many of the soft constraints as we can and and so you know this was like a lot of work and then um one of The one of the interesting things that I found really useful for LMS is that what I did was I ended up essentially like compiling the set of things that you know the final state we want to get to and like writing it out as a set of instructions that both a human and an LLM can follow. And so in this guide uh that I uh dropped a link to uh this is RFD619. And in this guide it is in section um 5.1. And so this 5.1 is kind of this initial migration, right? And so

1:10:48 again like you know spent like a couple weeks working on this on on like this whole RFT. And then what I did was like okay you know I'm like I just kind of fed this guide into the LLM and you know I told it to like migrate a small repo right one of the one of our smaller APIs and it just did that in one shot. So this was like you know not a very big API. It just did that right. I found that okay there were a few things I was unsatisfied but so I went back and like changed the guide. I updated the guide.

1:11:19 I kind of started from scratch. So, you know, I I like iterated on it. I want to say like overall like this guide kind of went through maybe a couple dozen iterations of like me looking at the LLM output and being like, okay, you know, this this is great or this is not good and so on and and you know, kind of basically ended up converging on something that is like this clearer very reproducible set of instructions that are simply way too complicated to capture in any deterministic algorithm.

1:11:49 Right? So this is like you know the the the there's like enough judgment here and it's just like this really complicated set of things that I mean you know there is no way I can write like a migration tool to do this. I mean may maybe someone smarter than me could do that. I I don't think I can. But what what the let me do is it kind of again like let me like design this guide once and then you know apply it like everywhere. So, it was funny because like there was one morning where I just like rapidly put up three PRs where like, you know, the first one was like a thousand lines of code, the second one was 2,000 lines of code, and the third one was 3,000 lines of code. And I got like all three of those done in like an hour. And that was just wild. like you know and this is like one of those things where like it turns out that LLMs are really really good at um following instructions that are like you know clearly written and are written in a way

1:12:43 that you know the LLM kind of works well with so I don't like this is again one of those things where like it sounds like so like you know mid- priority right it's like how are we going to you know migrate like 40,000 lines of code and like rearrange the types right this is the kind of thing that just falls, you know, people just don't do or like we might do it in the future and there's this long migration period or like you know this is the kind of thing that you do in like tech week but this would be more tech month right

1:13:14 um but like you know and LM just like as I said just nailed three different APIs in like one hour and that was just like it just blew my mind is like oh you can you can spend two weeks carefully designing a thing and then just have the LLM just like repeat that pattern over and over again. Uh it was also really helpful for like the process of iterating on the guide itself because like I would just like you know it's like if there's something I'm not satisfied with or like you know maybe maybe one of our co-workers had some feedback on something you know I could like very quickly like update the guide right and then I would be like okay run like JJ diff on you know the changes that I made and like replicate those changes into you know this like prototype that we're working on and it just like did that and it was amazing like it was one of those like, wow, you just you could just you could just do that.

Leveraging deterministic signals and secondary reviews

1:14:02 Something people have mentioned in the chat and we haven't talked about too much is that, you know, something that really helps the LMS in these kinds of loops is they have a signal like a verification signal that can tell them when when they're done and how far away they are from it and like types passing. Obviously, test passing is one of those things. But I'm I'm curious like how how you think of what the verification signal is to the LLM as it's doing this. Like is it just does it satisfy these plain these um you know natural language requirements.

1:14:30 Yeah. So, uh, so this is, uh, yeah, this this is an interesting question, right? So, in this case, you know, uh, we kind of had like a couple of hard verification signals. So, so the first one was just that, you know, what what you described, which is like, you know, like the code compiles, right? That that is kind of the most fundamental requirement. And then um the test pass and we have a lot of deterministic validation. In fact, a bunch of this actually uses uh the work Adam was describing on like comparing being able to compare open API documents to make sure that um if there are changes, those changes are only trivial ones. And so, you know, we put a lot of work into that. Um and so having all the deterministic validation was really helpful. Um what I ended up doing for for you know some of the more like the fuzzier signals here was that I basically kind of you know after it kind of did this work I would like start a new context window I would feed the guide again and I would ask it to carefully review the um you know the the the current performance with the guide right

1:15:31 and CL's like who the hell wrote this this is like I

1:15:36 yeah interesting

1:15:37 and and so that ended up finding you know a bunch of degrees of freedom which some of which I wanted and some of which I didn't but that was like that was a good experience. I would just do that like two or three times and and then you know obviously I would go through and manually review and make sure that you know everything kind of aligned. Um but again like that felt like a very quick process because you know I was just I was just able to like you know maybe like spend five minutes doing the migration and then another five minutes reviewing it and that was it. Right.

1:16:04 It's wild.

1:16:06 And I mean, this is kind of a just a much more elaborate example of really your the the IDQD example that you had where like, look, I've done this once. I need you to kind of do it in these subtly different but important ways that are kind of tedious. I mean this is just a a in many ways a much more elaborate version of that where it's like okay this is the the we designed this RFD very deliberately with and a lot of engineering has gone into like the way we think about doing this and we have that has come out from actually like doing it by hand and so on for a and now we actually need to like kind of knock this out for a bunch of these different services. Y um someone in chat described it as like using English as a programming language and yeah I mean that this is basically like you know using English as a um programming language for programs that are just too hard to write in a deterministic like computer language and and that's what it felt like doing. Um,

1:17:08 and and I think it's actually, you know, it's it's kind of remarkable like these are the kinds of things that you would absolutely have humans do before, you know, before before the advent of this stuff. And like it

1:17:20 or I mean like like I mean just to your point, it's like the work that is like it's just the the work is just like not done. And you have been like someone's like, "Hey, I was in this service and it has a different like what's going on over here." It's like, "Oh, we just haven't gotten to that one yet." and go into our this dashboard from two years ago and we're waiting for the next you know tech you I mean tech debt week it's like oh my god I can I I can feel like

Prioritizing software quality over raw speed

8 Highlight · Using LLMs to drive up code quality, not just speed

1:17:45 my the tech debt flu coming on for tech debt week so you know

1:17:50 I mean and like and for me like you know I think there's a there's a way David put it that was really memorable like a code like that uses like LMS extensively had better be the best freaking code on the planet right like if you're if you're doing this like you like all your like code should be extremely tight. You know, you should like your your you should like put all the work into refactoring like good documentation like all of these things that I think you know many of us feel like are are um you know maybe maybe kind of slip down our priority list there. It is very helpful to think of these tools as not ways to improve the velocity of what you do but ways to improve the quality of what you do. Um, and so I'm like, you know, if there's one thing that I think I want people to take away, it is like slow down, right? Like don't just like, you know, spit out as much code as possible.

1:18:40 Instead, like use the LLM, right? Which is a tool there to be like, okay, you know, maybe let's refactor this. Maybe let's, you know, split this up. Like there's so many things you can do to improve code quality along the way that will lead you to higher code quality than you would be able to do in the same amount of time. Rain, I I just cannot emphasize enough how important it is that if you're if you're listening to this as a podcast where you like, please go back and relisten to what Rain just said because I think this is so important and I think it is so important to realize that you've got this power now to go deliver a higher quality artifact. like yes the world emphasizes like the the the velocity which a term that I again don't like because it makes us all sound like projectiles but the um this is what what it allows us to do is do things that we simply never would have gotten to before that allow for

1:19:37 more rigorous artifacts and I think that you can make an argument that that the world that the software rewrite is going to kind of bifurcate Adam some of it is going to be your JavaScript in your static site generator which which to quote your own language back to you you quote do not give a about. Am I am I stating that correctly?

1:19:56 I stand by that.

1:19:58 Uh and but then in order for like underneath that is now these rigorous artifacts that we actually in a world where we're doing much more software, we actually need these rigorous artifacts to actually work much better. And

1:20:16 you know, I I think that like because Adam, I mean, I and I this is like with the and I I for um this is like the Fossil Time, Gen X Fossil Time where I maybe we can knock down some things on people's bingo cards, but when you talk about like software in the '90s sucked and operating systems had bugs that you would hit frequently, compilers had bugs that you would hit frequently. And the I mean what ultimately like the day I put C++ down is cuz I was dealing with two different compiler bugs simultaneously and just like the and getting basically random results. I was just and that was common in the '90s and man like go like comp go have a compiler bug to really like take the wind out of your sales let alone two of them. We and we needed to get to a world where we had open source artifacts that we could make much higher quality and the quality of software went way way way up. As a result, we could do more of it and I man I boy do I see that happening vividly here.

1:21:14 Yeah. No, I think I think you're right that uh you know the reason why I don't care about the quality of my blog is like yeah that that that's not a foundation on which I'm going to build uh you know decades worth of technical innovation. That's like yes, one and done.

1:21:30 And I think there's lots of software that kind of fits that model. And I think that's where you kind of get the slop, you know, uh slopware porative term, but for some of this code and like it's sort of fine like if you're if you're building something that is a one-off, it is associated with like some time and place and whatever, fine. Like whatever, I don't know. And yeah, there's going to be a lot more of it and that's frustrating. Uh but on the other hand for the stuff that is foundational that has always been rigorous and the rigor is increasing this becomes a lever by which the rigor continues to increase.

1:22:08 Yeah. I [clears throat] mean for me it's just like I there are so many things that I feel like I've been able to do with this to increase rigor. like my like I've you know coded a couple of things here and there but like my interest as a professional is really focusing on rigor and and my background is in dev tools where like correctness is like absolutely essential and non-negotiable and for me it's like okay you know there are so many more tests that I'm writing now like I you know the other day I was like I want to learn how to use scani right which is you know this model checker for rust and and I wanted to use that right and I'm like I there's there's always been this activation energy. We have to go read the documentation and stuff. And so what I instead ended up doing with this is that you know I took an existing project that I had which I felt like was a good fit for Connie and and I just asked cloud open 4.5 to hey like you know come up with a few properties that we can you can verify that way and it just did that and I'm like now I understand how this stuff works and what the limitations are

1:23:08 and stuff and just like like there are so many ways you can kind of use this stuff to to go like increase the level of rigor in your software. And honestly, it really bothers me that the dominant narrative is the whole like slop wipe code stuff, right? Because like or infrastructure engineers, there is so much more you can get out of it.

1:23:28 Yeah. But isn't that is that always been the case for the kinds of code that we care about Rain that like uh you know one of the things that's beautiful about Oxide is we go to a demo day where you know we show off you know Rain you show off this 30,000 line change or whatever or I show off like this library that compares one thing to another thing and it's like people are hooting and hollering as opposed to um

1:23:52 you know systems demos are traditionally seen as boring and the thing that's wizzy is when you know you can demo something cool and graphical and whatever and uh you know rigorous is not to everyone have the same kind of sex appeal.

1:24:09 Yeah, totally. I though I I think and re but you're also right about the dominant narrative and I was trying to think about I mean and clearly it is a truly a dominant narrative and that it's dominating kind of everything and Adam I was trying to think back in terms of our careers when have you had these kind of like big narratives where it feels like it's reductive you know one thing I was thinking about was the rise of Java was that way where the rise of Java was really suffocating because there was this idea and it's like very different so I I don't want to be too reductive here but with the rise of Java there There's this idea of it's like it's the end of every other programming language.

1:24:43 Like this is this is what we're going to do. And like this is kind of crazy to think about that the because this is I mean right it is it's like it's it is it's it's humorous now but it was at the time there was this idea that everything's going to be in Java. We're going to do the operating systems in Java. The microprocessors are going to execute Java bte code. we are l and and I mean at sun at the time it was re it was like I know this is not right and I like Java is like really powerful and important and it's going to allow many more people to write software and I remember thinking at the time like well at least it's the death of C++ um but what it it took a while for people and some like some failed experiments right it took it took nano Java and Pico Java inside of Sun and a bunch of and two different oss inside of inside of inside of Sunr and Java. Uh so there were a bunch of like where we got

1:25:44 and then people realized like okay no this thing is like it's important and it has a role but it's not everything and

1:25:49 it wasn't just all languages it was operating systems and operating environments right it was like the right once run anywhere meant

1:25:57 right

1:25:57 you don't have to worry about the the details of Mac and Windows and Unix and and all the different flavors of Unix. No, you just write it once and you write it run it anywhere. Uh, and and it meant all of that other stuff was just going to become meaningless. And the only thing that that was going to matter with Java, you're totally right that it it it took all the air out of the room for like a big chunk of like the the late 90s, maybe early 2000s.

1:26:26 Yeah.

1:26:26 Yeah. Totally. And if you were implementing in C, it's like, well, I I hope the past is working out for you. I mean this is this whole idea of like you are you are actually a living fossil and Java is actually going to come to replace you

1:26:36 and you know and in some ways it was like I actually I really do think it was kind of worse because if you were doing as what we were doing like you know we're in the operating system developing this thing in C it's like Java didn't really have anything for us you know it was not like oh I mean we did it around the margins but not like our tooling it I mean even the the kind of the value that Java legitimately delivered We didn't really realize any of that. Um, and you know, ultimately we uh we ultimately had a good relationship with Java, but it wasn't like whereas I think with like LLM like no, no, you can actually everybody can kind of up their game with this thing in a way that's really exciting and uplifting.

1:27:19 Yeah. Spot on.

1:27:22 Um, well, David Rain, anything that Elsa, we I know there there's obviously

Writing things down as a leverage multiplier

1:27:28 a lot to talk about here. I think we covered everything there is to say about LLM.

1:27:33 Finally,

1:27:35 I mean the thing I will say personally is like having a culture where writing things down is valued is like you know it is like a real multiplier here and so I mean on our side like I'm I'm very happy that you know all of this work that we do like we now have a new way to gain leverage from from all this writing work that you know we are culturally do. Um if you're if you're at a place that you know maybe doesn't have as strong rigorous like requirements or like you know isn't as committal as as Oxide where we ship hardware or whatever um I would still consider like you know doing work to write things down and produce good documentation good design documents because like at least the current generation of LLMs like really like that and so you know like kind of you know get a little more disciplined right with some of these things right um so yeah That's that's what I would say like write things down. That's a great advice and actually let me ask you to expand on that just a half a beat because I do feel as as part of of deep blue um you

1:28:38 do have especially and it's unclear to me by the way if this is truly young people of like undergraduates versus a a kind of a more mid-career malaise and maybe this like maybe deep blue cuts across all of it but people who are wondering like what is you know how can I what is my role in this kind of this new LLM age. Yeah.

1:29:00 Um what what would be some advice that you would give to an engineer that's early in their career um and looking at this stuff?

1:29:09 Honestly, like this is kind of the advice I would tip like I would say like practice, you know, like writing like for me like writing is not a natural skill for me. This is something that it's taken me many years of work to kind of get where I am now. Um, I would say like if you're starting out like like practice writing um don't have the LLM write things but like have feed it into the LLM and see how it behaves when you kind of do that and like p do you know that is the one bit of advice that I think I think this is the kind of advice that you know is like timeless in the sense that we have always written things down and we will always keep writing things down. There's always a lot of value to that. But I think the in the LLM age like this is one of those ways where you can really multiply the amount of rigor you have.

1:29:54 Yeah, that's great advice. I think the advice I would add is like hey, you can now you've got the ability to pick up a new language, pick up a new system much more quickly than before and you should use that as a way of getting into something maybe you would have been intimidated by. I mean I do think that like I mean look look kernel development feels intimidating to people. Lots of people don't pick up kernel development because they're intimidated by it. And if you view an LLM as like giving you the opportunity to jumpstart you in kernel development, go for it. That's great. Like that that's that is has got a very uh robust basis. So hop in there and you know hop [snorts] into alumos or or something that you wouldn't do otherwise, maybe a database, what have you.

1:30:36 Um and use that LLM to get you jump started and to get you mastery over this thing. elements don't judge like ask all the questions that you'd be embarrassed to ask the human like

1:30:49 if anything they could judge just a little bit more be like that is kind of a bad question but

1:30:54 this okay this is why pair programming never really worked out for me is because you always have someone being like you used what you don't use d'vorak like I thought like no I don't use or like you know like do you actually know that there's actually a faster key binding like no can we aren't we like trying to work on this problem together like why are you coming like you don't use syntax highlighting what am I even here for? You know, it's like, okay, we're just now having fights over things. And you know, it's just like we don't, you know, you don't have those those unfortunat I don't think the LM maybe I'm a little maybe I should confide to Claude code that like by the way, I don't use syntacting. What do you what do you think about that? See what it's uh um but yeah, it's free of judgment, which is really terrific.

1:31:33 Well, thank you all. I I you know I know this is a hot topic and um I I I think that I'm hoping that we can show that big moderate middle and um really show that there is a third path that by the way is the most likely path which is that we actually use these things as tools. They're not coming to replace you but they are actually going to allow you to do a lot more. And um the the one that should be most worried about LLMs is Stalebot. Stalebot for you.

1:32:02 Oh I hope so. [gasps] death to stalebot, I say. So, uh, Adam, thank you for for, uh, stoking that rage. Um, but, um, thank you all. I think this is really great stuff. Thank you for for coming in on a hot topic. Um, and thank you all in the chat, too. I think this is this is really important. This is not going to be, um, our last LLM episode this year. I don't think Adam

1:32:28 that's a great prediction.

1:32:29 Exactly. It feels like a luck. All right. Thanks, R. Thanks David. Thanks.