Why performant code matters (but gets widely ignored), with Casey Muratori

23,670 words · about 95 min to read

0:00 There's a misconception about the way you approach optimization. You run a profile, you identify the big parts, you make some changes, you measure statistics. I've worked with many extremely good optimization people and that is not how it is done.

0:14 Assembly language is not all that complicated right?

0:17 All of the JavaScript libraries, the DOM, CSS, React, assembly language, maybe there's 20, 30 instructions you might have to learn total. If you can vertically center a div in HTML, then you can probably learn assembly language, I would say.

0:30 How are you using AI tools, if you're using them at all?

0:33 We are not using them at all. The reason that I want to program things in a game is because I want to program them. If I just wanted an AI to program them, I'd just go get the Unreal Engine.

0:43 What are things that you think are non-negotiable for someone to be a great software engineer?

0:47 So, I find there's a lot of received programming wisdom that's just nonsense. Clearly, no one's ever tested it. And so in order for it to be received wisdom, you should Why do most devs not care about writing performance software and should we? Today's guest Casey Moratory spent the last decade arguing that we should. He also says that most software out there runs tens to 100 times slower than it needs to. Today we discuss why the focus on performance took a backseat across the industry and why Casey thinks the tide is finally turning. why you'll want to learn reading assembly if you're serious about high performance code and why it's less scary than it sounds. The saying premature optimization is the root of all evil, why Casey says that the majority of people use it to avoid thinking about performance when they really should and many more. If you want to get better at writing faster software and become a better engineer while doing so, then this episode is for you. And if you're one of the people who hit a like on this post by Ryan asking for this episode to be more about programming than about AI, this episode is also for you. This episode is presented by antithesis. If you work

1:48 with agents, your job is no longer just writing code. It's also specifying and testing it. Antithesis is the most effective method of verifying agentic code today. This episode is brought to you by Sentry. You probably already know what Sentry is because you're a developer. If not, just ask a dev and they'll tell you. I use Cry to monitor the back end of the pragmatic engine for any and all events and errors. Of course, Sentry doesn't only do errors. They also have logs, replays, spans, profiles, metrics, and more because they're all connected to the same trace. One new capability Sentry has I'm really liking is its ability to fix errors. Let me show you. Here's a list of errors on my admin back end.

2:21 There's a recent error on O that I want to check out. Let's have Seir run an autofix for us. Seir is Century's AI debugging tool. First, it generates a root cause analysis. It's finding some problem with HTTP versus HTTPS URLs. Cool. Now that we know what's going wrong, Seir can create a plan on how to go about fixing it. I could go and edit this plan, but I'm happy with it. So, let's create the actual code fix. Here's a code fix that Seir generated. Assuming it looks good, and in my case, it does.

2:49 Let's draft the pull request. And boom, the PR is created, ready to merge. What I love about Autofix is how Senu went from showing a list of errors inside my application to offering me a fast way to fix it and close the loop while I stay in charge of this bug fix the whole time. Debugging got a whole lot faster and a whole lot easier. Check out Sentry

Early exposure to computers through Digital Equipment Corporation

3:07 at centry.io/pragmatic io/pragmatic and start detecting errors, diagnosing your root causes, and fixing issues and regressions today. All right, Casey, welcome to the podcast. It's so nice to have you here.

3:18 Thank you so much. It's great to be here. Thank you for the invitation.

3:21 Now, I want to go back when we start to the beginning. How did you get into tech programming computers?

3:27 Well, I guess computers, it's like very very early on. Um, my dad was a programmer at Digital Equipment Corporation, which is a company that people will know if they studied computer history, but would not know if you just looked at the landscape today. They're they're completely gone, right? Uh, they got absorbed partly by Intel, partly by Compact, I think. Uh, there was, you know, they kind of got uh, broken up. At that time, it was kind of a really big computer manufacturer. You know, computers like the PDP11, that's a digital equipment corporation computer.

3:59 uh the vax like things that you may have heard of in computers.

4:01 Oh, these were these ma massive main frames.

4:04 Yeah. Uh mini computers as well. So like smaller also sometimes than main frames like the kind of next step down, right? And so in general that that era, my dad was a programmer there. He would later end up at Intel because, you know, like I said, parts got acquired. He never actually left his job. He just ended up at Intel through kind of uh digital's eventual demise. But as a result, we always had computers at home, even though at that time, you know, that might have been a little bit odd. Um, you know, I I learned to program when I was seven, which would have been in like, you know, 1982 or something like that. And so at that time, you know, maybe you might have an Apple or Commodore kind of computer at home, like some kind of early computer. I don't remember the exact line dates of those computers, but most people didn't. And it was only until a little bit later that you would and you probably wouldn't have had a programmer in your household to teach you more importantly, right?

5:01 So, uh, so I learned really early on and that's when I got into computers. How I got into games was, uh, I ended up randomly interning at Microsoft and I met people there and like I went and, you know, sort of went off into games through them. That was that was how that happened if that makes sense.

Sneaking WinG into Windows to enable PC gaming

5:16 But wait, how is the Microsoft/game relationship? That's not kind of a given, right? Microsoft is not They have one game, right? It's Flight Simulator.

5:26 Uh yes, at that time it would it would have been very weird. Uh the reason that happened was a guy called Chris Hecker who uh a lot of people don't really know his history because there was sort of two waves of bringing games to Microsoft Windows. Um cuz I mean now it's it's funny to think about now because people think of like what platform are you going to play games on if you have a PC? You know, Windows is like the default. Linux is now an insurgent to that. But Windows is the default. And so you think about how did that happen? Be you know uh because that wasn't the case if you were back in the early days. Uh Microsoft Windows not a gaming platform.

6:03 It's almost nothing on it. It's like solitire and mind sweeper and a few other uh sort of games.

6:08 How did it happen?

6:09 So uh what happened the reason for this this kind of gets into a technological reason. The reason for this is that it was very hard to actually produce images that could be displayed on the screen quickly. And to understand why this is is like its own kind of topic, but in general, you can just imagine you have these this operating system running, which is Microsoft Windows. It's controlling the graphics card. It's often running at a fairly high resolution uh compared to what a game might want to run at. You have to negotiate with it in order to display your bit map in some way that won't destroy all the things that it's trying to display. So on so forth. early versions of Windows uh up through like Windows uh three uh Windows for workg groupoups let's say if anyone remembers that name.

6:54 Was it after 3.1?

6:56 It was 3.51 I think is what it's called or maybe just 31. Yeah, there's NT351. No, so it's like I think you're right. 31 3.1 I don't know something like this. Yeah, Windows for workg groupoups was uh you know around that time. that version of Windows, which is in the 3 series,

1 Highlight · How WinG bypassed Windows limits to launch PC gaming

7:12 didn't really have a way to quickly use the CPU to fill pixels, which is what games need to do, right? There's there's no GPU acceleration really at this point. We there's a little bit we could talk about, but it's not mainstream. It's not in consumer. So, they they need to be able to to do this sort of thing. They need to be do it in a double buffered way so they can draw to a back buffer and then show it to the screen. And that has to happen very quickly. And there just wasn't a way to do this in Windows. In Windows, you had to kind of go through uh this API where you would produce sort of a bit map that wasn't necessarily in the right format for the display you were using and then it had to do a translation from that bit map to the other one when it displayed it. All this sort of stuff. So games on Windows like things like Doom, they're not coming to Windows, right? Uh that kind of future was not in the works uh for Windows. And that was kind of what some people in Microsoft wanted to change.

7:59 And one of the people who brought this change about was a guy named Chris Hecker. and he was like, "Okay, we could actually just make a library that did the fast blitz to the screen so that we could have a way that people could do these draws and get them on the screen quick enough to make gaming viable. Would it be 100% as fast as DOSs?" Probably not. But could you actually run uh you know, some of these new games?

8:26 You know, Wolfenstein 3D I think would have been out at this time. Doom was kind of on the horizon and that sort of thing right?

8:31 Yeah. And so he started this project that he was not supposed to do. He did not have the authority to do this called wing, which I believe at least stand for like win graphics or something like that. Total skunk works project. He had cover from his manager. I'm I'm assuming I can say all this stuff now because it's ancient history. He had cover from his manager. His name guy's name was Michael Edwards. And Michael Edwards basically just kind of ran cover for this, which is a thing that probably wasn't going to fly because they were in a division at that time which would have been Microsoft Research today kind of. It was called at Advanced Technology was the early version of Microsoft Research. You were not supposed to be shipping core libraries for Windows like it had nothing to do with that. Long story long, what ended up happening is that product did ship. Win, I guess you wouldn't call it a product. It's an add-on for Windows. did ship and it was the first step towards DirectX.

9:23 Uh people forget that Wing was the first way you did this and then eventually we had DirectX and DIB sections and Win 95 and all that sort of stuff. When I went to Microsoft, amusingly the person I was supposed to be reporting to as an intern was Michael Edwards. When I showed up

9:41 first day along with the other interns, uh guy named Rudy, a guy named Rajie, uh we were all supposed to report to him, right? because you get a couple interns in under one, you know, sort of manager. We show up and we're just taken aside, you know, after some kind of, you know, stupid HR orientation thing that was, you know, lame as it always is. And and we get taken aside by somebody. I don't remember the lyrics. They're like, "Look, bit of bit of a problem. Uh the person you interviewed with and were supposed to like report to, uh they they aren't here. Like I don't remember exactly how they put it. So, uh, you're going to go talk to this other guy and they'll find something for you to do.

10:18 Turns out the wind thing had blown up the previous week and Mike Edwards had stormed out of the building and had not been seen since. That is what actually happened. So, that's I arrive at Microsoft, the person I'm supposed to intern with, he just he just flamed out and left. No one knows when he'll be back. He did end up coming back like a week after I got there. Uh, but he was kind of moved over to a different division. like they you know there was a bunch of like triage work done there to figure out what's going on. So that that was my experience but suffice to say it meant that I was right in sort of like ground zero of games on Windows. So I met Chris Hecker. I got to talk to a bunch of people there. He actually took me out to see one of my childhood heroes Ron Gilbert who is the guy who did the scum engine. Yeah.

11:02 He knew all these guys because working on WG he had gone out to see a bunch of game developers and like work with them and this sort of thing. So, I got to go to Humongous Entertainment, which was Ron's new company. He gave me a a secret of Monkey Island, uh, mouse mouse pad, I remember. Uh, so anyway, it was it was really cool. And that's how I ended up getting into the game industry was was was through Chris Hecker. And he's kind of an unsung hero of getting games on Windows because, you know, he he just he wasn't out there making his story known, but but, you know, I am now, I guess.

11:32 But it's it's so interesting to hear these stories. Obviously, now you can share it, I'm sure. you know, like for a while this would have been like only with within the inner circle, but the fact that you know, of course, DirectX was a huge

11:43 success and and it it did like as far as you know, from my vantage point, huge reason why games are big on Windows, but now here's someone who just ignored wasn't asked anything, just was doing something, got into conflict, fights, and just like pushed an idea. It's more than you think because there were three people who really were the core people who pushed DirectX, meaning institutionally pushed it. There are tons of programmers like Todd Laney who did, you know, really important core work. It never would have shipped without people like him. So, not on the program side, I'm talking about institutional side. It's Angstrom and Alex St. John. I don't remember which one is either or Angstrom was the tester on Win. So he came from that team. So the start of DirectX, one of the core members of DirectX was on the Windy team. So it's a direct lineage. It's not even like an unrelated push. So WGI really was the start of it.

12:41 And then DirectX was kind of the actual full blossoming into an org with Microsoft's blessing at that point, you know, that that actually became powerful

Building tools and independent game development

12:50 internally. And then after after this Microsoft internship and getting exposed to all these folks, you actually went and and built games tooling, you then started your own studio as well, right? So like how how did that sequence?

13:03 I guess there's a couple of steps in there. I worked at a startup with Chris Hecker that didn't end up doing anything interesting. Then I went to a company called Gaspowered Games, which was actually a Microsoft guess they're not a Microsoft studio, but their publishing deal with was Microsoft. and uh they did a game called Dungeon Siege which is kind of a you know I don't know it's not a particularly well-known title. From there I went to Rad Game Tools and that's where I stayed for quite some time. Uh I did their character animation system that was a very popular product that ended up getting used in in lots lots of games. It's still used to this day, much to my surprise. Uh because that's a very very I haven't worked on it since 2004, but I guess other people had maintained it and some studios just kind of integrated into their you could get source licenses. So I guess some studios just integrated it into their pipelines and have never removed it. Uh and they just maybe keep updating it um to to keep it working the way that they want.

13:57 So anyway, uh that's what I did there. And then afterwards I' I've been independent since then. I just have a company called Malle Rocket uh where we do various stuff. I've done you know contract work for for people through that. We now do like the Substack through that where we do educational materials. So I've kind of just done random stuff uh since then. Although I have done some work on games. I noticed your checklist you were talking about the witness.

14:19 Uh obviously that one was an actual specific title that I worked on but that was that was mostly just because it was a very big project and I was you know I'm friends with John so I was just trying to do some helpful programming on the side. I did some stuff on how the movement system worked. I thought there were some interesting problems that we could solve there for games. And so that was a that was a really fun project to work on.

14:39 Yeah. And then today you're you're doing educational stuff on a bunch of subprogramming performance on your substack. And what else are you busy with?

14:47 So we do actually have a an unannounced project that we've been working on uses up sort of the rest of the time that that I have if that makes sense which is not always so much. um and that we will we're hoping to announce it sometime soon. Um but it it it's not quite out yet. Believe me, I will I will you will hereby I will send you an email as soon as we have an actual announcement. But uh given the fact that it is kind of like a split time sort of thing for us because we're pretty focused on making sure the substack is good and all that sort of stuff. We're we're trying to keep it fairly tight lipped until we actually know we're mostly done because we don't know how much time we can always devote to it if that makes sense.

15:25 Yeah. No, it's it's it's it's pretty typical games related things, right? Like tight lipped until you have something for good reasons.

15:34 Well, sometimes people play the other game. They go like, "Look, we're going to day one we're going to be very loud about this and try to build a community around the development of the game and all that sort of stuff." And that's great. Uh so, you know, that's another route you can go. But if it's not your full-time thing, if you have, you know, other responsibilities, that doesn't seem great, right? because you you don't

The root causes of industry-wide performance neglect

15:56 have any insight into how much time you will actually be able to devote to it, right?

15:59 Yeah. And on on Substack, it's called computer enhance and you started it with performance related topics and that's how we started to talk I think about three years ago when when you already had a Substack and we we had a direct message conversation. I remember you messaged me saying, "Hey, Gerge, why do you think in the industry people software developers just don't really focus on performance?" I I think you specifically wrote

16:25 I did

16:26 uh you were saying how there's there's little emphasis on performance. There's even though there seems to be overwhelming evidence that performance is critical to the bottom line of of most software and I wanted to ask you we we've had a good back and forth on this and actually I think initially I told you like oh here's why you don't need to care about performance when you're building I don't know distributed systems or like but since then what have you learned? Why do most developers don't care or even most companies, teams, engineering teams not care about performance all that much? It's a really good question and I think I do think your answer at the time if I remember it correctly is certainly an accurate one for some subset of of industries uh which you know you said something along the lines of look a lot of these pieces of software that you're seeing the user isn't the purchaser right like you were you were like this is some kind of thing where you know somebody very high up is going to look and say we need software for managing HR they're going to look at the cost of the software they're going to look at the compliance terms terms of the software, the legal liability, whatever, right? And then they're just

17:27 going to make a purchase decision on that sheet. They're not in there looking to see whether it takes like, you know, whether there's a 30-cond pause every time you want to try and access somebody's record, right? And I think that's very true. Like unfortunately, the situation for a lot of enterprise software probably is that way. So maybe an individual might well be upset about the performance of that software. And I certainly hear from people all the time who are upset about the software that they use. They might not be in any position to change it. I think that's one thing. There's thing two which is that in a lot of cases you simply have monopoly effects. Uh you know people aren't right now realistically going to challenge the social networks that currently exist. For example, people have tried. It's very hard. you know, Blue Sky and Threads have tried to assail X and you know, you've got Facebook and Instagram and Tik Tok and they kind of just own those spaces, right? And it's very hard to push into those because of these like network

18:28 effects and maybe performance could be part of a package where you try to take on one of those players. like, hey, look at how much more responsive our thing is than theirs. Might be a nice plus, but that's not going to be sufficient. If you just show up with no plan for how you get adoption, no plan how you get big influences over there, all that sort of stuff,

18:49 it can't sell a product on its own into a monopoly space, right? If you're just talking about apps that someone can choose to download, maybe you've got a shot there. But those are just they're forming a smaller and smaller subset of what software is, right? and this bigger and bigger subset is like these monopoly platforms you go on uh to to sort of work with. So I'd say that's another thing. Thing number three is I think now people sort of are caring about performance more. I think over the past decade uh the people uh including myself but many many other people who have been saying that this is a problem uh have actually had some effect like I don't think that it was a waste of time. I'm seeing a lot of new emphasis on performance, people talking about performance, people posting benchmarks on things. And so I actually think that the third thing is well actually it kind of does seem that pointing at this issue and saying this is something we should be doing better has not been completely uh a waste of time. I I do see things as sort of starting to turn around a little bit. I also see people attacking major product categories now with

19:49 performance-based pitches. Things like File Pilot or the Blick video editor, like things like this that have been coming out lately where it's like, oh, really performant software to try to take on uh incumbents in a space and they've been getting traction. So, I think that's also a really good sign.

20:03 Yeah. And I guess on on this last category, a really good example in the developer community is bun versus npm where bun just said like okay we're like 10x or 20x or 50x faster and devs are like what this possible and then it was so they there was this outrageous claim. I almost I I wonder if like you need to have these outrageous claims cuz devs started to pay attention cuz it was 10x faster in many you know categories. You know, linear versus was gyra is also a good example where linear have this benchmark of okay, they have 300

2 Highlight · The illusion of acceptable software speed

20:38 milliseconds for any action and gyra of course we know is is just slow because they have a bunch of complex you can explain why but it's slow. They it was never built for that.

20:48 Yeah. And you know if you think about something like a 300 millisecond budget for an operation 300 milliseconds is like an eternity in computing, right? And so if you're talking about like our pitch is that we're not more than 300 milliseconds, that just shows you how the bar was so far past where it probably should have been for something. And you see it everywhere. You know, I you go on to programs and you're waiting sometimes seconds for an operation. And I don't think people realize just what an eternity a second is in modern computing. Uh especially when you're sitting on networking like that has, you know, sub 10 millisecond ping times.

21:24 Sometimes you're talking about this, you know, the actual packets had to travel physical distance to get to this data center and that was being done far faster than this very simple operation that you were failing to do in a reasonable amount of time. It's just like we are massively underperforming and people don't believe it when you say 10x 100x but it's actually true and we've seen a lot of proof of it as you point out. I do wonder if one part of the not really much focus on performance is that a lot of developers don't know the the baseline thing. And I I'm reminded by Simon Ericson, the founder

Optimization requires measuring against theoretical hardware maximums

21:56 of Turbuffer, have this has this project called Napkin Math where he did a list of mostly networking operations. How long does it take to transfer one bite between like two AWS data centers? How much does a gigabyte take? How much does a terabyte take? how much does it take uh to to write an SSD to an N an MVM an N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N VME and so on. And so he had he had these numbers and he said that what he found is whenever inside Shopify they were deciding do we choose vendor A or vendor B as a database they would just run uh a benchmark that they would write themselves and they would get like okay like I don't know storing this and this it takes 2 seconds on this one 10 seconds on that one we will choose a two second one and he looked at it and said like hey like this doesn't make sense like the amount to to store in a file system like here's a theoretical limit which is I don't know 100 milliseconds.

22:47 Like there's no way that's going to be 10 seconds. And it often turns out that he found that the benchmark was just wrong. They were benchmarking the wrong thing and they were making decisions. So I wonder if there's a thing where many engineers, developers are maybe just not aware of how truly devastatingly slow this thing is versus the resources you have.

23:06 Uh that is the entire point of like my substack, right? So what you just said

3 Highlight · Why profiling alone only finds local minima

23:12 is exactly true and it is the thing that I hammer home on the substack through all the parts of like the courses on there which is that in general there's a misconception about the way that you approach optimization in uh like in computer science or in whatever software engineering let's say and that misconception is that what you do is you run a profile you identify where the like you know big parts of the profile are you make some changes to those and you you measure like statistics the better the statistics you know the the the more uh statistics you can get the better and you look to see if those statistics improved if they have that was a good change and you proceed as such and this is completely not correct that is not how anyone has ever you know I I've worked with many extremely good optimization people and that is not how it is done the correct way to do optimization is very much like what you just said you first go what are the operations that this system has to perform form. What is the underlying hardware capable of doing at its

24:13 theoretical peak? And then you measure the delta between that theoretical maximum and what you have achieved. And then your goal during optimization is to shrink that gap to something that you think could be plausibly explained and hopefully come up with explanations of why you aren't at theoretical. Because often times you can't hit theoretical. That's why we call it theoretical, right? And it's crucial that you do this because otherwise all you're doing with that other method is finding a you know with with the with the I'm just going to you know make something I think might be an optimization and look if my statistics improved. All you're doing there is finding a local minima.

24:48 That's all you're doing. You're just you're just you know you've got this this shape of your performance and you're finding some little spot and you're sitting in it. That's not optimization. That's improvement. But optimization means to make optimal, right? means we're going to find what we actually should be able to get this machine to do. And so, uh, you know, that's why I emphasize that approach because it's the one that I I've always seen great optimizers take. That is how they get good performance is by knowing what the maximum could be. In addition to that, it also is what lets you become better at optimization. Because no

25:22 matter who you are and no matter how much you already know, when you go to tackle an optimization problem, there may well be some things in the new like way that the system is laid out that you don't know about, new uh things that people have not figured out about modern CPUs, new things that are different about the network backplane, new things that are different about the GPU drivers, who knows, right? And if you don't have some theoretical maximum to look at and to measure your delta from, you don't know if there's some serious anomaly there. And we you would be surprised at how many times we find anomalies like this things in CPUs that no one knew about. And we, you know, like I've literally had them in the course of making the Substack. I've been like, "What is this thing?" And I look into it's like, "Oh, there's this new renaming this this new rat table thing that Intel chips seem to be able to do.

26:12 We didn't know about that." And that's like a new thing we have to model when we talk about how to do performance. And so the that's the other crucial part of I guess what you were calling napkin math. I also call it back of the envelope. That's the term I've heard used for it. Often times they're kind of interchangeable, right? Knowing what the theoretical is is how you learn as well. Uh how you learn about new hardware and new performance op options.

26:34 Interesting. Plus by doing this you're just learning. You're becoming better professional. you understand more about given hardware or or the inner workings of of your computer or or software stack or kernel, you know, all the stuff that I guess goes way beyond the the the b the vanilla programming language like cuz you can say I'm an I'm an engineer. I'm a software engineer because I know how to use this programming language, but I'd argue you're probably an

Read assembly to see CPU execution

4 Highlight · Using assembly to verify compiler output

26:57 engineer if you can go down the stack and you have that ability and you have like a good good understanding of some of it at least, right? And you can learn the rest. And I would also say that one of the other things that uh we do in the class is teach how to read assembly language. And people often ask what like why like you know assembly language what would I ever need that for? Uh there's a very good reason for it. And that is that everything else that you might use doesn't tell you anything about what the CPU is actually receiving. You know if I look at a Java program if I look at a C program if I look at Haskell OAML whatever right uh Rust all I'm seeing is input to a compiler. I have no idea what the CPU is actually going to be asked to do. If I look at the assembly language output from that compiler, I know exactly what the CPU is being asked to do. And it's not that hard to be able to learn to read assembly language so that you can see very quickly is the CPU being asked to do the things that I think it should be asked to do them. And in that way, right, you don't have to write it hardly ever. Uh it's very rare

27:59 that you have to write assembly language to do anything. um other than sometimes for test purposes it's easier to do that so you don't have to try and convince a compiler to output something. So if you're just testing something, sometimes it helps to be able to write some assembly language. But if you're just talking about the uh vast majority of tasks you might do in optimization, writing it, no reading it essential. And it also unlocks this sort of uh huge world of possibilities to you because once you know assembly language, you can now do things like read those CPU diagrams like you know when they announce a new processor, they put up a little diagram. That diagram tells you stuff like the fastest this thing could do multiplication and stuff like that.

28:36 It tells you that if you know a semi language, you can read it right off the chart, right? If you don't know a semi language, you look at that chart and like I have no idea do what I'm looking at, right? Like it's just this weird flowchart that doesn't really tell me anything, right? And so one of the really great things about s unlocks all of this knowledge for you because it's

28:50 the it's the it's the actual uh input language to the machine and it allows you to figure out how it's operating.

28:57 Plus, I guess we should add that assembly language is not all that complicated, right? Just just by by by nature, it's a far simpler language. Okay, it's harder to read if you've never seen it, but it's in terms of the number of operations. It's so barebones because you know that's what assembly is. Like every single higher level language will have way more keywords, structure, whatever you name it, right? Than assembly,

29:19 massively more uh and especially when you consider the subset that are actually used. If you look at the subset of constructs that you would need to understand to be able to understand um say just a website from today, all of the JavaScript libraries, all of the JavaScript syntax, all of the DOM, you know, all of the behavior that's going to go on there, CSS,

29:40 React, CSS, right? all of that

29:43 assembly language, you know, maybe there's 20 30 instructions you might have to learn total because most assembly most things in legacy assembly like x64, most of them are hardly ever output by the compiler. So you only need to learn a very small subsets. That's the ones that's actually going to be that you're going to be seeing in 90% of the cases. It's so much simpler. And also when you're looking at performance, you're typically only looking at a very small part, right? You've kind of understood roughly what's going on. and you've you've seen the basics layouts of your program. you've identified what's supposed to be happening and you're just looking to see like wait why is this part which I don't think should be running this slowly why is it running this slowly it's just a very small piece you typically end up having to look at as well so it's really much easier if you can understand how to center a div as they say if you can vertically center a div in HTML then you can probably learn assembly language I would say

30:33 okay you're super passionate about performance optimization you also have really good educational materials both

Avoid false serial dependency chains

30:40 both free videos your paid substack the free parts of it, etc. But let me just play devil's advocate. There's this saying that premature optimization is the root of all evil. And we typically use it or I typically use it so many times. We're like, oh, should we make this performance? Should we optimize the thing? And like, nah, let's not do that. Let's first build it. Let's see if it's good enough for our customers, for ourselves, and if we need to, we can always optimize it. I mean, you know, like it's it's not the hardest thing in the world. Okay, maybe not as good as how you mentioned cuz maybe I don't read assembly but and that's that's kind of a thinking of building you know like I guess SAS software building software at big tech. What is your reply to that cuz I I'm I feel really good that I made a really good argument here.

31:19 So I guess what I would say is the important part about that and I guess I'll divorce it a little bit from the saying. I have an entire lecture on that saying by the way. It's it's like two hours long and I I gave it at better software conference this year and I believe the VOD we're linking that in the show notes below.

31:37 Okay. It'll it'll be like a week or two I think till it's up. So it may be right at the same time as this. Uh but so if you want to find out the history of that phrase you can go look at that. But I wanted to talk about the idea behind it because I think there's uh I don't want to dismiss it entirely because it's not entirely false. And the idea is that well I'm just going to delay optimization work. I'm not going to think about that and then I'm just going to make whatever I'm going to make and then you know either myself or maybe I'll just hire some performance person to come in and clean up the mess later.

32:07 Right? So here's the positive side of that first. The positive side of that first is for some types of code that will work. If you happen to have written some operation poorly where the optimized version of that operation just looks like someone taking a loop and changing the loop from your really like you know naive version to a really well optimized version.

32:30 The the the typical of like I wrote a bubble sort we can later optimize that.

32:35 Who knows right? Anything of that form. Okay maybe we can just do that. So there are certain times where you do in your head want to be doing this where you want to say okay I could go spend a week researching the fastest hasht implementation here but part of software engineering is being smart enough to know it won't matter if I do that now or later the architecture around this piece won't have to change I'm quite certain because I understand the problem well enough so it's okay I can defer that to later maybe it's never too slow with the naive when put in there and then we don't have to do any work. Maybe it's too slow later. That's okay. I just target this one hasht implementation and we'll get as fast as we need. Right? If you're doing that, if you're applying that true engineering mentality to it, you don't have a problem. The problem comes when you don't know if the choice that you're making produces that kind of optimizable hotspot. And I'll give you a very simple example that usually people

33:37 have had experience with. A very simple example would be we write our entire software thing like we just whatever this massive thing that we're imagining doing where we're going to ignore optimization. We sit down and we write it and we use a paradigm where we ask the server for something. We have like some API, you know, that we've built for asking servers for things. We ask the server and it returns to us what the server's response was. And we that's like kind of how we architect this thing. So everyone writes, you know, hundreds of thousands or millions of lines of code and they all look like ask the server something, do some calculations, ask the server for the next thing, do some calculations, right?

34:12 Then at the end you find this is way too slow. But that's okay. You weren't worried about that cuz you're like when the end you call in some performance experts. They look at and they go, there's nothing we can do for you. Sorry.

34:21 Why? Well, the reason is because you

5 Highlight · Serial dependency chains destroy system performance

34:24 created a serial dependency chain. All of your code looks like wait for a network request to come back, do something. wait for a network request to come back, do something, wait for and that serial dependency chain can't really be shortened without just rewriting it. If instead you had made the paradigm and told your programmers, look, here's what you need to do. At the top of every operation, you need to figure out all the things you might want to ask the server for, you ask them for all of those things, right? And then you do all of your processing there. And you only create a chain of dependencies if you absolutely couldn't have determined what it was you needed to ask a server for. Now you're just in this situation because you didn't tell them to do that.

35:05 You have to rewrite all your code. Everyone is now going out rewriting all the code if they even can. If it's even possible to really do that in a way that's not slower than just rewriting the thing, right? So what happened there? Well, again, we talk about this a lot on on the Substack, but there's this idea of a serial dependency chain. It's when you stack things in order, right? And the performance of your software is generally determined by the longest serial dependency chain because it's something that cannot be parallelized.

35:31 If I have thing A that then B depends on that then C depends on that we cannot shorten that because it has to go in order. Everything waits for it. We can't multi-thread it because it's dependent. We can't uh you know make it run wide. We can't you know uh amortize the network request whatever. that kind of thing can be pervasive in the programming and we can't cheaply remove it cuz it's not a hot spot. It's a way that you did things. That's the part where that kind of thinking breaks down.

35:57 If every uh software engineer knew to watch out for false serial dependency chains, things where they were creating series of dependent operations that could not be optimized away or other sorts of architectural problems like that that cannot be easily fixed, then the world would look more like just wait

36:16 and optimize the hotspot, right?

36:18 Yeah. So this this is the architecture, the planning, right? Like if if in that phase you're like okay like as this thing grows like what will get in the way of performance what will slow it down or you can ask all these questions or like different flavors of the questions you know or from the other side and so on.

36:33 Yeah. Another way to think of it is cuz hotspots is the way that people talk about that like oh it's it's going to be hotspot optimization. We just got a few spikes. Someone will come in and clean up those spikes and we're done. Right. The way to think about it is your codebase will not end up that way by accident in most cases anymore. you have to engineer upfront for a [snorts] hotspot codebase that people can then optimize, right? And so that's the crucial takeaway is everybody on your team who is making architectural decisions, those people must know performance and they must make decisions that will allow the other people downstream of them to use an architecture which can be optimized later. If you don't do that, you're just rolling the dice. Casey just talked about how engineers making architectural decisions should know about performance.

37:21 This is also true when choosing your dependencies like which database to use. And this is where I want to mention our season sponsor Turbopuffer. You already know how Turboper is a vector and full text search engine. But here's an interesting story from Linear on what happens when you stop thinking of Turbopuffer as a search engine and start using it as a primitive to reduce latency. As context, Linear is a local first app. So each client keeps a local database and when that client goes back online, it needs to catch up with what happened and do it fast. Their biggest workspaces generate around a million sync actions per day. Doing catch-ups by reading from post was getting slow for large reads. So the tail latency got too large and adding more replicas did not help either. Linear solved the problem cleverly. They started using Turbopuffer as a serving index for each client. This is because Turbopuffer itself is built on top of inverted indexes. So for every index value, it stores the documents that that value can be found in. The lookup cost for such an index is constant. So linear took this structure and had each client's index point to the

38:22 changes that they needed to sync. As a result, not only did they reduce latency, but they kept it constantly low matter how long the change list is synced to the client is. Linear published a blog post about this refactors titled rebuilding linear's delta sync read path. Check it out. I love this story because it shows how important it is to choose the right primitives and how good primitives can improve your system. If you're building systems where you store a lot of data or serve a lot of data, Turbopuffer can probably speed things up or save on your costs. Learn more at turbopuffer.com/pragmatic.

38:52 I'd also like to talk about a presenting sponsor and this is while Casey deliberately does not use AI coding agents for his work. Most of us do. And when you work with coding agents, your job is no longer writing code. It's specifying and testing it. Antithesis is the most effective method for verifying agentic code today. Let me explain how it works. Antithesis runs your whole system in a hostile simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, and this doesn't only find bugs, it gives you a perfect reproduction of every issue. To create such a tool, the anticys needed to invent new kinds of debugging tools as well. For example, here's what's called a bug probability graph. The xaxis is virtual time and the yaxis is probability. As anticis runs a hostile simulations. It plots time frames when the bug probably increases which greatly helps with finding the root cause of the bugs and antithesis also has a log visualizer. Vertical lines going down represent events branching off from the same state. And the purple dots are where the bug happens. Antithesis is as good as it gets in being able to ship

39:52 agent written code. It's what teams at Jane Streetfly.io and the CD community used to ship with full confidence. Head to antithesis.com/primmatic to learn more. And with this, let's get back to Casey and how if you don't

Language rewrites expose architectural limits

40:04 design an architecture that can be performance optimized later, you're just rolling the dice. And we've seen so many projects. I have an entire video where I go through like look at all these blog posts of people who like say we, you know, it's Facebook, it's Uber, it's everybody. They've got blog posts of we had to rewrite this whole thing because the performance was bad. If it was always hotspots that made your performance bad, you'd never have to rewrite the whole thing. So, we know that that doesn't work anymore. Uh why?

40:29 Because of the things I just said. I was at Uber where I I was not making the decision but the teams next to me were and I saw or I I kind of understood why they were making but typically and right now it's happening with AI companies oftent times it's like we chose this technology which is Python and it's single threaded and it made sense at the time on the web server but now we're big and this happened at Uber it was it was Python and NodeJS and then they went to go and Java on the back end and now with AI companies it was Python open AI and Tropic are both going through this right now uh They're both either public about it or I I've written with Antropic. They they they share with me with with me, but I I put it out there. They used Python because data scientists or AI or machine learning engineers knew Python.

41:11 They put on a bunch of web servers. They had their API run on it. Initially, they just, you know, scales horizontally, but now they're like, well, if we move move over to Rust, they right now they're choosing Rust or or Go, but I think it's Rust. Well, we can actually have multi-threaded and the same machine can actually handle more connections. So, cool. I came across a lot of that because I think that's easy and safe to communicate because it doesn't look bad on you. But you're right, a lot of times I don't think on engineering blog post you'll get the real reason that these companies put out there like when it's kind of a very kind of you know easy to own mistake or not mistake but just the decision which made sense. they'll tell you. But if it's something that was an oversight, you're not really going to get that on a on a public facing engineering blog post, except for maybe some startups who are really there. But don't forget like a lot of those blog posts are going to help someone get promoted or get recognized and they will always be way more positive in especially when there's a content writer team which large companies do have. So it's it's not quite PR but it's

42:12 somewhere midway in between. And I mean, yes. And also, I would just point out the fact that like the fact that these things are happening though is all we really need to know for the signal, right? Because in general, this should not be happening. If the the ideas about activation were true, you'd never have to rewrite something in a language in a different language unless you just preferred that language. It would just the story would just be we rewrote it in this language because we wanted to use this new language. would never be um or for Rust it might be just memory safety.

42:45 We see those blog posts, right? It's like why did we write into Rust? It wasn't performance. It was just we wanted the memory safety or something like that.

42:50 If I'm a software engineer, programmer and I'd like to just get better at

Recognize orders of magnitude in performance

42:57 writing performance code, I'm interested, you know, maybe after this podcast or or looking into some of the things that you did. What is a learning path you would follow outside of your substack where you cover a lot of these things, but what are areas that you think are kind of like you need to understand these things to like get better at writing performant code? I think it's actually very simple and perhaps a little bit counterintuitive.

43:22 So, I'll start with the uh the very good news about learning to write uh performance software. The good news is that optimization of the kind that we sort of talked about, the like hotspot kind where it's like somebody's going to go in here, maybe they're going to even rewrite this routine in handcoded assembly or something crazy like this, right? That's very rarely necessary these days.

43:49 One of the reasons that you don't see hotspot optimization as a thing that really matters that much anymore and one of the reasons I advise that architecture and and not making bad decisions is much more important is because a lot of libraries already have been optimized for you that you might use. CPUs are incredibly good at taking bad code and running it quickly and so on. So, typically when we're talking about the causes of performance, uh, negative performance that aren't squeezing every last little thing out of the hardware, but rather just making sure this thing isn't running like a hundred times slower than it should be, usually it's more just about having an awareness again of what the computer should be able to do and making sure you're making uh software architecture choices that allow it to do that. And if you do those things, you will generally be within, you know, 2x or something, which is 50x better than the people who are 100x away, right? So the good news is in order to write software that's much better than a lot of the software you use today, you don't have

44:49 to know that much. So what do you have to know? What I argue and what we focus on the substack is I think you just have to go through the experience once of learning reading the assembly language seeing how the CPU works seeing the difference seeing why Python is slow which we show on the sub. So one of the first things I show is uh I walk you through the assembly language necessary to execute uh A plus B in Python and it's so vast that you know I have to skip most of it. It's it's massive right it's like this huge and whereas you know if you have the equivalent function in C it's one instruction add right so you know uh understanding basic things like that if you go through learn to read assembly language learn to look at some code learn to do some CPU timings and you just have that experience just spend you know uh a month or two of nights or whatever you want just understanding some performance stuff and going through a few examples where you play with it and you see the difference

45:49 and and just so I understand you're you're saying do this not because let's say you're doing iOS development or or web development read what React like you you will not look at the assembly that the React does but if you do this on a project you will be able to conceptualize what is likely happening what the layers are and you you might be able to decide like do I want this layer or do I want to use let's say WebGL If you're a React engineer, you probably haven't touched it. But again, you can skip a bunch of those things and it comes to trade-offs with maintainability, yada yada, but that now

6 Highlight · Understanding the massive instruction overhead of interpreted languages

46:25 you you you will know like kind of what you say by keeping this layer or not keeping it and so on. Is it do I get that right?

46:33 Essentially, yes. And like uh you know the simplest example is the Python example. Most people have never internalized the fact that it takes, you know, maybe on the order of a hundred more uh CPU instructions to do an ad in Python than it does to do it uh in an equivalent language like C for the same piece of text, just A plus B compiled in two different, you know, in two different languages, right? And so just understanding even just that is enough for you to kind of go like, "Oh, okay.

47:00 A, now I kind of understand why if I'm using Python, I kind of have to use libraries to do things." And those libraries were written in C. Because it's like if I'm ever going to do any operations on a on a large number of things, I can't do it in this language because the amplification factor is so high on each operation that it just, you know, kills the performance immediately. whereas these other languages don't have that, right? And so understanding those orders of magnitude and what's actually going on, I think that allows the programmer to know, okay, if I think through what I'm doing right now, can I afford the super slowdown that I'm going

47:38 to take? And usually I don't think you have to be a performance expert to make that decision. You could usually know like, okay, is this a part of the code that can afford to be 100 times slower than it should be or not? Right? And uh you know most people can I think make that decision fairly logically. And if it's not then now you know like oh okay if I'm in Python then what I got to do is either I got to go find a library call that will do these sorts of things and structure around how that library works or I should maybe get something like Syon or something where I can do compiled stuff inside my Python and make my uh code work around calls out to that kind of code. You know, you now have the tools you need upfront to make sure that when you write the program, you've put the parts that needed this and you've structured the code in such a way that you are only paying the 100x on things that you know are very infrequent or happen like only uh you know once per every so often things like that right that's I think the biggest thing is just

48:38 the knowledge and once you know you can start to make much better decisions in any language because it doesn't take you very long you know a simple le search or you know asking an AI or whatever is the common practice that you're going to do. A simple bit of that once you know what

Understand basic CPU mechanics for better design

48:55 you're asking for will get you this information back very quickly. Right? You just have to know that you should have been thinking about it.

49:02 Now, as a software, you mentioned it's good to understand how the CPU works. As a software engineer who is not a games developer, I'm not doing low-level stuff. What does that give me? because for the most part even in academia or or in computer science you know there is some level of of some basic CPU theory taught but usually we just kind of we stop at the code okay maybe you look at the assembly but you rarely go further than that the folks who you know you've you've taught and they they learn these things what do you see them get out of this that they wouldn't otherwise

49:39 so you're talking about specifically the knowing the CPU part

49:42 knowing about the CPU, knowing about the details about a CP because because you mentioned that that's also part of it, right? It's not necessarily just stopping at assembly.

49:49 So the reason for that is more the other way around the the reason to learn the assembly language is so that you know what the CPU is doing. So it's the CPU part that's actually important and the part that's important about it is that the CPU is basically you can think of it as a little machine whose internal gearings we are not privy to because for the CPUs that we care about. So you know an M series CPU in a Mac a Zen core CPU in a server uh or in a laptop or an Intel you know core series those sorts of things. These CPUs are not documented at the level where you're going to be thinking about how each little

7 Highlight · Mental model of CPU architecture for software engineers

50:29 individual part works. And to that end, it's unclear that you would have time to do so anyway because these are massive, very complicated machines that we're talking about, right? But from a high level, from a more blackbox perspective, they are machines that we can think of in relatively straightforward ways once we know kind of what their core instructions are that they tend to execute. And they break down into a couple different categories. There's how does data move into and out of a core. And this is basically how like load store units work, how the cache levels work, L1, L2, L3. Some we have like we have L0 now sometimes things like this. How does that work and why? What is the granularity of it? What is the policy?

51:14 How does the CPU go about actually working with those things? Understanding that part of the machine is crucial because when you're working with a lot of data, the the difference can be massive if you structure it in one way versus structuring another way, right? Again, architectural decisions that have nothing to do with hotspots, they're how all the data is laid out and what the access pattern is, right? Things that are very hard to change sometimes. So, that's one part of the machine you want to understand. Another part of the machine you want to understand is how the instructions flow through it. And you, you know, a lot of people have heard about like branch misprediction or

51:46 things like this. eye cache misses. There's words that you might hear, but you're not really sure what they mean. They're all actually pretty simple to conceptualize. Sometimes they're they're harder to pin down exactly how they work because branch predictors are, you know, getting more and more complicated and so on, but you can still categorize the behavior of them and understand how code flows through it and when you might care and when you won't. And then finally, there's the execution unit scheduling part, which is about knowing what's the raw sort of throughput for any particular type of operation.

52:12 floatingoint multiplies, integer additions, division, whatever it is that you might want to know. Right? Once you learn a little bit of assembly language, you understand what it's reading, you understand how it turns those assembly language instructions into micro opterations, which it actually does, and how they get distributed through that machine. That flowchart that they put like basically up on the we've announced the new Zen core, that flowchart, you can look at it and go, I know the performance of this machine roughly, right? Not exactly because like I said there's all these little edge cases that if you really want to be a crazy optimizer which again I don't really advocate people do. I don't think it's important that they be like crazy hyper optimizers. If you like to do it great it's a lot of fun but it's not the important part. Just look at the C like okay I see what the CPU should be getting in terms of like what I could do with this size data load that size data load. This is what I could probably get out of it for if I was doing a bunch of like how to do a bunch of like math ops on it you know. And and I think that's that's just something that should be kind of par for the cars to software engineering. You go to school for four years to learn this. There's no reason

53:14 you can't learn this in a few months. It's not that hard.

53:17 Yeah. Plus, I guess just from a craftsmanship perspective, like we should know our tools. We should know the machines that we're programming. Obviously, we know that our code will be if you're doing web, it'll be running on all these different things or if it's a if it's mobile on all these different phones, but from a conceptual point of view, like we should be able to know what's going on. So I I feel there's a bit of a pride as well. Like if if nothing else, you would learn a bunch of stuff. Like I I know some of it, but I'm now getting motivation to learn more about it.

53:43 I do think there's a craftsmanship angle. I think there's a a large number of people who maybe don't feel fulfilled when they I' I've certainly heard from lots of people who when they write something and it's just kind of this amorphous highle thing, they they don't get as much satisfaction out of it. And then when they learn how they can look more deeply at what's going on, they feel much more satisfied. even if they didn't change what level they were programming at, they feel much more satisfied that now they know what they're doing, right? And it's like, oh, I see and I understand why this thing was happening this way and this thing was happening that way, that's very satisfying, right? So there there's an aspect of that. I also want to emphasize another part which is that it's a percentages game. If we convinced enough library maintainers that this stuff was important and the libraries all get a lot faster, all of a sudden all the people using the libraries code gets a lot faster and so on and so forth. If the APIs start changing to make it easier to optimize the libraries because people now thought that through, right?

54:40 Like it's infectious. The more people are doing performance, the less people need to do performance, if that makes it kind of paradoxical, right? Well, plus plus I I I I do think that right now there's still an edge in in just being performant again and you mentioned but there are categories of software that is just winning by being much faster and to do that you need to do this and if you know how to do this maybe you're going to spot opportunities as a software engineer maybe right now you're not as happy in your position but maybe start something or do a side project that turned into a full-time thing and so on and so forth. So like I feel there's like and worst case you just learn like net new knowledge which will probably not be as outdated with AI which we we'll get into later but this stuff it it feels it just feels very interesting right like kind of it moves your brain. It does and thankfully like uh it's also not that hard to update your knowledge because again you get these presentations that that the CPU companies give and they're like here's the changes we made. So you kind of are

Early game development was defined by massive technical risks

55:36 aware every time a new thing gets you know there's little tiny things that creep in that you don't that you know but again there's people out there who are running lots of microbenchmarks that you will find out about and they often uncover these for you as well right uh so so I I wanted to talk about games you've been you've built games for like decades at at this point can you give an overview for those of us who are not in the games industry how is a game typically built from the games that you know of that you've observed or or you worked on especially trying to compare for you know like in typical SAS or distributed system or or something we're building a a website or service is kind of you plan this stuff you you know we'll we'll do an estimate we'll build it in a few months or a few weeks we deploy it and then we monitor it and then we keep tweaking it and then you know fast forward 5 years later it's now this like gigantic thing with microservices but but it keeps evolving right like it's we do this like a lot of prototyping thing for games It's pretty obvious right from the get-go as we're

56:36 talking like there there will be a launch but can you when you're inside or when you when you join a game studio like what would you observe there in terms of what the process is like and how it's different or or how it feels weird compared to like this I guess I don't know traditional software SAS software whatever development that is. So, I guess what what I would say is unfortunately I'm probably the wrong one to ask because my knowledge is outdated at this point because one of the things that has happened to games recently is I feel like they've moved closer. I don't want to necessarily say entirely in development practices but at least in terms of the nature of the product has changed somewhat dramatically to be more like something like SAS where you know if you take some of the most popular things that are being in terms of dollars let's say so I guess maybe popular is kind of nec might be hard to say specifically but let's just say revenue generating so if we were to measure the total games industry revenue

57:37 and you look at what are the largest slices of that, you're seeing things like Fortnite, like Roblox, like Grand Theft Auto 5 online, etc., etc., Minecraft. These things are starting to look a lot more like an always on live service kind of we ship incremental features to our customers uh kinds of things. And so I would actually say that, you know, I I'm I'm the wrong one to ask about what that actually looks like from the inside because I haven't actually gone and worked at one of those companies. I have friends there, so I hear things, but I'm not the right one to like give it an accurate picture of it. But I just would point out from my perspective, the game industry practices look different today than they did when I have a more intimate sort of experience with what we were actually building. But c can can we talk about it when you were building games which was you know 10 plus years ago when I understand these were the games where they were built they were launched you know maybe they got a patch

58:37 or two and then they were kind of you know the team moved on they were disbanded. It was this time box thing that was a lot of development a big launch and and either it it was it went big huge hit or you know a huge failure right and then the studio goes bankrupt. So how did that work? because I feel that's a world where okay today a bunch of games don't have those constraints but it has a bunch of constraints and I'm interested in in what worked in those constraints. So in the early days you didn't have licensable engines. So up until sort of the point like nowadays like this is why I say it's a lot different now than it used to be. You know nowadays you think of web development like I'm going to go go grab like a thing like react and I'm going to make this thing or whatever. I'm going to go grab an off-the-shelf database thing, Postgres or Oracle or I don't know like what would be the thing of choice, right? But

59:25 by the way, just definitely Postgress and not Oracle for most people.

59:28 Okay, sorry. Sorry. Postgres, uh I didn't want to slight anybody there, so I apologize. So, okay, definitely Postgres, sorry, Oracle. Yeah.

59:35 Um, so you're going to go use some kind of a variant of of one of these offshelf databases and so on. That's more, like I said, what people might be doing nowadays, too. Like they'll grab the Unreal Engine. They're not going to develop an engine on their own. They'll grab backend server stuff from people. It might even be some Postgres in there, right? Like, who knows? In the earlier days, none of this stuff existed for games. I actually worked, like I said, in middleware at the time. So, I was actually sort of one of the people who was working at the time on maybe changing that a little, like producing code that would get reused throughout games, which was actually fairly rare.

1:00:06 And and and so and so back in the day, every game built their own rendering engine, for example.

1:00:10 Correct. And so,

1:00:12 yeah. Yeah. Yeah. Uh and so really early on, right, if you if you rewind the clock far enough, uh then yeah, the the degree to which people were reusing code for their thing for like their rendering engine, it'd be like cuz I got some code from like Dave or whatever who was or we were both at Atari and somebody wrote this good routine so we used it, right? There was that kind of thing, but there wasn't like this sort of set uh engine.

1:00:39 And the time when that sort of maybe you could say first started happening a little more widespread was with things like what ID Software did where they sort of started having like oh you know like someone's going to build something with the Doom engine or someone's going to build something with the Quake engine. There was also the uh the build engine at the time uh made by Ken Silverman and some things like that. So there were there were some early cases where a few people would make a game, but they were making a game very much like that. Like if we did the Doom engine, we're going to make a game very much like Doom. So it really was the case that for most games, people were rebuilding most of the things from scratch, at least for their studio. And studios often their existing code base was kind of part of the value of the studio, too. Like if you are uh think Blizzard and we just built Warcraft 1.

1:01:26 Well, rolling all of that knowledge and code into Warcraft 2 is a huge advantage for us because everybody else who wants to build a competitor to Warcraft 1 has to do all of that from scratch. They have to make the path. They have to make the level editing tools. They have to make the rendering. They have to make whatever. And so, you know, that was how things were traditionally done.

1:01:47 No wonder the games industry is so secretive compared to the rest of the software engineering industry. Like seriously,

1:01:53 used to be

1:01:54 used to be at least. Yeah. maybe now it's changing

1:01:56 and so there were two really big risks typically uh when you started a game project in in those days one was the engine risk would we be able to make something that would be technically able to do what we need to do for this game and that risk comes in a lot of flavors one will it happen at all two will it happen fast enough for us to actually reliably build the game on it right one of the things I mean I don't how detailed an answer you're looking for for this question. So stop me if I'm going down too many tangents, but one of the things you also have to remember is that at that time, and this is sort of still true today, but at that time it was very important, there was no way to really buy something all that much faster than what you had. There was not a huge strata of PCs that you could, you know, buy or anything like that. So the rendering engine, there wasn't like a way your level designers could like be playing on a faster thing than the consumer would have really. you can only have the machine that you have now and if this game comes out in a year that's

1:02:57 sort of roughly what the consumers might have or a little bit but you know so there are people who started doing things like buying SGI workstations because those were actually faster enough right and and things like that is what you know you kind of had to do uh and so on so anyway so there was a huge engine risk and some games just failed because they couldn't produce a thing that could technologically do what they needed you saw houses who survived on technological prowess you had ID software that was kind unrivaled at making those kind of first-person engines. You had Bullfrog who had this engine that the pseudo3 engine that they did for like the racing games, Magic Carpet, Dungeon Keeper, like they were all based on this, you know, one core tech and all the sorts of things.

1:03:36 There's that engine risk that was huge. And how do you mitigate it? You didn't. You were just grit, right? Because there wasn't a way to buy one off the shelf. So, you were just kind of uh gritting your teeth. The other big risk, and this one is still somewhat true today, but it's just much less because you can start, you could do prototyping early. The risk is, is the game any good? Like, what are we building? Is it interesting? Is it fun? And when you think about this problem of we can't even really run the game as it will be cuz we're just building this engine and we we don't even really have a way to test the game super well. And we can't really build much of a final level because we don't have level editing tools yet. Those are just coming online. trying to guess what you are actually going to be shipping in terms of gameplay is incredibly hard and there are games famous games um I want to say like Thief the Dark Project a very famous game looking glass game uh it was formative in the stealth genre launched a franchise which was Thief you know I want to say uh everything I heard from people on that team was that like the final core gameplay only sort of came together like right at the end right and so the game just could have

1:04:38 been a lot more could have been polished a lot more but it just the timing of these things coming together was so hard and so it really was an incredibly different thing and nobody really had a way um around it. Eventually, there was sort of this push towards something. Well, it was basically early vertical slice prototyping where as games started getting bigger and people were like, we can't keep doing this, especially if we're going to be putting millions of dollars on like this is not like an option, right? They started to move toward this thing about look what we're going to do is we're going to focus the entire studio on building one vertical gameplay slice as fast as we can, as hacky as we can. Whatever we have to do, do that. prove that that is engaging to play and then start building out everything else because we simply can't afford to not know what that thing is and then we can start building like spreadsheets that'll schedule what are the assets we need cuz now we actually believe in the thing we can see it running and it fills in all those details right and that I believe you know I'm not a game historian so take

1:05:40 what I'm saying with a huge grain of

Licensable engines democratized creation but flooded the market

1:05:41 salt that I believe was a pretty big paradigm shift for the industry when they started going okay we got to actually know and it became much less seat of the pants after that if you will.

1:05:51 Now I'm interested in your observation. I know you're not a game historian but you were in the industry and you still remain connected to it. What happened when game engines became widespread. they became not only licensable like you we're talking about like Unreal Engine for for larger studios but ones like Unity or or God do which now amateurs could also afford or I mean amateurs in the sense that you're a college kid or or you do some side project you can already afford the license and you can build stuff because now that risk is gone for clearly the studio so that risk is eliminated and it also I guess it now opened up so much more people who who can now have a shot at creating a game because you no longer have to either have this massive amount to license this super expensive game engine. You no longer have that risk. The only risk is is it fun. What have you observed happen in terms of both for the industry for for for development pace those kind of things. And the reason I'm asking because I I I wonder if there's going to be a parallel with AI where okay, you

1:06:53 know, like you needed you needed to have an engineer who was

1:06:56 right. I I I feel games might give us a bit of a a hint of what we might expect at the broader industry.

1:07:02 So that is actually I would say that's a brilliant analysis of the situation for for not having lived through games and for noticing that. Uh that's that's impressive. I'll say that first. Uh and I totally agree with that. I've I've said to people in the past who have asked about sort of AI impact on games in that sense and I've sort of said as much I've said like the licensable engine thing kind of was our AI transition already unfortunately and uh I regret to inform you that the news is not probably that positive. So there are some uh definite positive things that happen early on because as you say uh it opens up the ability to make games to people who could not have uh marshaled the technical sort of uh the sort of the technical staff necessary to produce a competitive engine. And so giving them the ability to make games is a pretty important thing. and it allows a bunch of people to make sort of some artistic expressions that made they just wouldn't

1:08:04 have been able to do early on. That tends to be a net positive because you just have some more games coming out. Maybe some of them aren't that good, but ex, you know, some of the existing games aren't that good. That's not that different, but then you get some really cool games coming from some sources that just simply wouldn't have been able to do it. Thumbs up. problem is it rapidly kind of accelerates into this kind of a nasty scenario where you just have massive numbers of releases. And I think at this point we're at the point where I want to say Steam games are in the like tens of thousands or hundred thousand per year or something like that. It's it's so massive that there is no way that your game will be organically noticed anymore pretty much period. So essentially it's this really nasty problem where you just have the market flooded with products and there you know it used to be that if you made a quality game if it was fun people would find it because there were

1:09:04 so few games that someone would play the fun one and tell people about it and it would get purchased. Right? Like word of mouth or just exposure on a storefront would be all you really needed to get, you know, sort of the word out about a game. You didn't need a huge marketing budget or anything like that. Fast forward to today where we have this sort of massive influx of games. Again, pre-AI, it's just because now the barrier to entry is very low.

1:09:30 Uh, and you really need a strategy to make sure your game gets found. Is it possible that sometimes, you know, a small indie game with no marketing plan or nothing will get discovered uh and become a huge hit? Absolutely. It does still happen once in a while. the chances that you will be that game are like zero. So, you kind of now need a marketing strategy, a real marketing strategy, uh, and going into the market for games without one and expecting to sell any significant number of copies, uh, above, you know, maybe a few thousand at best is really unwise.

1:10:11 If you want to hit reasonable numbers of sales of a game, you have to have an idea of how people will find out about this about this game.

1:10:20 So, if I'm getting this right, it sounds like the game itself being good as table stakes, but not enough on its own, right? That distribution, marketing, getting people to hear about the game is much more of the differentiator because there's just too many good games out there and now they're easier to create.

1:10:36 I think that's exactly right. And uh and that's just the unfortunate reality of it now. Was that a good trade? I don't

Modern games compete against live services and older titles

1:10:43 know. Um but that's what happened. And so that's where we are in the industry. Now, there's this other thing that I heard about which is how new games not only compete with other new games, but with old games as well, right? Like the other day, I spent a few hours playing Death Rally, which is a game from the '9s. And every year, there's more and more good games to play. They all take away from the time that the new games have.

1:11:05 Yes. And that uh problem will only get worse because one of the things that the game industry could rely on in the past that is much harder to rely on now is that older games would look dated technologically in ways that consumers cared about. And we have now kind of also crossed the threshold where there is a segment of the market where people really do care about the latest like ray trace lighting and all these sorts of things. and you know more photorealistic rendering or whatever it is. But a large portion of the gaming market by revenue doesn't really care what the game looked like all that much uh in a sense that whatever we're doing today is good enough. So 10 years from now if the games look much better for some reason no one will really think of that as a huge differentiation differentiator in terms of sales. You go back to 1995 and technological advances were a huge differentiator in terms of sales. You

1:12:06 come out with something that you know looks good that takes advantage of the hardware of that day and boy did it look cooler and feel more responsive and all these other things as compared to earlier titles, right? And so that's also going to increase the degree to which the thing that you're talking about will happen. I can go play an older game because it doesn't feel obviously dated in an audiovisisual way. I don't have to be an appreciator of retro gaming to go play something from 2017. It just looks fine probably, right? So, there's that. The other thing that I'll just mention, which we kind of already touched on, but that ties directly into your point, is that also live service is such a prominent thing now. People are just logging on and playing Fortnite for several hours or something. that's also taking away from the possible revenue that might be spent on buying some indie game or some new AAA game even. So you have these sort of incumbents, people playing Minecraft, spending their time playing Minecraft, spending their time playing um League of Legends or Dota and that's taking up a

1:13:06 huge amount of their time that's it's it's zero sum, right? They can only they can only spend their hours in certain places just like Netflix or anywhere else. They have to start thinking about, you know, they're competing with

AAA sequels like GTA 6 function as platform replacements

1:13:16 everyone else for entertainment hours. Okay, so I need to ask you this. GTA 6, how is that in 2026 at a time when we have better tools than we have ever before and we can build software and games faster than before? Like how do games take 10 plus years to develop? Is this some kind of outlier or has AAA game development taking many many years just not changed at all? What do you think is going on here? So from a player's perspective, I can understand why someone would look at it and go, "Wow, Grand Theft Auto 6 has been in development a long time. How does that make sense?" Or, you know, something like this. From a business perspective, you have to understand that Grand Theft Auto 6 is not a game that they are selling to players who are going to play the game. That's not what it is from a product standpoint, right? What Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5. Grand Theft Auto 5 at the time was, if I'm not mistaken, by far the most revenue generating

1:14:16 entertainment product in existence. The online part of that game was generating like billions of dollars. And like I said, not a game industry historian, so you know, take what I have to say with a with a huge grain of salt, but Grand Theft Auto 5 was kind of like Fortnite before Fortnite, if you will. They were a huge huge live service revenue generating product. So from Rockstar or Take 2's perspective, right, Grand Theft Auto 6 is not just let's try to get out the next Grand Theft Auto as soon as we can cuz we'll make money selling that title. It's a we are going to replace the most profitable thing we have ever built, which is still generating a ton of money for us, with a new thing. And you can better believe that they want to make sure that they are going to do that right because the last thing you want to do is ship a new product that cannibalizes something from your old product and then is less revenue

1:15:17 generating. Right? So I'm sure that their planning around Grand Theft Auto 6 is not just about trying to produce a Grand Theft Auto that their fans will love and will buy as the original singleplayer gaming experience that it was. I'm sure they care very deeply about that. Just from a reputational and from an artistic standpoint, I'm sure there's a lot of people on those teams who care about that. But from a business standpoint, I am sure there's also been a tremendous amount of thought and work put into what does the live part look like? And that's a huge undertaking, you know, that I'm sure that they've been planning for quite some time as well.

1:15:54 So, it's a massive massive thing that they're doing here. How well will it succeed? I have no idea. But it is not just a new Grand Theft Auto is I guess the way that I would look at it. Grand Theft Auto 5, I think, was somewhat of a surprise to them. I don't think they knew it was going to generate that kind of online revenue. I mean, maybe they had hopes, but I don't think they knew that it would be that kind of a massive money maker that it was. And so, this is the first product really where they know they will have the audience. For Red Dead Redemption, they kind of tried to do Red Dead Redemption 2. They did a similar thing where they tried to have the online thing. It didn't I don't think it hit nearly as big as Grand Theft Auto. Grand Theft Auto 6 is the first time they're shipping a true update to what is their flagship. And so, you know, it's equivalent to like a

Clean code rules block compiler optimizations and degrade performance

1:16:41 relaunch of Google search or something like that is what they are doing here. Uh and so I I you know, I'm sure if I was in charge of that project, I would be sweating bullets. So, I'm sure that they are putting a lot of thought into it. Uh, and it's a very massive undertaking, I'm sure.

1:16:56 I'd like to switch gears to SoftwareCraft. You made this video titled Clean Code Horrible Performance, an essay/v video showing how Uncle Bob Martin's polymorphism based refactoring pattern runs about 1.5 to 15 times slower than a plain table switch version. Can we talk about the responses to this piece?

1:17:17 Well, I guess I can put that in context. So, that is sort of from that course on the Substack. So, it kind of goes with a bunch of other videos that are part of like the Substack thing. I guess the first thing I'd say is I feel like the response to it was very positive. I was kind of surprised. There are plenty of people who didn't like it. Don't get me wrong. It's controversial to be sure, but I was surprised at just how many people were enthusiastic about it as well. But what I would say is it's really I I don't really think it's should be so controversial because there's one thing where people want to just use the term clean code to mean code that they like or think think is written properly and that's not something you can argue against, right?

1:18:00 Because that's just, you know, I probably have a version of what I think is clean code and obviously I don't think that's bad, right? Like it's it's my idea of what good code looks like. So if your idea of clean code is just whatever you want it, you know, whatever you happen to think are good programming practices, I might agree with those programming practice, I don't know. So in this particular video, I was talking specifically about the things that were advocated that are like very specific things that are said like don't have functions over a particular length or these sorts of things, right? Uh things should not know the type at runtime or whatever, right? There's all these like kind of rules about it. Preferring polymorphism always, right? If you look at those things, they're kind of just bad programming practices. I I don't really know how else to say them. They don't mesh well when you put them together. In isolation, some of them might be fine. So, for example, if you really prefer lots of small functions, that's actually fine if the compiler can see all those functions and know that it can safely inline them and collapse them as necessary. This is a this is a part a

1:19:01 lot of people missed about the video I guess because it's a pretty short video so I didn't explain anything in detail but a lot of redundant code happens when you have lots of tiny you know little these little tiny functions and if they're all virtual functions in C++ let's say the compiler can't know for sure which ones of them are being called and so on even if you put things like final in them there's all people have a lot of weird beliefs about how the code works you can just go do this testing when you have lots of these little functions if they're all like statically defined and aren't virtual calls if they're just known calls like or just member functions. When I say static, I kind of mean just known to the to the translation unit, not external. The compiler can put those together, collapse all of the redundant code and actually produce something reasonable that will run pretty fast out of that.

1:19:45 It can also do things like widen the code path if it needs to vectorize to

8 Highlight · Why clean code polymorphism destroys compiler optimization

1:19:50 like run in SIMD and stuff. The compiler has all these options to take what is fundamentally not particularly great code in terms of how you would want it to run at runtime, but it might be able to turn it into that because you know compiler optimizing uh optimizing compilers are pretty heroic these days and the sorts of things they can do. If instead you use all of these, you know, things that were recommended, you completely block out the compiler from a being able to do those things. Because if it can't tell what it's doing at runtime, if it has to leave open the possibility that you substituted in a different class here or something like that, then you end up in a situation where the compiler can't do any of that work. And people mistakenly think that this is just because like virtual function calls cost too much or something like that. That's not what it is. It's not the cost of the virtual function call. We could talk about that as a separate thing. Um because you you can analyze that cost as well. It doesn't have much to do with specifically whether it's virtual or not. has to do with a lot of things like branch prediction and how much stuff is getting pushed on the stack and whatever else, right? But it's the cost of the compiler not being able to do any

1:20:50 optimizations. That's the actual cost. And that cost can be severe. I showed only I think a pretty mild degra degradation compared to what you would actually see in production if you really had a huge number of things doing this. And I think it landed pretty well. It,

1:21:06 you know, it's a very widely viewed video and a lot of people seem to really like it. I thought it would be, you know, probably even more controversial than it was. So, I was pleased with that. Um, but yeah, I mean, all that stuff remains true today, I guess, is what I'd say. And I think it's good for people to hear because they need to hear opposing viewpoints. I think you can write code that is maintainable and easy to read that doesn't follow those principles in that way and that doesn't have those problems. I don't think you have to do those things. So I think it's worth exploring other options that are still maintainable, that are good code, but that allow the compiler to do the right thing.

1:21:44 What is your take on test-driven development? You know, when you write the test first, then you write the business logic. you you you've talked a little bit about about this as well because it's it's a practice that used to be super popular in the you know like especially when you're building services some of those things especially in the 2000s kind of got a little bit out out

Testing strategies should be pragmatic rather than dogmatic

9 Highlight · A pragmatic perspective on test-driven development

1:22:03 of fashion and now it's unclear if it'll come back or not with uh with agents or not.

1:22:07 I don't have that much of a spicy take on that one. My take is very pragmatic which is that if you can identify tests that will save time in total that's usually what I try to emphasize. In other words, if the amount of time it takes to create and maintain the tests will actually save us total development time because they will identify bugs that would be hard for us to find uh in production or in or would be very costly to get to if they uh got out then great.

1:22:36 And I've used them before like I talked about working at RAD game tools. I had a regression tester that I ran on like the core libraries there that I had written for the you know they're not really called libraries but the core like routines to make sure that you know anything that I could be testing for our customers I sort of was and so I think there's good times for testing. I would say the part that I don't like about test-driven development is the testdriven part. I don't think development should ever be driven by tests. I think tests are a thing that you should be aware of.

1:23:09 You should know what your options are for testing and you should make intelligent engineering decisions about tests. Now could that decision be that for this particular project we are going to drive it primarily from the tests? Yes, that could be a decision that you make. But you shouldn't really think of development as something that is primarily test-driven like by default cuz that might be a very bad decision for some other project where it just ends up costing you more to have done it that way. So like with most things I would advocate for a pragmatic approach to testing. You should understand the cost of testing, the cost of developing, maintaining the test, and the cost to your codebase if it makes it harder to change your codebase because tests have

1:23:55 to be rewritten and you therefore don't make changes you should make. All of that stuff should be in your brain and you should make an intelligent decision about what your testing strategy is. If that decision intelligently made turns out to be we are going to have a lot of testing on this project, that may well be a good decision. I don't think

Good code aligns with machine execution

1:24:14 there's an absolute thing you can say about how many tests there should be. Some projects probably shouldn't have very much. Maybe some projects should have a lot. And I think knowing which of those you are doing is part of being a good software engineer is I guess what I would say.

1:24:29 You mentioned being a good software engineer. But before we get into what is a good software engineer, what does good code mean to you specifically? So good code to me usually means that you have written a something that is as straightforward to what the machine actually needs to do to solve the problem as it can be and also hopefully that you have I guess I'll say properly identified ways of breaking it into easily digestible pieces and named those pieces in ways that are easy for someone to understand especially yourself because you are very likely to be someone who's going to have to modify it. So that's the way I tend to code. I try to identify what do I actually need the computer to do. I try to write as simple as possible the thing that will do that and then I try to put that in terms that are you know I would say least redundant. So you know I don't want to see the you know the equation

1:25:30 for uklidian distance scattered throughout my code. I want to have a function that's like compute that distance and I want to use it right. I want I want it to then be nicely broken into the pieces that it represents and I want those pieces to be reassembbleable properly by the compiler in a way that will produce code that runs very efficiently. Right? And so that's what I'm usually trying to do when I'm trying to program. And for me, I have never really understood the sort of mentality of there's a difference between code that is like well architected by some principles and code that runs quickly because in my experience usually the code that is architected properly is also the code that runs quickly. And yes, there is a point where if we decide that something absolutely has to get as close to theoretical maximum as it possibly can, yes, we will start to make that code harder to read and modify because we are now like really over

1:26:30 specializing it for this piece of hardware or whatever. That's true. But that point is like, you know, way out on the curve. It's not the common case. Most of the time, assuming you just want code that runs pretty pretty darn well on most hardware, the simple readable version of the code is actually very fast. It's only once you think you need to have 27 factories and 8,000 microservices and all these things running that it starts to be this thing that's like good architecture, but also like hard to modify, hard to read, run slowly, right? All these things. So I tend to think of like good code there's like this nice nexus of runs pretty pretty darn well easy to read easy to maintain isn't as close to theoretical maximum as it could be but it's close enough and the the paths towards theoretical maximum have not been

Software engineering resembles specialized sports roles

1:27:22 foreclosed. We left the door open with the way that we wrote it so that if someone really needs to come along and boost its performance it's set up to do that. Right. And related to this, what is a good software engineer to you? Is it just someone who writes good code or it goes beyond that?

1:27:38 I would say it really depends on the the environment a little bit because I think I've seen a lot of different kinds of good software engineers. And so I would liken it more to a uh you know, if you want a sports analogy, you'd imagine something more like a baseball team where it's like what's a good baseball player? Well, it's like are we talking about a pitcher or a designated hitter, right? Uh and it changes quite dramatically. So, there might be some things like, hey, if someone's pleasant to work with uh and and you know, doesn't you know, goof off all the time and actually gets their work done. Those are obviously things that we would say are true of any software engineer, you know, there are some general personality traits that might be positive. But when you're talking about things that are more specific to just software engineering and not just being a good employee or something like that, I would say I've seen a couple different kinds.

1:28:30 I've seen people who are like the utility infielder. There are people who just like they can identify and go and try to fix a problem and succeed. Even if the codebase is kind of wacky and out there, they're good at getting the lay of the land very quickly of identifying something that's going on. and they're not afraid to go in and like, okay, this is kind of this code base is kind of ugly here. It's okay. I'm going to patch around. I'm going to do what I need to do and get things done. That's a great engineer to have around. I've also seen great engineers who are the exact opposite of that. They are just like, I take this one particular problem that we have and eight months later, I have ground out every last thing there is to know about this. and some sometimes to the point of like producing new algorithms that no one's even known before, right? That are like these, you know, breakthrough things, right? And that's a great software engineer to have on a project if you happen if you're going to be having that kind of thing.

1:29:25 And so I've seen a lot of different people that I would consider great software engineers and they aren't all the same person, right? So I think that it's kind of important if you're asking it from the standpoint of like, hey, you need to put together a team to go build this project. What's a great software engineer? I would say the best advice you could give someone in that position is think about the roles. Think about what kinds [snorts] of roles there are going to be here and don't think great software engineer. Think great that role, right? Who is going to be a great pitcher? Who's going to be a great first baseman? Who's going to be a great outfielder? Who's going to be a great this that the other thing? Great third base coach, whatever it is, right? And that's what you're trying to put together if you're trying to build a team to me.

1:30:07 Yeah. So, like it's just not one sizefits-all. But I I still want to push you a little bit like what are things that are you think are non-negotiable for someone to be a great software engineer? I you know we we talked about the things that we talked about which is a recurring theme with you is just going deeper and deeper and understanding the next and next layer you know like understand if if you're doing web development understand react once you understand react understand what's going on in the DOM go all the way to assembly once you've done there understand how the CPU is doing operations and branch predictions and some of those things like to me that's a skill of like curiosity driving deeper crafts whatever you call you know, there are different ways we could do it. But along these lines, what are those traits that you think no matter, you know, what kind of role we're talking about, but if you think back of of some of the different types of roles that you work with, like do you see some overlap that that they all had something? I would say that it's pretty unusual, I guess, that I can't think of someone I would think of as a

1:31:08 great software engineer who like didn't know how to like read assembly or something. That is true. It might be that having that curiosity about how things work and and knowing at some level what's going on is kind of maybe something that's going to be very common to a great software engineer. But I would just underscore the point. The degree to which they are employing that knowledge may vary quite a bit. For some of them that may be their bread and butter and they're doing that all day.

1:31:34 For others it's just really a thing where because they know how a computer works, they're not making those stupid architectural decisions that come back to bite us later. Right? And that's great, but they may not really be doing all that much actually at that kind of level or or thinking about at that level. They're just going like, "Yeah, I know we got to kind of push, okay, this stuff's going to have to be done in batch because I just kind of know that that's, you know, how the machine's going to have to handle it. So, we'll, you know, I'll make sure I I write the code that way or whatever." Yes. But, so there's a little bit of that. The other thing that I would uh say maybe is

10 Highlight · Skepticism toward unverified programming dogma

1:32:07 actually like not being dogmatic about things that they haven't actually themselves proved out is probably a huge one. I find there's a lot of like received programming wisdom that's just nonsense. Like clearly no one's ever tested it and if they did they would have found out that it's that there's no actual basis for it. Doesn't necessarily mean it's false. It's just there's no like there's no actual tangible way you can demonstrate. And sometimes it is like you could demonstrate that there are actual concrete downsides to this received wisdom, right? And so in order for it to be received wisdom, you should have to be able to at least demonstrate concrete upsides, which often times cannot be done. So I would say people who actually focus on what works in practice is a huge plus and you could apply that anywhere into anything, right? Not just saying, "Oh, the flavor of the month is that we're writing everything with classes and virtual functions and hierarchies or whatever."

1:33:00 It's like, did you actually determine that that results in less code or that the code actually is mermaid? Like, did we do any testing to figure out if this is helping us rather than hurting us? And the answer oftentimes is no or if if it was at all. It was extremely shoddily done and you would not take those uh results as conclusive in any way. And so it's like being more skeptical about coding practices and actually trying to focus on what is working in practice and what we can demonstrate and measure in some kind of a uh repeatable way is I think a really great thing for a software engineer to have as well. So people who don't tend to fall prey to that just like I watch some presentation and someone at Google says always call me MEMS set or never use if statements or whatever it is like if that's the level that you're thinking at then I

Preserving the traditional craft without AI

1:33:43 probably am not going to put you in that category of of really good software engineer because that's not how it works.

1:33:48 Plus it's not that hard to try these things out or in or set up or or do run an experiment. Now, the the final topic I wanted to touch on, which I deliberately didn't get into until now, is AI and how it's changing your work. And I'd like to start with that, like in in the work that you're doing at Molly Rocket with this this project that is un yet unreleased, how are you using AI tools, if you're using them at all?

1:34:16 We are not using them at all.

1:34:18 So, you're you're you're doing it just like before. You're writing your your code. What made you decide to to take this path?

1:34:25 Well, we're a little bit different obviously uh in the sense for two reasons. One um is that we you like I said we are kind of we have like sort of two projects here and the substack is our primary focus and this other one is a thing that we're doing because we want to do it. And when you think about that perspective it's like well why did you want to do it? Well, the reason that I want to program like things in a game is because I want to program them.

1:34:53 If I just wanted an AI to program them, I probably, you know, first of all, we'd probably just go use a licensed engine, right? Like I wouldn't I wouldn't even bother asking AI to do it. I'd just go get the Unreal Engine,

1:35:04 right? Or something like that and so on. So, uh I think a little bit of that decision is probably not that relevant to your audience because it's more about what do you want to do? Like what why are you spending this time, right? It's a philosophical question, not a productivity question. So, it's not like I evaluated it and said, I don't think this will save us time or I or you know, or I have questions about the copyrightability of it or the ethics of it or all the sorts of things that you could rightfully evaluate AI on. It wasn't necessarily that. It's more just like this does not further the goals of the project to use it. So, it kind of was a non-issue at that point. Right.

1:35:39 stepping out a little bit more uh to a broader philosophical framework about AI. I guess what I would also say is I think that if you regardless of what you think will happen with AI in the future because obviously we don't really have any way to predict what it will look like 10 years from now. It's anyone's guess really. I think there will probably also be at some point point a notion of like traditional handcrafting that will come into play because we've seen this in most other times when you automate something. So, if you automate making furniture and you have like IKEA or whatever, that doesn't mean that there isn't some weird guy down in the industrial district of your city making crazy wood tables with iron and welding and something. And that that's just a thing that people are still doing and some people want that table. I don't necessarily have an explanation for it and I'm not trying to argue that it has more or less value, but it's just something that happens, right? And so if

1:36:41 I imagine what I want, what I love about computers and what I want to do with computers, and you asked me, move that into some other context, which of these people would you be? My answer is always, I'd be the organic farming guy. I'd be the guy who's making the weird table in the industrial district. I have no interest in managing a division at IKEA. I literally couldn't care less about that, right? And so I think for me another reason why I'm not that interested in pursuing AI is because I would like to be part of whatever the set of people are who are going to keep this traditional craft alive just cuz that's something humans do. Not because we're trying to say that that's the right business case, right? If that makes sense.

1:37:28 Yeah. And and at this point there's a bit of a tradition if you will even if we assume that these machines will do as good or better than cumizit for like what 60 plus years we've we've only exclusively handwritten software because that's how it got done right like a a lot of us anyone who started coding before 2023 or the end of 2022 or probably honestly 2024 when these things have gotten like decently good you just wrote it by hand a lot of it or tap complete still counts.

1:37:56 Yeah. I and and I guess I would say like again it's just you know why why do that right is the question it's like I don't know why humans do that humans do that because it's something humans do right humans like to do things themselves sometimes you know people can buy a hat they can buy a a wool hat trivially or they can buy whatever and then someone's out there knitting a hat right now that's just it's just something humans do they like to make things by hand sometimes and at varying levels of handmadeness You know, there's some people just buy the the wool or whatever or buy the pre-made yard. Some people raise the the sheep or whatever and shear it, right? Like you can go arbitrarily far down. You could find somebody who's going all the way, right?

1:38:39 Even further than probably I would ever go if I was in that thing. So, you know, you could imagine someone making their own hardware these days, right? Uh I'm not doing that. And so that's kind of my my take on it. Um, so I'm kind of the last one to ask about, you know, AI coding or what you might want to do with it. I I really have no nothing of value to add.

1:38:59 Yeah. But I am interested in asking you through the lens of the games industry and and we touched on on games engines arriving and and now so many more people can make games. Not everyone, but it's it's a lot easier to enter. What are you observing in terms of most people outside of who are still handcrafting code because they want to are are using these AI coding agents for for two reasons. Either it's either it's it just makes sense and they realize well this thing can now generate code as good as I did which was a turning point in January. I I I had that turning point actually myself or some are actually just pushed with corporate mandates of like you need to use these tools and and eventually they kind of get on board whether willingly or or unwillingly. But so many folks are are having AI write the code for them. They're, you know, they're prompting it, but they're doing it. What do you observe of the effect having, you know, from your vantage point? May that be on on quality, craftsmanship, on just output, speed, etc. What are you

1:40:01 seeing? I think it's a little too early to assess to be honest because kind of as you pointed out obviously there's been people who maybe uh you know we might derogatorily call AI shills who have been saying that it was producing as good a code as humans for you know two years now or something like that right

1:40:22 but in reality the people who opinion I would trust more none of them thought it was really all that usable until more much more recently Right. And so we really haven't they haven't had very many months to actually be figuring out how to use this thing or to determine to what extent they can use it and how what it's best at, what the workflow looks like that makes it produce the best results. It seems like at the moment I would say probably need to give it at least another six months if not another year or something to let everyone kind of shake out like what what are actually the best ways to use this thing. I know tons of people in the game industry are using it. So I know that they are um doing various things with it. Whether those things are the same sorts of things they will eventually think are the are the way they like you know like the the things that they're doing right now may be like oh that was kind of dumb like you shouldn't have used it that way you should do this other thing with it and it's way more productive or

1:41:22 something. So I feel like it's probably too early to assess. Uh we haven't seen any real like obvious like oh wow like you know the the you know Fortnite ships once a week now and it's bug free like nothing particularly interesting has happened in terms of output there but again it's been what like 5 months or something. So it's just it's way it's way too too early to see how it actually gets integrated into a reliable process.

1:41:49 Right.

1:41:49 Yeah. and and there I I know there are some companies who are now tying up let's say agents fixing bugs but that's only a few months old the oldest software that's widespread that is written close to 100% by agents is from the labs open AI's codeex and entropics cloth code but even there it's been since November or or some parts of it December so like maybe six months and it's a it's different right that is a product they're selling so there's uh I'm not sure we'll we'll we'll know for sure like is it truly 100% how much you know there there's a

1:42:24 marketing angle or not but there's a there's a self bias there so like I would put those aside in terms of trustworthiness and you're right that the rest we just we just don't really have the information it'll be I'm sure there's so much experimentation so but to to your point it takes time to bake right to see the the impact

1:42:42 most of these things are currently presented as tools meaning a human has to operate them at least in some way like at least setting it up to do what it's going to do and therefore uh you have to give it some time you know you know nobody currently is selling a product where it's just like oh just turn this thing on and it will just ship Fortnite by itself forever and you can just get rid of all your engineers like no one's actually selling that product yet right we could evaluate that product because we'd be like did anyone do it did it start shipping Fortnite on its own right so if it's still something where humans have to kind of figure out how they want to like slotted into what they're doing, then it's entirely possible that the reason that we haven't seen some big uptick in productivity uh that would be obvious to an external observer is because it's going to take a while for people to like shake that out or maybe the AIs need to get a little bit better. Maybe like we've got to go through some more update steps or you know whatever. I'm not sure. So there's all that's on the table. Then there's another possibility

1:43:42 which is that it actually already has worked but just the productivity boost isn't as big as would be obvious if people got 10% more productive. That would still be pretty impressive because it's hard to get a 10% across the board uplift. I've said this before on podcasts. I'm like if you have a tool that can give everyone 10% off of that's great. Almost no one would know, right? It's like you can't it's not really externally observable that clearly if that's what you got, but it may have happened, right? So, it's really hard for all of those reasons. At some point, if the AIS are really fantastic and people figure out how to use them really

Autonomy shapes developer burnout from AI

1:44:19 well, it should be obvious. It should be like five people are now shipping Fortnite instead of 5,000 or whatever, right? But, but until that point, it's really hard to know because it's just like especially if it was small, it'd be hard for us to see.

1:44:32 Well, this is anecdotal, but I'm getting a lot of data points and messages from software engineers and managers. One impact it's having is there's this kind of like AI fatigue/burnout for from software developers who are like look I am good at coding. I've always been good at it. I I enjoyed the the work to various extents. But since this AI thing happened since the end of the year, beginning of the year since it's actually I'm now prompting and now all my code is generated whether that's corporate mandates or it's just faster.

1:45:02 I'm starting to lose my drive like why am I here? like anyone could do this and I think there's a sense of like I'm using a lot less of what I'm capable of. There's all this pressure from above to be more productive with it and it's I think we should like I I'm seeing more and more signs that it's it's what do we call it burnout, AI fatigue, etc. loss of motivation. I haven't seen a technology or I don't remember technology having this widespread impact like everywhere. I'm hearing from folks at some of the leading like kind of not AI companies per se but like you know big enough like database providers who are now hugely into AI and they're powering a lot of the things traditional companies modern everywhere. Have you observed some of this thing and would you have any any advice or any uh pointers to folks who are feeling like this right now? I guess I would say observed. No. Uh heard about. Yes. Uh I

1:46:02 guess is what I would say. Like I have talked to people who have been like such and such has been having a really hard time with this or such and such has been having like there's there I've definitely heard that interacted directly with someone. Not currently. No. Uh, and part of that is is probably largely because most of the people I talk to have a fair amount of latitude with what they do and how they do it. A lot of the people that I talk to on a daily basis are able to make their own decisions about what they want to do with AI and so on. And so I don't necessarily hear from as many people who are going to be in a position where some manager told them this is just what you have to do. This is very interesting because one thing that keeps coming back and Armen Ronacher was telling me the same thing on the podcast is he's he's observed that autonomy like at your work how how autonomous you are at your work like how many decisions you can make on how what you work on how you do your work the people who have a lot of that are typically like oh great I can use

1:47:03 this for this like I can use this tool but the people who are told you know like in beforehand you're given a ticket or the PM tells you this they don't have much wiggle room and Now those folks are seeing it way more as a threat because of course subconsciously or consciously they're thinking well this thing could automate my my job. It's now or or it made it made from that little effort I had to do that it took it away as well. So I wonder if there's a connection here. I mean that sounds totally logical, right? Uh if you're somebody with a high degree of autonomy then when are you going to reach for an AI? Well, whenever there's something that you didn't want to do, right? So kind of by definition, I think at that point you're going to have a much more positive experience with it because worst case just doesn't work. In which case, I guess that's not great. You're going to be like, "Ah, this thing was kind of crappy." But assuming that it's able to accelerate some part of that, that was great. It's like, "Hey, I didn't want to do this thing already. I had this AI do it for me and now I have the thing."

1:47:57 That's just a positive experience for them, right? Whereas, yeah, if you're just told like you had this thing that you wanted to do, and you were told you can't just do it yourself. You have to do it with the AI. Uh, you know, and by the way, we just had layoffs or whatever. You know, a lot of that stuff obviously could totally change your mental reaction thing because now it's not you deciding to use an AI because there's something you didn't want to do that you thought the AI could do for you. Now it's you just being told that you're supposed to be using this AI to automate whatever your job used to be.

1:48:25 You can see pretty obviously why that would have different psychological effects on people, right? So, so I guess it it might be just an idea for folks in this situation that now it you might want to evaluate your current position or if you're interviewing your next position based on how much autonomy will you have because the more autonomy you'll have, the more likely you're going to have control over how you're using this stuff, how much you can experiment versus being given a mandate that I don't know, we're expecting you to have this output increase or output change, whatever that is. I I I wonder if this will re-evaluate some of you know like what is considered an attractive position because like for example big tech was considered a great place to work because high compensation pretty clear expectations like easy to understand career advancement but now they're the ones who are starting to measure uh your AI usage which is going to like giving a kind of a bit of a handcuff of what we're expecting you to do or there's where you might have there might be mass layoffs which again you have no control over right like again one more or or inside of metaphors

1:49:27 reassignments of like you will now do labeling for x months.

1:49:31 I mean you could sort of think of you know could we coin the phrase are you using an AI to do your job or is an AI using you to do to do your job right like because at some point it definitely it definitely felt like meta for example uh from from your reports on it and I have seen the same thing said by other people so it does not sound like a a one source kind of a thing. It sounds like this was kind of just accepted as fact that they kind of just were using you as AI training, right? Like that's what you were kind of, you know, you're just there to train the eye to do it so that we don't need you anymore, right? And so thinking about that from a from a perspective of choosing your job. Uh it does make some sense if you do if you have any latitude, right? Uh but yeah,

Reading technical papers builds deep expertise

1:50:14 as closing, what are one or two books that you would recommend that had an impact on you? I'm gonna uh have a hot take here if I if I might because it's sort of a it's sort of a a push I've been on recently. I don't think people should uh necessarily take a book recommendation

11 Highlight · Reading academic papers for deep knowledge

1:50:33 from me. I want to recommend that people read a paper. I'm trying to get more people to just to just read papers because I realized I read a ton of papers. Like I am constantly reading papers on things that I am interested in. Like if I'm going to go do some programming in a in an area that I haven't done before, I will read a ton of papers. I'll crawl the references on papers. I'll read a survey and go gather all those references and read those references and crawl them back. And I find that I learn a ton that way. And I feel like a lot of programmers just don't do that. And so my recommendation would you don't have to read a specific paper. I'm not going to give you a specific paper. Read this one. Just think about the domain you're programming in. Do a search on Google Scholar for some part of that that you're interested in. Try reading a paper, following the references, see what you think. I think it's a great thing to do. And I get a tremendous amount of not just enjoyment from the education of it, but also just like more knowledge about what I'm doing. pretty

1:51:34 much every time I do this, even if it's just to learn a little bit more about the historical record of how things got discovered, but a lot of times it's just like I learn about whole new techniques I just was not aware of. Uh because there's way too much out there for any

1:51:47 one person to know. Uh and and I don't know to what I again since I don't currently use AI in my workflow, I couldn't say. But my assumption would be that AIS would also be very good at helping you find some papers to read if you were interested as well because that's you know chewing through a lot of uh the technical record is something that they do. Uh and so maybe you could even ask your favorite AI uh to to suggest a paper that you might like based on some things that you tell it. I don't know if they're good at that, but I'm guessing that's something they could do.

1:52:18 Casey, thanks a bunch for this this conversation. This was great.

1:52:21 Thanks so much for having me. It's been a pleasure. I've been wanting to talk about performance with Casey for such a long time, and I'm glad that we finally made it happen. I kind of wish the industry had more people as excited and interested in high performant code as Casey is. If you made it to the end of this episode, you might just be one of them. I appreciate that Casey did not beat around the bush. If you care about performance, you want to be able to read assembly and spend some time reading it. Reading assembly is several times easier than writing it. If you can read assembly, you can see what's happening at the machine level.

1:52:50 And it's a lot easier to understand, for example, why a programming language like Python is much slower than something like Rust or C when you see the assembly code for simple operations. I was chuckling when Casey talked about these blog posts about how we rewrote our services in a new language and got 10x performance improvement and how those rewrites are usually not about the new language with fixing the architecture that caused the performance issues to start with. And although we did not talk much about AI, I found it amusing for Casey to say that the games industry had its AI moment years ago when game engines became accessible to pretty much anyone wanting to build a game.

1:53:22 Before large teams were needed to build both a game engine and a game, and now teams of one or two can create full-blown games. After a brief spike of positive effects with lots of new good games released, games have flooded the market in such great number that it's now impossible for a new game to become a hit organically. So marketing and distribution becomes mandatory even for great games. For more deep dives related to game development and performance software, check out the link that Pragmatic Engineer deep dives on these topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. And a big thank you if you also leave a rating on the show. Appreciate it and see you in the next one.