Distributed databases with Peter Mattis

24,120 words · about 96 min to read

0:00 met Larry and Sergey from Google. Somehow one of them looked me up, asked me to come interview at Google

0:04 when Google was three years old.

0:05 Three years old. And uh I said, "No,

0:07 you did not."

0:07 I've always been a very prolific coder. I look back at my GitHub output. It's like peak years, maybe 100,000 lines of code in a year. PreI, right? This is when you're back doing this manually. I kind of look at that, it's like there's kind of a max that you can hold in your head at a time. We did your coding in classes. That was kind of a big breakthrough. You have nine chunks of data, but any five of those chunks can be used to reconstruct it. And what this means is you can lose any four copies and you can still reconstruct your data.

0:30 What do you think good software engineering looks today compared to four years ago before we had a

0:36 I think the ambition has to increase. He built a image editor while at college designed the original storage system behind Gmail and built many more large complex and widely used systems. This is Peter Mattis, co-founder and CTO of Cockroach Labs. Before we sat down to talk, Peter told me, "For the last 30 years, I've always been a prolific coder, but my current output is a bit insane. And this isn't vioded junk, but database worthy, highquality, high performance code thanks to working with strong coding models. Today, we cover why B trees are so important when building databases, and why Peter kept reaching for the data structure over and over throughout his career. How he wrote 100,000 lines of production code per year pre-ai, stop coding between 2022 and 2024, and why he is back now. why he thinks AI agents are lazy about testing and how this is an easier fix than it looks and many more. If you're interested in distributed databases, distributed storage systems, or knowing how Peter manages to use AI to produce unusually highquality and productionready code, this episode is

1:37 for you. This episode is presented by Turbopuffer, a ridiculously scalable, fast, and cheap hybrid search engine built on top of object storage by an engineering team I've really grown to like after spending time with them. The Turbo engineering team is doing something really, really cool. They're completely redesigned their storage architecture from first principles to make storage faster, cheaper, and more reliable at scale. If you follow Turbuffer, you know that their storage architecture was a massive part of their early success. Redesigning a winning architecture is a big deal. It's one thing for your query plans to pass unit tests. It's another to bring each query plan to performance par while maintaining correctness and reliability in production. Here's a cool part. Turbopuffer is documenting the whole thing. Their new storage architecture, which they're calling Tuff V3, demands some hardcore systems engineering, and they're building it in public, sharing design decisions and benchmark results as they ship. They're keeping a worklog of their journey, and

Building the GIMP and ignoring competitor pre-announcements

2:31 the first post just dropped today. Follow along at turbopuffer.com/v3. That is turbuffer.com/v3. Peter, welcome to the podcast.

2:41 Oh, I'm happy to be here. This is awesome.

2:43 I I wanted to get into How did you get into tech originally? When did you figure out computers are interesting?

2:49 I figured that out in kind of elementary school, high school. Gaming was a little bit of a gateway drug for me as a software engineer as many other people. Uh I remember like early on my mom did programming at IBM at some point. I'm never even quite sure what she did, but we had computers always around our house like Apple 2 Plus, Apple 2GS. I'm from that era. um on up and you know go to the li uh the bookstore I'd find a book on basic or magazine type in programs no idea what it was doing but you know just kind of like I was addicted like you could produce put stuff into these computers and you get interesting stuff out and then I got to college and I was like I didn't think there was any money in computers I didn't know anything about it I started as a mechanical engineer following the footsteps of my dad

3:27 you started mechanical engineering as as your specialization

3:30 as my major yeah and I got in there and I was like doing like I'd done some computer stuff before it was a real foolish move to do this But first semester of doing these homework assignments in mechanical engineering and they were god awful like six pages for a single problem and I happened to take a CS course at the same time and it was so easy and then everybody almost was failing it. I'm like I'm in the wrong I'm in the wrong field. Let me switch.

3:52 And and then you switched.

3:53 Then I switched. Yeah.

3:54 What was the first software that you built either at college it must have been at college that you were like all right this is a piece of software that I'm kind of proud of. That's a complete piece of software. Well, I mean the big thing that I did along with uh my roommate in college, we had this course. Was it a compiler's course? I'm not even quite sure anymore. This is like 30 years ago. And we were kind of bored with it. So, we wanted to do something like kind of fun on the side. And I'd uh done journalism in high school, my sen junior senior year, and I knew stuff about kind of computer graphics and wanted to do something like Adobe Photoshop. So, started just kicking the tires and we built up this program that a lot of people know of called the And along with the I did a lot of the the graphics library, GTK. This has since evolved just massively since then.

4:36 It's kind of interesting cuz after college, I kind of stepped away from it. Didn't really stay involved much past my first year out of college, but definitely people still know me. And it uh it led to other some interesting uh events in my career. And and it it was like starting it was literally just you saying, "All right, I want to do something like Photoshop. How hard could it be?" And then you just this wasn't through what you learned in college, right? This was like you figuring out how to build, you know, like a graphical I I guess engine, rendering, drawing, data structures, all of that stuff, right?

5:06 All that stuff. Figured it all out. I remember trying to look at some papers back then. My roommate was looking at papers. We're just figuring it all out. And it's one of these things you almost kind of need to be a little bit naive to start anything like that because if you know how hard it's going to be, you would never start it. So, I think there's this like you have to have this level of like if you're really truly wise about the effort, you like never

1 Highlight · Ignoring competitor pre-announcements when building software

5:24 get into it. But then when as you get going, it just keeps on snowballing. It gets far further and further along and then at some point you're like, "Wow, this is really awesome." But there's an interesting little tidbit associated with our work on the which is kind of a little bit known. Um I think we've talked about this before, but it's just kind of fascinating where we were uh getting to the point where we're like, we should release this to the public. And back then there was Usenet news groups where people would post stuff and there's one on graphics. And I remember like just a couple weeks before we were going to uh release our first version of the someone else came on there were like, I've been working on this graphics program and it did everything the did, every last thing and then some. And we're just like, well, that sucks. I guess we'll just keep on working on it's been fun. And then we release the Never heard from this other guy again. And I just took a little lesson with that. There's always going to be someone else working on your idea. You can't get dissuaded if they pre-announce it. Nothing ever comes of it. And a lot of the marketing behind some of that stuff that you might hear is like I mean I don't know if we like we stole his thunder. I'm not even sure what happened. He never traced down what

6:25 happened. But there's a real lesson there. So there there there's a possible future where you read this announcement of someone saying I'm going to build all of this thing and you go like uh you know someone did it and you kind of go back and just do something else and the game never happens.

6:38 That's right.

6:39 Wow.

6:39 Yeah. I I guess especially with today with startups, you know, this sounds like just do your thing. Put it out there at the very least, right?

6:46 And the advice I give people is like you might have a unique idea, but most likely there's like a dozen people out in the world who've had the same idea. And there might be a couple of them working on it, but a lot of people don't even work on it. They're just like whatever they can't get going on their idea. So, you know, I wouldn't be

7:00 concerned at all if like you hear someone else working on your same idea. That's probably the case. You know, we're working on some cool stuff now at my current company. I guarantee you there's other competitors out there working on the same thing. And just know it's like it's a competition. You got to enjoy that aspect of it, not be afraid of it.

7:14 And what were some fun things that the game led to?

7:17 Well, you know, um I got kind of tired with graphics and that's why I kind of moved away from it after college. I got into storage systems and I bounced around uh uh you know I worked at one of the early search engines in me and then I I got over to another startup and um I met through that time period in that first startup I was doing um met Larry and Sergey from Google because it turns out that Larry I think it was maybe Sergey I'm not quite sure the very first version of the Google logo was done in the so they knew of us somehow one of them you know looked me up asked me to come interview at Google I did back in 2001 and uh

7:50 this is when Google was 3 years old.

7:52 Yeah. 3 years old. And uh I said no.

7:55 You did not.

7:56 I said no. Yeah. No. Here's the the calculation in my mind. I was like Google's down in Mountain View. I was living up in San Francisco and I didn't want to do the commute. So I uh went and worked at another startup for a year and uh that one after a year I like I saw it wasn't going anywhere. And um they called me back up and they said, "Hey, do you want to interview again?" I was like, "Sure." Like we're not going to be able to offer you the same stock options you got before. I'm like, "Okay." I can't remember what it was and I still can't remember what they offered me the first time. Um, but I probably would have made a lot more money if I had taken that first. But I did all right.

8:23 I'm not like complaining, but it's one of these things like, yeah, yeah, now I think back about that. I'm like, oh, a lot of good stuff came out of it. Some point along the line, Red Hat was going public. They actually offered friends and family stock during their IPO to a lot of people. We got offered friends. I made a little bit of money off that. Not a lot, but it was like just after college. It was actually quite significant at the time, you know. So, I mean there's a number of good things that kind of came out of that that

8:43 and I I also use the as a you know I was looking for like oh I can't afford Photoshop and here's the and it did so many things. So like I'm sure there's so many people like you had a positive impact of being able to use this thing and and the nice thing that I what I really liked about it is it was free but I wasn't like stealing any software from anyone. You see what I mean? This was at a time where free software was not as common as today. Free and open source was not as mainstream as it is today.

9:12 Yeah. Yeah.

9:12 And then the other cool thing I've heard from a lot of people because people still like I mentioned this like oh I've used it and then some people you know software engineers like I learned how to program by looking at your code and I'm just like wow that was the code I wrote 30 years ago and I wasn't nearly as good as a software engineer as I am now.

9:26 And then you got into Google second time around you you you said yes. What did you start to work on?

9:31 Yeah. Yeah. Well I got in there they

Architecting Gmail's backend storage and message threading

9:33 said like hey you know this was early days 2002 was April 1st auspicious date. Um your sorry date. Yeah. Uh April 1st, 2002. And I got in there and they're like, "Oh, well, you know, we're going to actually build uh email uh Google email." And it wasn't called Gmail at the time. There's a code name internally. It's called Caribou.

9:52 Caribou. Yeah. Um and got in there, started working on it, and I was kind of tasked with working on the the backend threading and message storage and indexing system. And worked on that, you know, pretty hardcore for the first year and a half. Actually, I was on maybe it was three years um on it through the launch which actually happened to be April 1st, 2004. Do you remember this? There was a uh it was like considering April Fool joke. Yeah, I remember. So, correct me if I'm wrong, but the the launch said Gmail with 1 GBTE of storage or or something infinite storage and like star one gigabyte. I'm not sure what it was, but back then most email providers would give you about 10 megabyte of storage, the free email providers, and then you could pay for maybe 50 megabytes or 100 megabytes, but that was really expensive. This launch, it looked like an April Fool's joke cuz who could possibly give you a free email service with 20 to 50 times or 100 times more storage?

10:47 We were talking about this internally. It was like kind of a shock and awe campaign for the industry. I I think it was actually four megabytes for like Hotmail or Yahi Mail and then you had this and not only that we had uh so much more storage but it was indexed really fast. So you know you could do a search and it would come back almost instantaneously. But c can you tell me can you tell me internally like when the project started okay we're going to do email for Google how did you and the team arrived to the point of like okay we will offer all this storage and we'll do it fast because this was at a time if you can take us back but what I remember is hard drives were still expensive they were relatively slow we're talking HDD we're not talking necessarily about SSD if I remember but if you can take us c can you take us back of what what it was like what the constraints were like and then how you innovated like to to actually like do something that's never been done before.

11:36 Yeah. Yeah. So, at the time, Google internally had this large distributed file system called GFS, Google file system. Yeah. And uh we were like basically looking at numbers and being like, "Yeah, we think we can build on top of this." They had a lot of knowledge about how to do um search and retrieval. It ended up being that like some of the existing systems they had for search and retrieval, they started building the prototype on there and then that was completely rewritten. That was actually what I got involved in because I got in there and there was already a prototype and then it was completely rewritten. So, I was on that, you know, uh, threading side. We decided to have message threading right from the get-go, which is also kind of innovative. That wasn't common in email systems

12:09 and and spreading to have the message thread, which I assume needed data structures on on the back and it needed like, you know, storage and and figuring out read intensity, those kind of things.

12:18 Yeah. You know, there's uh some B trees involved in that. I think that might have been the second time I implement B trees and I think I've implemented them like a dozen times now.

12:26 How do B trees relate to email threads? uh it's just like you have to you know have some storage there where you have a thread ID a message comes in you have to look up it was using the search index to match actually the the subject or the message ID into the thread and there are some other things taking place in the B tree as well because we were keeping track of the unread counts of of threads and and whatnot so you know I can't even remember all the details now this is ancient history this is 2004 what are we in 2026 22 years ago so it's like left my memory but there there was definitely be trees there was also this you know inverted index taking place there all this code has since been completely rear end.

13:00 What were the economics like in the team? You must have done the economics of being able to offer free email which of course I'm sure you did some maths of like how it could be subsidized but to like not make a terrible terrible loss.

13:12 Yeah. Yeah. Yeah. Yeah. No, I mean we were kind of uh kicking around ideas and um you know one of the ideas that had come up at some point was hey maybe we can put ads on this and it was one of these crazy things. So I didn't wasn't involved in this. This is Paul Bukite went on to do some cool stuff at uh friend feed and Facebook and ended up I think a partner Y cominator at some point but he just like one night he's like no I think I can just take some of our existing ad functionality and incorporate it in there and it just went like gang busters and there's a huge business that you know kind of grew up from that which is kind of incredible.

13:44 So I mean I think there's a lot of things that it's just like you kind of have a sense of what can be done like oh we think this is going to cost you know per user a couple dollars per year. how do we monetize that? And you know, we didn't want to charge for it. We eventually they did charge for it because you have the whole Google Workspace stuff, but at the time it was more like, ah, can we do this for free? Oh, it looks like it can be economical. And uh it it took a little bit of a a leap of faith. And do you remember the launch? Uh the reason I'm asking because I remember that it was an invite- based system. like not everyone could get in and I assume that must have been to control the expected demand cuz again you were you offering something for free that was paid before and it was kind of pretty obvious that there would be massive demand.

14:24 How did you think about it kind of monitor demand decide how many people to on board?

14:31 Honestly this is a team effort and that I wasn't involved in that. I mean I do remember being worried about like the load that would happen and then someone came up the idea of like hey we should do an invite based system and this also had like it played dual roles so it kind of constrained the growth but it also kind of drummed up this excitement oh can get me a Gmail invite. I remember people asking me at the time I was like yeah I can get you as many Gmail invites as you want.

14:51 And then after after after you built the the system where did you move on to what was your next project? Was was it the

Transitioning from monolithic Makefiles to high-level build systems

14:58 build system? Well, there was the build system and that was kind of all muddled in my mind because I was kind of doing that part-time when I was doing um Gmail as well at some point. You know, Google actually had this large monor repo. I think they actually still have the monitor reporter and they started out with the Google one um repo. That was before my time. Then they moved on to Google 2. That was what was there when I got to Google and at some point we saw the the strains of Google 2. And Google 2 was just one single monolithic make file. I think it actually had some submake files but it was this like really large unwieldy make file. someone came to me and was like, "Uh, I think we could do something." I'd had some interest in the build systems and I, you know, kind of put the uh foundation in for Google 3. There was a bunch of people involved. I was kind of doing like kind of the initial work on Google 3. And Google 3 uh the initial insight was like, hey, make files kind of suck to write. We introduced this thing called build files and I decided to do it. It's just this stripped down kind of Python language, but it was still Python at that point. And what Gconfig spit out at the end was a monolithic make file, but one you didn't have to write. Over time this evolved um it became uh blaze

15:59 internally um there's some other systems are associated with now I don't even know how complicated it's got I haven't seen it for a long long time um but then that became basil externally it became buck there's some other you know people went to Facebook and they like that was awesome at Google

16:11 what was the reason that make make files

2 Highlight · Why monolithic Makefiles fail at large scale

16:13 or make didn't really work was was it were we talking about build performance are we talking about maintainability or readability

16:22 yeah so like I think make itself is like It's okay in declaring dependencies, but it's a little bit like kind of assembly language and you didn't want to write, you know, all your dependencies in assembly language. And if you didn't know what you were doing, it was easy to make a mistake and miss dependencies and whatnot. And so the the kind of my the thought behind the build files is like, hey, you need to express these dependencies but at a higher level and just like kind of cleaner semantics associated with it. And from that you can compile down into the assembly language. And then eventually folks were like, oh, you don't have to compile down to that. we can just kind of implement you know the dependency kind of update engine directly

16:55 and then that's where the performance improvements were able to come in. So like because at Google scale like when you have a large repo in general like my understanding is that the reason Basil and Buck are so popular for large code bases is it can help you improve your build performance. It gives you a lot

17:11 more le lovers to play around with from caching from being smart about cash generation to obviously just raw performance.

17:17 Yeah, that's exactly it. Yeah. So you kind of just dabbled and like okay I'll I'll make this built file did that people took that over but what was your next main focus? Uh well so I mentioned GFS earlier, Google file system and at

Scaling distributed file systems with Reed-Solomon erasure coding

17:30 some point we realized that there was some limitations in GFS scalability bottlenecks because I've been working on the storage system for Gmail. Um I actually dabbled in another storage system, you know, kind of a research thing that never went anywhere. Um but because we're working on that, um got invited to participate in like the founding team of Colossus, which was the successor to GFS. And as far as I know, Colossus still exists. It's gone through multiple iterations at Google. Um but it's like the second generation, you know, distributed file system. So what is Colossus?

17:58 Yeah. So when I say a distributed file system, externally you might think of something like S3 kind of blob storage, it had a flat name space kind of like S3, you give names and there's like a minor hierarchy there, but it's like very limited. Um but it's not like a PZIX file system. So you don't have the full directory hierarchy. You didn't even have the full like kind of permission system. A lot of that stuff got it added later, but the files are not stored on your local machine. There is a fleet, you know, kind of a service out there that has all the files. they're writing it down to their hard drives or now SSDs and your client can access that and it's all replicated. So

3 Highlight · Replacing full replication with Reed-Solomon erasure coding

18:28 if there's any crashes and whatnot, you're not losing your data. I don't know when um at some point S3 got eraser coding. We did eraser coding in Colossus. That was kind of a big breakthrough.

18:37 What which coding

18:38 we we used Re Solomon. Um so for the audience who's not familiar with eraser coding, you might think of like I want to have replicas. Um and there's a this thing in hard disks called RAID where it's like I don't actually have to have full replicas. I can actually you know if you take a plus a and b you can exor them together and then you can have this kind of third kind of version and there's more and more complicated versions of that read solomon is kind of you know I think there's actually better codes now but it's like one of the known ways to known ways yeah and we we had to you know kind of pioneer internally like oh how are we going to actually make this work in distributed file system

19:11 were you focused on on latency on on being able to store data more efficiently

19:16 storing it more efficiently so in GFS and there's varying costes where it's Well, you're not storing two copies or one copy of your data or two copies, you're storing three triplication, three times as much storage you're having to use. And with read Solomon, you can get that down quite a bit lower. I can't remember offhand exactly what our um what we used for read solid, but I think it was like essentially 2x. But you you get that with also the redundancy, too. So, it's like it's smaller and the redundancy is higher.

19:41 So, cuz cuz I guess the the na the naive thing if if you're saying all right, I I want my data to be replicated at three places. You take three nodes, three machines, physical machines, and you say like copy one, copy one, copy one, I have it in three places. Great. If one explodes, I still have two. Wonderful. And then you're you're saying that the you know the algorithm here is you could take not 3x the data but 2x the data, split it smartly across machines or or maybe you could take it lower and you still have the thing where like oh one of them explodes, I still have all my data because it's it's split in a

20:10 that's exactly right. and like kind of the mental model you know if you just want to understand at a high level which is essentially you might want to say like I want to have eight replicas of this data or I think I can't remember offh hand I think S3 might use nine they've actually talked about this publicly you have kind of nine chunks of data but any five of those chunks can be used to reconstruct it and what this means is you can lose any four copies and you can still reconstruct your data um and oftentimes it's more like you know the the first five chunks are exact replicas and the the other four are kind of parody ones I've kind of forgot some of details escape my mind but it's along this line

20:43 but but when you come up with an algorithm you can then prove that this algorithm will work right like this is a little bit like I know in software engineering like maths and algorithms is a bit out of fashion but in this case this is really important because once you once you can prove it that this algorithm works it will work

20:58 it will work and um you know it's like the math behind here is like kawa fields over like GF2 something like that I don't even know that the I never actually understood the full uh math behind it I always regretted not doing or math in college, but you didn't have to like read and saw them and proved how this worked. I think it's like back in the 1970s something associated with like communication network. So you just take that and uh you know kind of use that expertise but leverage that and have to do all the engineering behind it to make it work in a storage system.

21:27 And then when building a distributed storage system like Colossus, what were other things beyond okay, you want to store data in a resilient but efficient way, what were other things that problems that you needed to solve? I'm I'm thinking things potentially like

Solving metadata bottlenecks and adapting to modern NVMe hardware

21:40 sharding and resharding or or or or or metadata uh being important those kind of things.

21:46 Uh particularly for the scale that Google wanted to operate at GFS kind of had this scalability limit. I believe it was like you know you could have a thousand machines in a GFS cluster and it kind of

21:55 which I guess sounds big until until like today it's kind of ridiculously small right

21:59 it sounds big for and then Google was like no we need to have the scale up to 10,000 machines and there were some bottlenecks there was this GFS master as a single node it was a bit of a bottleneck we're like ah we need to have a distributed master to store the metadata for all the objects and Google at the time happened to have this system called bigtable and so colossus stored this metadata inside big table and one of the things that I'm kind of proud and like kind of like also you know a little bit embarrassed by uh one of the design choices I I went down but actually worked out is we wanted to use big table for the metadata for classes

22:33 and the metadata just for for those of us not as into distributed system what is the metadata in a distributed file system

22:39 yeah it's like the uh the names of the files and for each of the files the files are broken into chunks what were they 64 megaby chunks and then you have to have the list of chunks for each file

22:48 and then you you have to um periodically you know the master has to be scanning over this and doing repair um repair work and you know but there's more metadata than that but that's it in a nutshell

22:57 so we wanted Colossus to have this you know kind of scalable you know service big table to store it metadata the biggest user of GFS is bigtable so we want big table to work on top of Colossus

23:08 so wrap your head around the circle dependency

23:11 yeah well uh there's a bootstrapping thing right

23:13 bootstrapping so yes

23:15 yeah which one starts up like you need to mock something somewhere right Yeah. No, I mean the way it actually worked at the time and they've since replaced this, you know, because this is like the way to get started, leverage what you have and eventually you kind of get rid of it. But there is the uh kind of a foundational big table that big table didn't use Colossus. There's the Colossus using the big table and then there's normal big table sitting on top of Colossus and it all worked for years. So I don't even know when they got rid of that. They got rid of it at some point, but uh that was

23:43 But I I I guess sounds like you can make like hacks that go really long knowing that they're hacks and they get you off the ground, right?

23:50 They get you off the ground, right? Cuz if we had to like, you know, implement that kind of big table layer from the

Hardware advances push storage to machine limits

23:54 get-go or it just delayed how long it took to get, you know, Colossus built. One of the things that strike me about a system like Colossus is it promises or this was internal to Google but but even distributed file system that are are external they will promise high throughput high availability and low latency and to me it's always a bit conflicting of like well it's pretty easy to I guess build a distributed file system with my limited knowledge I could probably do something where I have either high throughput but high latency because whenever I write something I write it out all the replicas. You you already like like mentioned one one technique of of doing it, but how did you kind of reconcile like how how did you get like low latency while you have high throughput while you also have replication going on on the file system?

24:40 Well, I mean the these distributed file systems, I mean the latency isn't super low in particular when they're when they're running on hard disks which Colossus was doing at the time which S3 does. You actually notice the latency. So like S3 is a high performance system. Google um GCS the the competitor from Google which is built on top of Colossus it's high performance has incredible throughput but the latencies are like 20 to 30 milliseconds per first read and that is bounded by your hard disk latency if you put it on SSD it gets down to closer to SSD latencies but not actually kind of the state of art SSD latencies which is kind of this crazy thing that's been happening in our industry it's just like how much faster the hardware has been getting reading from a hard drive maybe 5 to 10 milliseconds nowadays I never touch hard drives reading from an SSD over NVMe 30 microscs, 50 microsconds. So that's a microsconds. There's 1,000 microscs in 1 millisecond. So we're talking like a huge huge difference.

25:32 Well, there are now startups or infrastructure companies that are starting to take advantage of the fact that they can they can have an MVME layer and and they they they they pull things up either predictively or not. But but as you say, like when the physical reality changes, you can build systems on top of it that that should take advantage of it.

25:48 Yeah, absolutely. He knows thought that the discs are so much faster with SSDs. The networks are so much faster. I mean, just kind of crazy how fast like the intrazone latencies are in a Google or Amazon center. And it wasn't like that when we were building Colossus, you know, I can't even remember what the numbers were, but milliseconds to do a network round trip and now it's down in, you know, 100 microsconds within a zone. I mean, I just look at these things, I'm like, holy crap. You know, the hardware guys have really done a good job.

26:14 Yeah. Some sometimes I feel that our software should be feel way more snappy and there are some snappy software but sometimes I almost wonder if we're getting too complacent with all the abstractions or we're not even doing this like napkin math. Simon Erikson at Turbopuffer talks about this napkin math where like like you was like all right here's the theoretical limitation of of the hardware read from SSD might be I don't know 30 m microsconds and then like how can I build a system that is as close to this as possible as opposed to the other way around saying okay like you know a human will notice like 20 mill like 100 milliseconds let's build around that.

26:49 Yeah. Yeah. And sometimes like when you're architecting something, you need to think about the human kind of perceptible latencies, but oftentimes when you're dealing at the storage system layer, you know, I know Simon working on Turboper, they're doing great stuff over there. You have to think about the machine scale and the machine speed, which is a lot faster than human perception. Like a human can tolerate maybe 100 milliseconds of delay or if you're playing a game, maybe you need to have like, you know, frame rates of like every four milliseconds, but the machine wants much much faster than that.

27:16 Someone says napkin math. I call this speed of light numbers and like sometimes it's literally the speed of light bottlenecking you. The cross zone latencies between zones, cross region latencies is speed of light in fiber. You want to hear a kind of crazy fact?

27:29 I I love hearing crazy facts.

27:30 The fastest way to send um packive data across the globe is to send into space.

27:35 Is it because the speed of light is faster in vacuum?

27:39 Quite a bit faster.

27:40 Or well, it's not full vacuum. No way. So So you cover a higher distance.

27:45 Yeah. Well, you actually uh I believe the the way to do this, you send it straight up and then you with Starlink, you send it straight up, you bounce it around between Starlink, you send it down to the other side. So, you want to get it out of the atmosphere as quickly as possible.

27:56 But now, if we're if we're talking speed of light, this will also go there as like a digital transformation happening. So, you would need to calculate how long it takes for that system to to process and do it. And and but you're saying that even if if you do this like really well, it will be faster than beaming it through an optical c and an optical cable slows down the speed of light, right?

28:13 Yeah. Yeah. Yeah. Yeah, the speed of light is is only the speed of light in vacuum. In every other medium, it's slower.

28:17 I was talking I did a deep dive on uh the hedge fund industry and they didn't tell me exactly. They said that they do use satellites and microwaves and some of these things. They they will not tell you because you know this is their thing. But I had a suspicion that they might have found a faster way and I think this is like somewhat wellknown but the the details they're not going to get into because again that but like Yeah. So so they're probably bouncing stuff in space.

28:40 Yeah. Yeah. So, I mean, one of the things that they do in the high frequency train is like between New York and Chicago, it's actually not far enough distance-wise to make it worthwhile to send it into space. So, they were doing microwave beams there. Yeah.

28:51 But, I mean, if you really want to get faster, you need to build like a vacuum tube between them and like send a maybe they're doing that. I don't know. Maybe they're doing that.

28:58 Yeah. But, I mean, the thing that I think is like just kind of awesome about performance nowadays is I mean, there's so many layers you have to be paying attention to in terms of performance that the rabbit hole just goes so deep there. you're almost certainly running on multi-threaded systems, right? And you're like, "Oh, I need to have a multi-threaded program. I need to have synchronization in there." Well, you get the best performance if you just kind of avoid the synchronization. And part of this is lock free programming, but part of it is arranging. So, you don't need locks at all. You need to carry about your processor caches. And there's this whole like kind of uh setup of caches above the CPU. Like you can think of registers as a cache, and you have L1, L2, L3, um even your memory and onto disk. Um and there's just you have to pay attention to all those levels. And if you do, your performance gets way better. And if you ignore it and you're like, "Oh, I'm not worrying about kind of kind of, you know, kind of the cache accesses. I'm just accessing data all over your program will just be way way slower." And some folks pay a ton of attention to this. The high frequency trading, I mean, they do this all day long. And and in so many other places, like no one pays attention to it, and

29:59 you get this kind of gradual degradation in the the performance of the hardware or the software. I did want to talk a bit more about low-level stuff, but not about the the speed of light, but low-level data structures and programming language features. You made some contributions to the standard library right?

B-trees improve memory performance over binary trees

30:15 Well, I've done a couple um not quite standard libraries. So, uh I mean I just like have been always fascinated by data structures. It's kind of awesome. I mean, I I think just algorithms in general, you just like sorting algorithms, they're just kind of awesome. You could probably explain, you know, insertion sort and you know, kind of everything. Could could you explain quick sort as well?

30:34 Quicks sort's a little bit harder. I

30:36 that's the thing.

30:36 Yeah. And then

30:37 No, no, but it's it's a smart one.

30:38 Yeah. And then you get to these levels of like, you know, it's like, oh, wait, someone really smart came up with this. So, one of the things I worked on just as a little bit of a side at Google at some point was a colleague came to me and he was like, you know what, we're

4 Highlight · B-trees outperform binary trees via spatial locality

30:51 using the STL map structure all over the place. And the STL map structure is a balanced binary tree. I can't remember if it was red black trees or one of these other bouncing algorithms that many CS college students implement. And he came to me, he's like, I think we could do better because, you know, there's actually a cache problem here. Every time you every node you're traversing down, you're going to a different cache line. And he was thinking about using something else, a skip list to do this. Um, which, um, that's another awesome awesome data structure everybody should kind of look at. But at some point, I was like, actually, this feels more like a B tree.

31:22 So I'd implemented B trees a couple times before and figured out like how to implement a B tree that implemented almost all the semantics of the STL map. It couldn't quite do it perfectly and the reason is uh when you insert into a B tree node you have to shift stuff around so you don't get pointer stability. This is just kind of fundamental but if you can you know you don't need that for your use case you actually can pack more data in. So the thing about B tree like the the the real

31:48 easy way to describe this is like you just have a small list of items like eight items that the the best way to store that if you want to you know kind of fast access in sorted order is just to sort the items right literally not to have a tree at all. Yeah. for like eight items

32:01 just have it in a very simple list

32:03 very very just an array sort the array and then you can either do a linear scan over it you can do a binary search and oftentimes linear scan is faster and then you think about that like well if I want to store more than eight items I can just have one node that has eight items and then I have another node and then you have a parent node that connects them together and that is essentially like the you know you build it you think about building it bottom up you start with just one node of eight items oh I need to insert the the ninth um thing I'll split into two chunks and the two chunks one will have four the other will five and then you have a a parent node that points to them and then you just kind of recurse on that. That's the B tree algorithm in a nutshell.

32:36 Everybody go implement it. Actually, nobody should implement this anymore because nowadays we we have something else that will implement this and all the optimizations because there's a crap ton of optimizations that you can do on a P tree. But but I just want to go back to this like there was already an existing implementation for maps and then so your your colleague looked at the code and said we I think we can do better. I what I want to figure out is like in my mind again someone sitting outside of you know I'm not involved in in how some of the these libraries are or data structures are built. I always thought and again this is might be naive but really smart people sit down they kind of look at the state-of-the-art they implement it and there's no way it can be faster. In fact I've had arguments in the past saying like oh let's let's write a faster sorting thing. Like it's surely it is the fastest. But if if you could bring us a little bit of like how like you you've been inside how it act how it happens and how other people like yourself and and your colleague can say like oh what what if we what if we try something else?

33:35 Yeah. Yeah. So I mean my recollection here is he was working on this kind of the big um internal system I think it was called Gaia that actually had the mapping from you know you log in you have your user ID and you have to look this up and they were storing you know all all like this the map from user ID and email to whatnot to the metadata about the user

33:55 in STL maps and you just noticed like well there's a lot of memory usage here and it shows up on profiles and then we're like well what can we do to do better and that was kind of the genesis of it and you know he happened to be working on it me happened and they were working with me and like we just started kind of noodling on this problem like oh can we do something better and it's not one of these things like I think now with Google they have a whole team working on their kind of internal libraries at the time it was more of like you know everybody working on their own systems and contributing to a shared base but but I guess it still goes back to what you were just saying of like just go down the layers try to understand and if something just doesn't add up like suddenly like oh there's this big explosion of memory usage like you know just ask the questions why why is this and if you're able to or you're happen to be like, oh, can we do something about it? Right.

34:38 Yeah. And and one of the things that, you know, he observed early on, I think part of one of the things was it was like a map from integer ID to something else. And like if you look at red black tree, every node you have your kind of value that you're storing in the map and then you have two pointers. You might have an integer ID that's like four bytes and then two waste.

34:56 Yeah. And you look at that, you're like, "Oo, that seems like a lot of overhead." And like

34:59 just you could use a bit. Well, the

Swiss tables optimize hash map performance in Go

35:00 beach tree actually has a lot better um it has better spatial locality and that's what made it faster. Um but it was actually smaller as well at the same time because you had less pointers involved.

35:10 You also contributed to Go, right?

35:11 Yeah. Yeah. Well, that came later.

35:13 Yeah. Yeah, it came a lot later. Can we talk about that?

35:15 Yeah. I just one of these other things, you know, I I pay attention to like, you know, when there's research papers coming out about new data structures and like hashts. Hashts are like the the one of the earliest things you learn about in college in data structures. like how do I map keys to values where the ordering is unimportant that's when hashts come in and there's like you know the the very earliest ways to do this I've implemented hashts multiple times is like you take your key and it might be a string and you you put it through a function and it spits out an integer and then you map that into an array of buckets and if multiple things map to the same bucket you have to have a link link list

35:49 you have a link list yeah this is a naive implementation

35:51 naive implementation used quite frequently and over time people discovered like a lot better ways to do hashts there's very Like that's called chaining of your hash. There's another technique called open addressing where instead of actually having a link list, you just uh kind of hash it again and move on or kind of walk down to subsequent buckets to find out like or sub subsequent slots to find out where you should be. And I remember reading about this uh new technique. It came out of some folks at Google. I believe it came out of their Swiss office. It was called Swiss tables.

36:20 Um I believe that's where the naming Yeah. Yeah. I believe where that's where the naming came from. um not 100% sure about that, but I remember reading about it and then I was working on Go for a long period of time and Go has this built-in map structure and it's a hasht.

5 Highlight · Implementing Swiss tables in Go

36:33 a very highly optimized um hasht because the go team is very competent um the go runtime tag team and various folks had taken an attempt at like you know um putting together a Swiss table implication for go and I tested some of them and I was like this is kind of fascinating what Swiss tables do and I'll explain how it works in just a second but I looked at it's like well it's really hard to beat the performance of the runtime the runtime was really good and I kept on I noodled on this for a little while and uh eventually I ended up having to take this business trip to uh to India to Bangalore and so I was on a long flight. No.

37:07 Yeah.

37:07 It always starts like this

37:09 and I just like I'm just going to try to uh pull on this and I pulled on it sufficiently that I can get some of the benchmarks to be faster.

37:15 Wow.

37:15 And then like you know that like is kind of like catnip for an engineer like can I make it all faster figure all the rest of it you know got some help from the runtime folks. there's a an issue on the go issue tracker that you know where other people have been attempting this because people proposed like hey let's use Swiss tables and like the go like well you know we don't quite know all the details you're going to have to navigate this and that and there were some ideas there that combined them together and um got to the point where we had kind of a complete implementation that was faster on on most benchmarks not quite all of them but most of them and then the go folks eventually picked this up and pushed over to the finish line and

37:48 and then so you you know like you came

37:50 with the idea you you got to the point where you were able to show an implementation that showed how some of the benchmarks were faster. And then you started to work with some folks on the Go team, too.

37:59 Well, I didn't I it wasn't quite that I I came up with an implementation that we ended up using at my company. It was good for our use case, but actually putting it into the runtime is a whole other, you know, kind of level.

38:09 But then you you just showed like here's this implementation and then they they they kind of took the inspiration and the ideas and

38:15 yeah, they're like this is great. We want to make all they they always are looking for ways to make runtime faster and there's like you know

38:21 oh wait and then so so you did this in in this contribution this was a lot after you left Google right so this was from from the outside

38:28 this is from the outside

38:28 that's awesome yeah I know that people contribute stuff to the outside as well you know we had another colleague um he contributed one of the CRC implementations you know adapting some stuff that

38:37 CRC

38:38 uh cyclic redundancy checks sum you know Intel published some papers about here's how to do the CRC in assembly very fast and contributed one of the implementations you see a number of those things where you know people just like are contributing externally. It's not a lot honestly. I mean I actually don't know the full details but you know people are regularly contributing to these things. Peter just described how the Swiss table work came together using an issue tracker in the go tracker with different people contributing to the work and then the go team pushing all of this over the finish line. This is where I need to mention our season sponsor linear which is a place to coordinate work between humans as well as agents.

39:12 One thing I've noticed about how most of us work with agents is how it's a pretty single player thing. You open a terminal UI, go back and forth with an agent, and it usually produces a PR, but the rest of your team has no idea what happened in that chat unless you tell them or copy the whole history. And when everyone in the team works like this, a lot of work happens that's invisible to the rest of the team. Linear's take is that agent work should be teamwork. Even today, teams already use linear to define the work to be done. Now, linear can already delegate an issue to a coding agent. This agent could be an AI agent that Linear integrates with like Codex or Cursor or Linear's own agent or a custom agent. Either way, the engine you're delegating stays responsible for the outcome. What I really like about how Linear works is how the work stays visible. Your teammates can follow the session of the agent, check out the PR producers, and join the review. We've gone from single player work to multiplayer engineering work with agents. Oh, and one more thing I like about linear, a focus on costs. Linear agents auto routing chooses a model that is the best suited for the task. Teams can also inspect usage and set limits,

40:13 track usage, so you can use capable agents without having cost balloon out of control. Hop on board at linear.app/pragmatic. Peter also previously mentioned Gaia, Google's internal system that map login to users. It's not surprising that Google customuilt all systems, including this one, but most of us won't build our internal Gaia. This brings us to our season sponsor, Work OS. You can think of work as something like Gaia for the rest of us. Identity infrastructure you've otherwise spent quarters building yourself. Workers includes single sign on, skim directory sync, audit logs, rolebased access control. Basically everything a big customer security team asks for delivered as a handful of clean APIs. It's how companies go from we have a login to we can sell to a fortune 500 without standing up their own internal identity platform. And work is already building for the next version of the agentic authorization problem. Their newest product is airlock. the authorization layer for AI agents. Think about what happens when you hand an agent a task like clean up the sale opportunities in our pipeline. The last thing you want is for this thing to have

41:13 standing permissions to delete whatever it likes. Airlock sits between your agent and the tools they call. It evaluates every request against the agents intent and your rules and then it allows it, denies it, or routes it to a human for approval. You write down the policies in plain language. The agent never sees your credentials and every call and verdict gets logged. It works with coding agents like Cloud Code and Codeex and with MCP gateways. So if you're working out how to let agents do real work in production without overpermissioning them, take a look at work airlock at work.com/airlock.

41:48 And with this, let's get back to Peter and why he left Google after building Colossus. You were at Google, you're building Colossus distributed file systems. you were at this point probably working on probably the largest system on the planet honestly. Why did you even consider leaving?

42:04 Yeah. Yeah. Well, uh after Colossus I kind of dabbled in this other project called Google Goggles for a little while. Remember the glass holes? Yeah.

42:12 He keeps coming back the idea by the way. So

42:14 yeah. Yeah. Now it's still here and present.

42:16 It seems like Google's early.

42:17 Yeah. Yeah. And you know I I think that was a technology before its time. I don't think it was ready to do at that point. And looks like the uh actually doing the glasses is quite a bit harder. You know the Android phones we were trying to power it on were you know not powerful enough. And then you know I just kind of got wanderlust you know like h you know am I just kind of stagnating here. Google which is a strange thing to say but you know some other people feel as well and decided to go off and try my hand at another startup that didn't work out. We got aqua hired by Square.

42:45 So yeah I want to pause for a second. So this company what was the company name?

42:49 Uh the company that we founded is called Viewfinder. viewfound viewfinder. Yeah.

42:52 Yeah. It was in the mobile photo sharing space which should sound familiar. This is like Instagram. This is like this was in 2012. Yeah. Right. As as Instagram and all of them were were taking off.

43:02 Yeah. Yeah. No, we were right there in the play and uh we just didn't have the the right go to market kind of like how to attract the users, how to get viral growth kind of thing

43:09 cuz from the outside like what I read when I when I checked you know the story just like oh you know like you you you co-ounded a startup it got acquired by Square. Hooray. Like sounds like you had bigger ambitions and this was a a decent outcome but not not the the dream, right?

43:25 Yeah. Yeah. No, it wasn't the dream at all. So, uh the term I used was aqua hired. So, sometimes a company will get acquired get bought for you know their IP for their product for their business and other times they get bought just for the talent the people the people and we got bought just for the talent. They acquired the IP but I don't think Square ever did anything with it. It wasn't kind of like where they were working, but we had built up a kind of strong technical team and that's what we were hired for. And you know, like I can't remember the details. We'd raised a small amount of money. We were able to pay our investors back, make them whole.

43:55 Maybe they got a little bit of a haircut, but maybe they actually got a little bit of a but it was it was okay. It's essentially they got their money back, which is like

44:02 like you know, as a founder, you uh you know, investors are big boys. They're used to losing their money, but you kind of feel bad if you lose a lot of money for them. So, you know, getting them paid back kind of makes you feel a little bit better.

44:13 Yeah. Yeah.

44:14 And and then you you spent a little time at the company that acquired you was just Square. Yeah. And then you started issuing that a little bit again.

44:20 Yeah. Yeah. Yeah. Because, you know, we've been working on these, you know, distributed file systems and storage systems. You know, Colossus, uh, one of the kind of sister projects to Colossus

Spanner introduced immutable storage and LSM trees

6 Highlight · Distributed filesystems versus distributed databases

44:29 is Spanner. Um, and how how is Spanner different to Colossus?

44:33 Well, Spanner is a essentially a distributed database. Colossus is a distributed storage system and kind of the the way I think about the difference between a distributed storage system and distributed database. You might think, oh, they're both storing data. I

44:46 I was about to ask because a a distributed database will at some point be a storage system,

44:52 right? Right. So

44:52 So what's the difference?

44:53 Yeah. So for Colossus, you know, was targeting large files, large appendon files. You can't update them.

44:58 Append only. Yeah.

44:59 Yeah. Large appendon files. So you know, 64 megabytes, maybe up to gigabytes in size. But if you're a database, you want to be storing like kind of small like you know kind of you know if you're using SQL or like the relational data you you might have table with you know billions or billions of you know rows those rows are broken up into columns the columns are typed that it just has a very different nature to the engineering challenge for a database than it does for a distributed storage system and usually distributed uh databases are implemented on top of some kind of distributed storage system and that was the the relationship. So Spanner was implemented on top of Colossus and some of the design decisions in Spanner kind of directly fell out of the appendon nature of the files in Colossus. You can't update a file in place. So you have to you know make the files immutable in your database. And this is where like you know log structured merge trees you know come into play. And they weren't invented at Google um but Google really popularized them with level DB which emerged out of the work on big table and spanner. Um that got

45:59 popularized into Rox DB. I subsequently reimplement one of these things and this is what we use at Cockroach Labs. It's

46:05 called Pebble. So I'm very familiar with the internals of that but it's kind of all based on this idea that the the data is kind of stored in immutable files. So how did you decide to found Cockroach Labs while we were working at Square? Um my co-founder and I was actually three of us. Uh we were all at Square and one of them is Spencer I mentioned earlier. He was working on the He's my college roommate and uh he was also at Google.

46:28 He was also at Google. um this other guy Ben Darnell who's also at Google, he joined us um at Viewfinder, ended up at Square. We you know like we're kind of just noodling on a project to do and we we'd actually had this design back in um Viewfinder. We're like ah we didn't we looked around for a data base to be using in order to uh build Viewfinder on top of we didn't really like the things that were out there. The technologies inside Google looked better. We had the big table. We had Spanner and whatnot. We were looking around and you know like HBS existed but I wasn't quite happy and there's some other systems like React and others and you know at one point we just kind of like you know came up with a design for um cockroach to be an initial design then I was like no no guys we're doing a mobile photo sharing site we shouldn't build a distributed you know database so we we

47:11 put it on the back burner which I think was absolutely the right thing maybe or maybe we should just pivoted away from doing the mobile photo sharing site given the way things worked out and then we got to square and we saw some of the same problems that they were experiencing with data storage systems And the Spencer is very convincing, managers convinced some of the management that like, hey, can you just work on this part-time, you know, to see if it had life behind the the design and then kind of conscripted uh Ben and I into it and eventually started getting, you know, attention externally and we're like, hey, can we go and spin this off into a company? And that's what ended up happening.

47:39 And so you started a company, but I understand you didn't raise VC funding initially right?

47:47 That was the case at Viewfinder at uh we did it differently at Viewfinder. we kind of assued the VC money and you know in hindsight I wouldn't recommend that um but

47:57 you wouldn't recommend

47:58 yeah I wouldn't recommend taking the VC money because my experience the VCs are very very intelligent and they can help you navigate a lot of challenges you know I think sometimes there's this perception you know the VCs will push you into various areas and maybe there's some bad ones out there that do the VCs I've had experience with just like some of the sharpest people you know I've ever met and

48:16 so so you kind of have like an extra like like person helping you on the team pretty much

48:21 mentoring you, giving you guidance, telling you what they're seeing. They give you advice, seeing what they're seeing in the market, where things are going. That is very hard for sometimes for a founder,

48:29 especially as a technical founder, right, that you're you're focused on the engineering part.

48:32 Yeah. Yeah. Yeah. No, we actually took money right away for Cockroach Labs. Um, it was almost like as soon as we left, we went out and did a little road show, you know, kind of in the the Bay Area and got some interest and, you know, got a investor right away.

48:44 I have to ask about the name though.

48:46 Yeah.

48:46 How how did the name of cockroach?

48:48 Yeah. I uh so you know we named the Yes, that was uh that was mine. Pul fiction come out in college and like oh what should we name this thing? Oh the new image manipulation program. I think we're thinking image manipulation program initially and I'm like oh it's obvious. It just stuck at some point you know we're newly on the this new database and you kind of want to give things a name. Um you can't just say oh we're working on this distributed database. You kind of need something and this master is like oh cockroach DB like cockroaches are unkillable. I want these things this database to be unkillable.

49:17 you know, cockroach is going to survive the nuclear apoc apocalypse. So that that was where the genesis was and just stuck.

49:24 Yeah. We're we're whenever a bunch of our nodes goes down, this thing will still be up.

49:27 Yeah. Yeah. And you know, this is where we're at today with cockroach DB. It's like one of the things that I point out, I'm like, "Holy crap, this is awesome. You kill a node. We we did this whole campaign last year, which was really just to prove out something already been present. The campaign was performance under adversity. But just like you can run a workload against it, you can kill a node, you can sometimes kill a whole region and the system keeps on going. And it's like stories like that, you know, like what we did on the marketing side there, but also we hear this from our customers, too. They've had fires in data centers and all the other data

Outgrowing single-node databases drives distributed adoption

49:54 systems go down and cockroach DB keeps on going. I'm like, that's awesome. When you started out, who who were companies, startups that who wanted to use cockroach DB and and how has it changed since? Because you know like just making the case like okay I'm I'm starting a startup it's a small startup like I will need a database and I'll I don't know I'll split up I'll typically choose a postgress right it's free everyone's using it I I'm running it on node at what point did you see that typical typically tech companies are like okay like this is not enough for me that it's running on a node either because it can go down or because I'm out growing it what was it they outgrowing like I'm I'm trying to get a sense of like at what point did companies say like tell themselves like we need something distributed

50:39 in a database.

50:40 Yeah. I mean often times we have companies calling us up after they've had a disaster.

50:46 So so so like a node went up or a hard drive failed that kind of stuff.

50:50 Uh you know we it's not quite like we're ambulance chasers but if you see an outage like a big outage from some you know company you know like we'll sometimes be trying to knock them up but also they they uh will call us you know and mean like there is a very big bank who's now a customer. um don't think I can name them but you can go read they had a very serious outage due to a weather event and after that

51:11 which probably knocked down I'm assuming a region or or database or a networking cable or tree fell on something

51:16 I think it knocked down a whole region you know was a regionwide power outage you knocked down the region and there's a mandate from the CEO it's like no we just have to you know be able to drive these things and that gets pushed down all the way and you see this in other places where you know one of our early customers they were running on AWS and they just got to the maximum size you can run in A rur instance on and then what would typically happen at that point is then you have to shard your database. You know this is very standard practice. You you take your single node database you create 10 or 20 or 100 shards and this is what Google done for some period of time and that's a a heavy burden on the application developer and

Contiguous key ranges power automatic sharding

51:50 the way we always phrase this is like I mean the application developer is becoming a database developer at that point and they're doing it poorly. You know they're trying to implement distributed transactions or indexes and whatnot. We felt the burden for that belongs on the database developer. Can we talk about automatic sharding? I think it's safe to assume most of us will know what what sharding is when when you're but actually let's start from like manual sharding and then how you can implement automatic sharding and if if you can tell us like you know tactics that a database like Hogwarts DB can can do to actually just take that load off of you. Yeah. Yeah. So I I I think the the very basic form of sharding is a little bit like the hash table. Let's say you know you have a fixed number of shards like let's say it's just a 100 shards. your data model is a a user with a lot of data associated with the user, you just take the user and you say like oh they map them to one of the shards and you know you kind of just rely on the hash function to get like fairly even distribution. The problem with this is at some point, you know, one of your shards will get full and you have to kind of reshard and that's a very very ownorous pro um

52:48 resharding I guess simple way to do is like if if it's just a hard drive, I don't know, per per node where you write the user data gets it gets full and you're like okay well I now need to split it somehow. I need to move it. I need to remap it. I need to reject my metadata which knows where this data lives. That kind of stuff.

53:03 Yeah. Yeah. And depends on exactly how you're doing that mapping from like you know the user ID or whatever your shard key is to the shard. you might have to remap them all, right? This is very typical. So, exactly. So, I mean, and this happens in hashts where if you know oftentimes in order to grow the hash table, you just have to essentially create a new hasht, double the size and copy all the data over. Now, that's kind of like the very basic straightforward way. And there's uh various levels of complexity you add on it. One of them is called consistent hashing. Um, and there's various techniques to do this.

53:31 It's kind of fascinating like uh how they all work but in consistent hashing you can add an additional node and then it only moves a fraction of the data from each shard over there. Um there's various systems to do that. I believe

7 Highlight · Contiguous key spans replace hash-based sharding

53:42 this is like when Cassandra the way cockroach DB does it is more akin to big table more to spanner more to h base where instead of actually hashing we actually take you know you can imagine all your keys in a system and this is always true in a system that you can imagine just one big contiguous key space and then you kind of partition contiguous spans of that and then you have to build up an index on top of those contiguous spans and you what I just described there actually sounds a lot like a B tree. So that there's this index on top that is like that that maps you from, you know, like I need to have this range which node is it on and this is a little bit like a B tree, you know, you kind of squint, you know, it's like I think you squint and everything's

54:20 either B tree or it's a hash table and you know that index structure, but it's it's now starting to make sense cuz when I I remember when I read it might have been the Wikipedia article on B trees, it it said this is data structure that is frequently used in databases. It's now coming back to me because I didn't think too much of it. I'm not a I'm not not someone who builds databases. Now that we're talking, we just like organically keep touching all the trees again and again.

54:42 And the the other place that it comes up in databases. So this is where it kind of comes up in distributed databases and no one ever really calls it a B tree. I just kind of squint sometimes and I see like it's actually kind of a B tree. But the other place it comes up in databases is for your indexes. So if you have a table and you have like an index on your email address um and you want to be able to scan over those email addresses in order, that's a B tree under the hood in a database. And you know like any kind of index you have it usually provides sorted order. There are hash indexes but oftent times it's the B tree index and they are ubiquitous in databases.

55:14 There's actually a paper called the ubiquitous B tree. And you know

Strong consistency simplifies application development

55:17 basically I just identified that I think that paper was written back in the 80s and they're still ubiquitous today. They are the foundations of single node databases and pretty much every data system I worked on has had bries at some point place in them.

55:29 I want to ask about strong consistency. Uh so Cockroach DB offers strong consistency. Now for people who are a bit more newcomers to distributed systems, can we talk about the consistency models and then why strong consistency is important and why it's hard to implement it?

55:45 So I mean there's multiple ways to kind of approach this but I mean if you've used a database you probably heard that uh of transactions and transactions are a way to perform a whole bunch of mutations uh atomically. So databases talk about atomisticity, consistency, isolation, durability. The durability is really easy. It's like when I write data to the database, it has to be durably written. So if anything crashes, it comes back. Um the atomicity is just referring to the fact I I want to do a whole bunch of changes. I want them all committed or all aborted at the same time. I don't want to have like some kind of partial operation. And why is this important? Why is the admissity important? Well, the atomistity is what gets you to the point where it's like as an application, I can do a bunch of operations and if there's an error hand an error occurs, it all kind of gets rolled back and it's a much simpler um development model to work within. And then there's the consistency and isolation which kind of get you know kind of muddled. The isolation is referring to isolation between transactions. I don't just want to one run one transaction at a time. That's easy to do right. I want to run a lot in parallel. Oh yeah. And make it so that when they're running in parallel they are running as concurrently as possible.

56:46 But you want to have the appearance when they're running um as concurrently as possible that there is kind of a serial order to them. So this is like kind of the the whole trick. The the kind of the gold standard for um isolation is called linearizability. Don't worry about that. There's a the step down from that is called serializability. And that literally refers to having a serial order of your transactions. But you're having to construct that in a way that you're doing everything as as concurrently as possible. And the benefit of this the serializability is again it's a very simple model for the application program. They don't have to worry about weird kind of defects occurring in their program. And some of the ones that can occur is like the classic description is of a bank, right?

57:24 where I might want to uh read, you know, like have an operation that reads and says like, do I have $100 in my bank account to transfer somewhere else? And you could like arrange for lesser isolation levels that you might be able to subtract that $100 twice. And that's bad, right? You know, we want to keep accurate, you know, track of your bank account or, you know, what's in your shopping cart or, you know, it's kind of like the the use cases are endless there and you do this wrong and you have very egregious bugs. But now going back to to to weak consistency and strong consistency.

57:56 Anything less than linearizability or serializability might be considered kind of weak consistency. But there's also like you know kind of strong consistency and eventual consistency. So the eventual consistency is like sometimes I can do an operation and it might not be immediately I might not be able to see all the the updates.

58:12 The read results will not necessarily be the current update

58:14 but they'll come back you know at some point you know I've written part of it and the rest of it will show up at some point. And often times when you're doing

58:21 the easiest thing is your credit card balance right?

58:25 Yeah. And uh it's faster to do it that way. Faster in terms of just what the performance you can get out of the system. But again, it's a little bit harder for the application to deal with. One of the places this often comes up in in distributed databases or databases that have any sort of replication is that I could write to the primary replica and then I read from the secondary and it's not there yet. This is that's eventual consistency.

58:43 Yeah, that's eventual consistency. And you can often work around this but you just like puts bigger burden on the application developer because you have to pay attention to that.

58:51 And then with cockroach DB you have strong consistency meaning as soon as you're writing it when you're reading from the database you already get the written value back.

59:00 You get the written value back. You know it's like you you read whatever you wrote. It doesn't matter if you're reading from the same you know node you wrote it to. If you read it from another node you still actually get the data you just wrote. Is the trade-off logically not that you would have higher latency because clearly to implement strong consistency you would somehow need to in the naive approach you would need to write to all replicas right

59:22 what are you doing inside cockroach

59:24 well we are writing to all the replicas so

59:26 well well yeah

59:27 yeah yeah you're doing it fast

59:29 you were just doing it fast you're making that efficient I think this is one of the things that kind of also fascinates me about the software industry is we keep on finding ways to be more and more sophisticated in order to provide you know like do do things that make it easier to write the applications but do it in a very high performance way and we've gotten you know very very good at this over over the years and this is the area I know about the databases this is happening everywhere like I'm just fascinated by how fast graphics have gotten where when I enter entered the industry like you were literally like writing out each individual pixel and now you have these GPUs that are doing like billions of triangles per second whatever the current numbers are and I'm just like kind of astounded like how much sophistication got has gone into every area of computer science wherever you look at it

1:00:07 one more thing on cockroach DB I want to ask about the raft consensus. What is the raph consensus?

1:00:12 Yeah, I mean consensus protocols, the original consensus protocol is called Paxos and it was famously hard to implement. Raft was uh you might think of it as a variant of Paxos. It was kind of like an alternative to Paxos, but in my mind today

1:00:25 and then the consistence algorithm being that you have like a number of nodes like three to a lot more and then how do you get them to agree on what do you typically agree on?

1:00:34 Yeah. Yeah. So you agree that the right

8 Highlight · Why consensus requires at least three replicas

1:00:35 occurred. So like the way to think consensus. So you might think I want to replicate data and I I I write it to a primary and I write it to a secondary. And you you can't actually have consensus when you only have two replicas. And the reason you can't have consensus is if there's a crash and I come up like if I'm on the secondary, how do I know if something was written to the primary? If I'm on the primary, how do I know it was written on the secondary? Right? And you're always going to be in this kind of confusing place where it's like you either have to roll back a little bit or like you know you lose some data. And consensus requires at least three, but you can have consensus across more than three replicas. And the idea with consensus is I'm going to write to three places. And normally you don't actually when you're doing a read, you don't read from multiple of them. But if there's a crash, I have to do recovery. Then I'm reading oh, I can read from uh two of the uh any two of the three and I know I can like kind of determine what had happened previously. Um and it's usually just on the the recovery time that you're actually doing that consensus read. So reads are typically just happening from one replica. You have to

1:01:33 write to all three and on a crash during that kind of failure is when the consensus read occurs

1:01:38 and inside cockroach DB how many replicas do you choose for the consensus?

1:01:43 It's typically three. Um for some system tables it can be five and then customers also have control of this. At the database level you can write to five, seven. Five is like, you know, if you're really concerned, uh, you know, about kind of the data durability, you might use five. But there's a slowdown, you know, the more you're writing to. It's like the more storage space that

1:01:59 and and obviously like we're talking like of slow down with nodes, but of of course if they're like between regions, there's now you have a lot more resilience for let's say an earthquake or a power or whatever. But now you will have additional latency. Just speed of light basics, right? Speed of light latency, right? and it's you know tens you know or up to hundreds of milliseconds or even higher if you're going across the globe. So you know you have to be very careful with that um in terms of how you architect your queries.

1:02:25 I mean one of the things that is just a general truism of distributed databases is you don't want to have a lot of back

Mission-critical tier-zero systems demand high availability

1:02:30 and forth. You want to kind of like do all your reads in one kind of parallel read set get them back then do your writes right but if you're doing like kind of serial operations where I I read a row I write a row I read a row I write a row I mean the the latencies just add up. We talked about founding Cockroach DB, but how how is the company growing and where are you today?

1:02:47 Yeah. Yeah. I mean, we're being used in like we're powering mission critical applications. That's our bread and butter.

1:02:52 But by the way, can can you elaborate on mission critical because it's like uh

1:02:55 Yeah.

1:02:56 If you're not you are in the industry where you know what this means, but from the outside it can feel hard to put a thumb on what is mission critical is is is is my SAS that is like showing as mission critical probably not.

1:03:07 Yeah. Uh so mission critical to my mind is like these uh kind of the other term of art is tier zero applications the ones that are like just the the core crown jewels of what's running a company you know like a trading system you know your trading system can't go down if the trading system goes down this is kind of a critical problem for the firm that is running the trading system you know banking systems um but also you know like we work with Door Dash you know like some people might think that delivering burrito is a kind of a mission critical system and it certainly is for Door Dash right? You know, if that goes down, it's problematic. We power shopping carts. Um, you know, other stuff like this where it's like, well, if the shopping cart goes down, you know, that company is losing, you know, hundreds of thousands, millions of dollars per hour. So, that that's kind of the critical you think about.

1:03:51 Yeah. I guess of course they're losing, but but this is like when Yeah. their customers are also like they're used to this just working like running water and then when it's not the same thing as when your utility breaks, right? Your water, electricity is out, you'll survive, but it's not what you expected.

1:04:05 Yeah. Yeah. Yeah. And everybody's like what? How how what age are we living in that the electricity goes out? We kind of had this ingrained into our heads at Google. It's like Gmail cannot go down. People are, you know, relying on it. Search cannot go down. You know, if it's go goes down too long, people are going to move to other systems. Um and it's not like, you know, like in some ways search isn't as mission critical except oh wait, every single search is ad dollars behind it. And you actually notice the blip in the revenue. And it's not just the blip in the revenue, it's the blip in reputation as well. I mean, I think that's the one that really poisons companies is like if your bank is down for a serious amount of time, the reputational damage there will be horrifically bad. And you know, like we often talk about like, oh, and then grandma won't be able to pay her rent and she'll get evicted. You got to take this like responsibility really, really seriously.

1:04:48 No, but also like just Gmail being mission critical just on the way here. We only exchanged numbers later, but we were communicating over email like, "Oh, I'm I was telling you that that you were telling me that you're here. I emailed." And I never for a second thought that it it could go down. And I think we were like responding within 30 seconds, right? And it's just I just know it's it's there. Like I I didn't like bother setting up a secondary communication line. So

1:05:11 yeah. Yeah. No, it's like when people just like when you have that kind of level of of trust with your users, you got to maintain it and invest in it. But then it leads to this kind of freedom for the user as well where you just don't think about like I don't have to worry about it. It's just going to work.

1:05:24 And then in terms of the the company like how many engineers do you have roughly?

1:05:28 We have some hundreds. I don't actually know the precise engineering number 110 but there might be 150 in R&D overall. Um there's other folks besides just engineers. I'm clear engineering managers. Those are engineers as well. And then you you know like we've been growing steadily. It takes quite a while to build a distributed database.

1:05:45 Not for the faint of heart.

1:05:47 So it took a couple years a couple times.

1:05:49 Yeah. Yeah. Uh yeah. Well, I did a distributed storage system. Now I did a distributed database. It's not for the faint of heart, right? So there's a lot of work getting it to a a level of stability then a level of kind of quality beyond that level of stability. Um getting all the bugs out um and then continue to innovate and put more performance into the system adding functionality to integrate better within within enterprises. Our revenue has been you know kind of steadily growing over the years and you know it's at this place now where we we see a path to future success as well. And I I want to ask about your coding habits. So when you co-founded the company, how much code did you write for the first few years?

1:06:26 I wrote a lot. So I've always been a very prolific coder, I wrote a lot of code early days and early days, I mean I was uh kind of we were all technical co-founders, Ben Spencer and I. Um and we were all writing a lot of code and I was no exception. But you know like I look back at my GitHub output and you know it's like kind of peak years maybe 100 thousand lines of code in a year which is a year which is a lot.

1:06:50 Yeah.

1:06:51 Yeah. No. So I mean

1:06:53 like we're talking preai

1:06:54 prei right this is when you're back doing this manually right? You know at some point you know we started out using a system called Rox DB which is an LSM. At some point I think it's back in 2019 you ran into limitations with it. I decided um we wanted to rewrite it and did a big push to you write and that might have been like 40 50,000 lines of code and then a bunch of other people come up and helped and like you kind of look at that output I'm just like oh my goodness that was a lot to keep in your head a it's a lot just to type you know 100,000 lines of code the average kind of like uh that the industry talks about is 3,000 lines of code from an engineer in a month and so if you multiply that out maybe 36,000 in a year that's good right so I was doing a lot I kind of look at had. It's like there's kind of a max that you can hold in your head at a time. Um the tools have gotten a lot better since I first entered the industry. We gotten better debugging techniques, better testing techniques, but still um quite significant.

1:07:48 And you were CTO from from from the beginning, co-founder of CTO, but there was a time sometime around like 2022 when you decided to kind of be a bit more hands-off, right?

1:07:58 Yeah. Yeah.

1:07:59 Can you tell me about that?

1:08:00 I mean, the general rule of thumb for uh engineering leaders is well, you got to have your team. at a manager team and we had a VP of engineering. Um, but I was kind of getting to the point of like okay is my are my coding days done you know like kind just direct from a higher level and you know I got this advice for a long period of time and I pushed back on it but you know I kind of acquiesced at some point and I think it was the right advice I'm not saying that the advice was wrong at the time um but there was a time period from about 2022 to 2024 where I was like my my output declined. I think I actually did the Swiss tables thing in that time period, but I wasn't doing much on the the core

1:08:35 the the business.

1:08:36 Yeah, the the core business. You know, I would get in there and do some work, but at like it's really hard that if you're in meetings all day to also do coding. I think this is the fundamental tension if you like.

1:08:46 So you kind of took on the kind of the meeting burden, the coordination burden, the yeah the the stuff that was if I'm reading correctly before you spent a lot of your head in the code and now you're

AI coding agents transform software development productivity

1:08:56 spending a lot of your head like above the code, the business, the engineering, the or the whatever customers that kind of stuff

1:09:02 the the customers and just being an executive as well. Yeah. So very hard to wear all those hats simultaneously. And then I got back into it because AI started to emerge. And

1:09:10 so how when did you get when did you start using AI? But when do you start to find it useful in terms of coding?

1:09:16 Yeah. Well, it's interesting because you know those initial versions of like kind of glorified autocomplete came out and

1:09:23 we're talking about the the GitHub copilot, the cursor

1:09:26 co-pilot. I that was the one I I had the first exposure to. We dabbled with cursor at the time, but they're all like kind of glorified autocomplete and it was kind of crazy that you could just start typing something you like the rest of the function. You look at I'm like wait kind of got that right. This is crazy, right? And um you know we're trying to encourage our engineers to use this and at some point you know I can't remember if this is my idea or my co-founders or someone and basically like you know like in order to like really guide people about how to use it you have to be a user yourself you know I think this is true of like engineering management in general like you want to like manage engineers you have to know how to be an engineer like if you if you don't know how to be a good engineer it's like really hard to manage other engineers

1:10:05 I feel like you'll have a hard time like just relating to them at the very least. Exactly. Exactly. So kind of took it on being like no. I mean this is like it

9 Highlight · AI coding agents enable unprecedented generation speed

1:10:13 was clear very early on like this is probably going to go somewhere but it wasn't quite clear how far how fast it would go and you start dabbling this and I was like oh okay well it's not quite good enough. It's not quite good enough but you know let me start getting back into the coding very rapidly. You know you start seeing the signs of life like you know kind of the opus models coming out. It was Sonnet first, then the Opus, and you're like, you're looking at these and you're like, "Oh, wow." Okay. Well, they seem to be able to do quite a lot, but you know, the code is still not great. But then it was just like the just this cadence of continuing improvements. And, you know, I was starting to do a lot with Sonnet and then starting to use Opus. And, you know, I had that same moment everybody else did. And this was last year, last uh November. Uh,

1:10:58 no. November, December, winter break, right?

1:11:01 Yeah. It was Thanksgiving. I distinctly remember it because you know

1:11:05 you were not chilling, you were coding, weren't you?

1:11:07 You were agenting.

1:11:08 I was uh agenting just like everyone else. I had this thing I'd wanted to do for a long period of time um on cockroach DB which is like I wanted to like test like all these configurations of cockroach DB across like you know different vertical scaling like how many CPUs you have on a node, how many stores you uh discs you have on a node, how how many nodes you have in the system and test across all this huge matrix of it. This is one of these things I couldn't ever quite get prioritized appropriately because it never seemed like kind of critical, but I've like I always had this intuition there was something there. And then over this like four day span, the code just like materialized as I was using um I think that was Opus 47 or is it 45? Whatever the number was like the recollection I had is, you know, like I'm pretty fast typer and then I just remember having this feeling of like well the code is just like materializing before my eyes. you know, you would ask for these things and I got out of the habit of actually typing and you just kind of like ask for something materializes. If you've ever seen like some of those um you know, a movie where they they have this kind of archetype of a a software engineer gets in front of

1:12:10 the keyboard and they start typing, you show the screen, it's just like it's going way beyond human speed and then it was it was beyond that, right? like it was I think as software engineers like we used to laugh at these like I I still remember swartfish the the thing when they're visualizing the things are going or the code appearing you know you're seeing that the person's typing and then it's a big line and a software engineers like we're laughing cuz it's not how it is but but it's crazy that like that effect right

1:12:36 yeah yeah but now now it's even better than that effect right

1:12:39 because it actually works this time

1:12:41 and it's that's slow in comparison it it materializes faster than that like you can literally No, like we've been talking about B trees a whole bunch. I implemented another B tree in in the past month. Of course you did.

1:12:51 Yeah. And it took about 30 minutes to implement probably 10,000 lines of highly optimized Rust. I mean it's just like it boggles the mind. Um I mean I we

1:13:01 you should look up later 10,000 lines is like a crazy amount. You physically cannot type that fast. you you started get back to coding like are we talking about kind of vibe coding prototyping or are we actually talking you started to contribute like proper production ready code that is up the level of what you're doing at DB

1:13:17 well it started with this tool um that was like this kind of benchmarking tool that tested this matrix I I'm the CTO we have an office of the CTO um the office of the CTO's mandate is to innovate and I was looking for places like we can have innovation and one of the things that we want to innovate in um was you know better uh autoscaling of a

10 Highlight · Using AI as an architectural sparring partner

1:13:39 cockroach DB cluster and kind of in the January time frame I came out with like what I would think is like kind of a a research breakthrough you might say and it came about because I was dabbling in this area and just working with models trying to understand it and they're not just good at coding they're also good at like helping you explore design ideas and I think this is kind of the fascinating thing where it's you have to get out of the mindset of like I know exactly what I'm going to build um but more like hey we have this problem talk through it be a partner with mem Mhm.

1:14:05 And

1:14:06 be a sparring partner.

1:14:07 Be a sparring partner. You know, like there's this advice you might have heard that like, you know, if you're stuck on a problem, you just go yellow duck it. Go

1:14:14 rubber duck.

1:14:14 Rubber duck it. Uh I think I've heard is yellow duck. You know, go just go talk to something. You don't even need it to respond. Now you can talk to this system. Uh this intelligence and it will give you back stuff and it's not always right. I mean, this is the thing. Even today, these models are fantastic. Fable's fantastic. Astro is fantastic. And they're not always right, but they they kind of like they know. so much. It's just encyclopedic and you can explore ideas super super fast and then be pointing out well that doesn't sound right to me and it'll be like oh yeah you're absolutely right you know like I I hate that sick fancy it's like it kills me

1:14:49 or you're you're right to push on back on you were right to push it back on that that's absolutely right

1:14:53 yeah and yet like just able to make such

1:14:55 fast progress um and I I I find it absolutely incredible so it quickly moved from just doing kind of side things building up some tools and whatnot to building you know essentially over the last eight months it's January um we've been building towards you know a new launch and you know like a lot of that has been produced gently powered engineers and like the entire company's got on board with this now where I think it's like probably 100% of engineers are using it to a greater or lesser degree and a lot of code is being produced I think it's very high quality code these models not test things adequately they don't look quite intently enough about performance But if you're like uh you can guide them in the right way. And I think there's actually a huge advantage to anybody who's done management before. It's a little bit like being a manager of people where you're a manager of a large group. You're not looking to every line of code, but you're definitely kind of helping architect the system. And I think there's a very strong analogy there.

1:15:51 I sometimes push back on this analogy. The reason being I I was an engineering manager. My take is that working with these agents, it's not like management because management has so much of the human stuff. the like as a manager when I think of all this a bunch of stuff I dealt with which was the people side of things the conflicts between people the performance reviews the meeting etc and you have none of that you do have the orchestration like like and this this is like almost like I guess a really like naive way of management where like you know they don't push back they they start to do it sometimes they're like unreliable but you know I almost like to use like orchestration a bit more because I feel management is is so much more involved like I think this is the thing where like a tech lead who has no management responsibilities but they have a group of interns, but they don't need to do with their performance, with their anything. Like, it's a lot closer to that, if you know what I mean.

1:16:37 Yeah, I I I do agree. I I use the engine manager shorthand, but it's really about being a tech lead for like a 30 or 40 person organization, you know, or being an architect for one. I think the the term architect gives me a little bit of a distaste, but it's being an architect, he's also on the ground,

1:16:52 a hands-on architect.

1:16:53 Hands-on architect. and you don't have any of the management stuff, which is a blessing and curse, but it's kind of remarkable that you can spin these things up. If they make a mistake, you can keep on correcting them until they get it right. We've always been able to do that on the human side, and I can do it faster. I think the ultimate result

Non-engineers rapidly build custom internal applications

1:17:09 for me is you just have to be more ambitious about everything you do. You can produce more higher performance, higher quality, more secure. So, our ambitions have to raise up. You also mentioned that with with AI like you're everyone's using and you're building stuff but also with Cockor DB you're now building something that is also related to AI. Can you talk about that?

1:17:31 Yeah. Yeah. We're building multiple things. I mean AI is the future.

1:17:35 Well, it's here to say I think that's

1:17:36 here to say. Yeah. I mean like uh you know one of our thesis which is uh not crazy. Every application in the future is going to be written by AI. You know I think there will be some kind of bespoke software you know handcrafted software. I think we will see that um continue to exist um just as people write assembly still you know um but it's going to be diminishing in size so you know asmtoically approaching 100% of software will be written by uh AI

1:18:00 generated right

1:18:01 yeah it's hard to say if like when it's going to end where the humans are the agent you know kind of like supplying the agency and the vision behind it I think that might exist for many many years but like I think the code will like fundamentally be written by the AI and I think we're going to just see this explosion of applications and We're we're seeing that inside Cockroach Labs. We're seeing it elsewhere. Earlier this year, we kind of rolled out this um internal platform where non-engineers could write kind of many applications. I mean, this is like you're hearing this at other companies. We did the same thing. And over the course of just a couple months, you know, 500 applications, a thousand applications

1:18:37 by by non-engineers,

1:18:38 by non-engineers, primarily by non-engineers. And I thought it was awesome. Like our HR team is building like these little applications. Like this is the thing I've always dreamed about doing and like they were never serviced. Like like your CFO is all over the place and ours is no exception. Producing dashboards that they can never created before. And I think it's very empowering. I'm married. I have a wife. She needs software. She cannot produce that software on her own. And I have never actually helped her produce the software which is my own failing. But um I think there's a world in the future where she gets like everyone's getting custom software built for them. And you're just going to also see greater and greater applications and higher quality um uh systems being produced by every company as well.

1:19:15 Today, what is your stack? What what do you work with in terms of harness model? How you run agents? Is it one agent? Is

Managing multiple concurrent agent sessions and sub-agents

1:19:22 it multiple? What kind of terminal do you use?

1:19:25 Yeah. Yeah. It's evolved over time. So, you know, like when I first started dabbling AI stuff again, it was, you know, GitHub co-pilot, was an Emacs user for like 20some years. I got convinced to move to VS Code, but that's all gone now. Um, at some point I moved to using cloud code. That was a thing that whatever reason I just got started using it. It was cloud code in the terminal of late. Uh, I use a mixture. So, um, I'll just describe my current setup. I use the cloud desktop app, cloud code via the cloud desktop app. It's fantastic.

1:19:53 Good job, Anthropic. I also sometimes use the codec desktop app just so I I have an alternative model to turn to for some very critical things we're working on. I will get, you know, one of those models sometimes like the best model. oftentimes I'm using the best model like Fable, you know, I've recently started using that. Sometimes it's Astra, sometimes it's the other ones, but you you like you have one produced design, you have the other one being like, "Hey, my colleague produced this. Can you uh kind of tear it up, you know, adversarial review it?" And it's not always perfect, right? You know, but I think there is utility especially for something that's super super critical.

1:20:25 We're pushing towards the launch really soon and kind of rushing towards the finish line. I'm telling folks on the team, it's like, you know, especially our very senior folks like you should use the best model right now. Um it's worthwhile to do that. I'm generally just using the best model. And part of the reason is I don't actually know that I'm getting a lot more intelligence from it, but I don't want to have the cognitive overhead of deciding on a case- by case basis. Yeah.

1:20:46 Should I use Sonnet? Should I use Opus? Should I use Fable? Should I use Soul or Astra? And I think, you know, maybe if you like I really need speed,

1:20:54 I would make that decision. But often times I'm like I'm doing things in parallel. So you're asking how many agents I'm spinning up.

1:21:01 Well, it depends. Like often times there there's like a certain number of sessions you might be using. I find like kind of cognitive overhead is about five to 10 sessions concurrently but sometimes those sessions will have many sub agents doing things.

1:21:15 Yeah.

1:21:15 And they'll be long running.

1:21:16 Yeah. Yeah. It'll depend on what I'm doing. So as I was coming here I was on the train I spun up something to do a little bit of research and that was just probably using three sub agents. Other times it might be 20 sub aents you know with uh Anthropic has dynamic workflows. I can't remember what the codeex thing is called, but they have various ways of spinning up kind of graphs of agents and, you know, prosecuting them. I mean, I I think it's sometimes I'm probably using 100 sub agents, others it's only like five. It depends on what what's happening and sometimes I'm not doing any of that agentic stuff because I'm off thinking about something else.

1:21:46 Since you were an Emacs user for 20 years, you know, using the terminal, right? Or well, Emacs, but how how come you move to graphical interface with agents? A lot a lot of people still use to uh terminal UIs.

1:21:58 Yeah. Yeah. Well, I so I moved from Emacs to VS Code. I was just like a a colleague was like, "Hey, these IDs are really good. You need to try them out." I was like, "Emac is an ID." And then I I kind of moved and like, you know, some of the stuff was just more seamless. And like Emacs keeps on catching up. It always feels a little bit, you know, 6 months to a year behind and like every time they have a new version, my setup would break and I just got tired of that. So, I moved to VS Code. But then I was like, I'd have many terminals open in VS Code. Um, my setup would always be like uh terminals in VS Code tabs. the terminals and tabs, you know, have like eight of them open and then they started taking over. It used to be like just cloud code in the terminals. Yeah.

1:22:35 Yeah. And then, you know, just some point recently is like six weeks ago, a colleague was like, well, have you tried out the cloud desktop app recently? It's really good. I was like, yeah, no, no, I'm very very happy. And I made the switch and it was a little bit awkward at first and then and suddenly I'm like, holy crap, this is awesome.

1:22:49 You you can like manage the agents a bit better, easier. I mean it's like you have the sessions there the sessions are down the side you know and it's like they're kind of like tabs in some regard and it's just but the whole integration that anthropic's been doing and openi is doing the same thing now by the way like I'm not dissing cursor and factory and cognition they're all pushing the same thing it's incredible how fast these systems are innovating right now and

Enforcing rigorous testing and security with AI

1:23:12 evolving what do you think good software engineering looks today compared to like you know four years ago before we had a has it has it changed has change or or that's not really changed.

1:23:22 Yeah, I I I think the ambition has to increase, you know. Um I think the quality has to be higher, security has to be higher. I

1:23:29 I'm glad you mentioned quality. Can we talk about that cuz I I'm seeing across the industry just quality declines, which you cannot fully put your finger on AI, but often times it is people pushing out more and more and just not paying attention to just small regressions here and there. again like you're you're building a database like yeah

1:23:50 have you noticed any or have you gotten feedback of any quality regressions or or if not like how come because like what when like this is just basic you know like we talk about the law of of physics this is an observed thing that when you start to have more output you increase your deployment frequency you often not always but you often have um like more regressions and more bugs if you produce a certain number of lines of you're probably going to have a certain number of defects per line of code. And now you can produce more lines of code, so you probably would have more defects, right? But the thing that pushes against that is that you can be telling these agents, you

11 Highlight · Enforcing advanced testing techniques with AI

1:24:26 have to give them quite a firm hand at this. This is a thing that I I hope that the model providers are listening to, but you need to give them quite a hand. They kind of get a little bit lazy on the testing side, and you have to make sure the tests are comprehensive, but also they're using all the tech testing techniques. And there's a lot of testing techniques out there. we have a lot of knowledge about how to do testing well and you know what the agents are lazy humans are a little bit lazy so getting the humans to actually be very disciplined about their testing is also challenging and I think it's actually easier with the agents you know you can kind of instill that you can get them set up you have to give them the guidance use property based testing use metamorphic testing use like these advanced testing techniques deterministic simulation testing you know there there's like technique after technique you can use and humans have always like I I'm lazy myself Like I'm pretty good about being disciplined about testing. At some point you're just like okay that was enough. We gota ship and like now you can get a little bit like you know a little bit stricter and firmer. I think the same thing applies over on the security side the performance side. I mean we've always

1:25:27 had at cockroach and in the industry we've known security coding practices.

1:25:31 Yep.

1:25:32 You can literally have every single line of code on every commit reviewed by a security expert

1:25:37 doing like an advers adversarial uh security review by an agent or multiple agents

1:25:42 or multiple agents. Right. And you know,

1:25:44 we're going through a tough time in the industry right now with like kind of um hacks and security leaks and whatnot. There's only a limited number of bugs that can be in software. I think we'll get it, you know, added on the security side and on the quality side. And then also just on the p when I I say quality, it's not just bugs, but it's also like, you know, just little things like, oh, that UX element is wrong. And there's no excuse for that right now. Like fixing it is so so easy. And what we're also starting to see evolve is like, you know, designers using Figma. Well, I think that should be a thing of the past. Right now, our designer just deals with HTML and CSS and JavaScript directly and sometimes is even just producing pull requests, you know, PRs, which she loves. We all love everybody's happy with this. There's not any this waterfall.

1:26:26 So, she she's producing pull requests for the production codebase.

1:26:30 Yeah. Yeah. And you know, this is on the UX side, not on the core database,

1:26:34 of course. But this is the area that that like she owns, right?

1:26:36 Yeah. Yeah. It's just I mean, she loves it. Everybody loves it. I mean there's no downside.

1:26:41 This is not new.

1:26:42 Yeah,

1:26:43 for sure.

1:26:43 Yeah. Yeah.

1:26:44 What about code review? What's your take on code review? I think it's pretty it's starting to get a bit controversial like is it going to stay or not? Because it's been a practice that's been around like think about I mean you you worked at Google. Google has been so big on code review as I understand there you there are two code reviews, right? The the domain expert reviews it and then there's a language expert reviews it. So you went through that. How did you think of it and how are you thinking of it now? I used to be part of the I was one of the early members of the C++ coding

1:27:12 review as I'm asking right okay you will have strong opinions tell me.

1:27:16 Yeah. Yeah. I mean I I I see that code review I think was fairly essential before the age of AI but now we're getting to the point where I'm reviewing and looking at code. I I I look at most of the code that the the agents producing still and yet it's like you're not giving it the same level of scrutiny, right? And this has always been the case. If you get a a poll request from a junior engineer, you have to give it more scrutiny than if you get it from your most senior engineer and you ask any, you know, tech lead, any engineering manager, any software, they'll they'll say, "Yeah, yeah, the junior engineer or someone new to the codebase, you know, you just have to give more scrutiny." And what I'm finding as the agents are getting better, you're having to give less and less scrutiny. You do have to give scrutiny to some things. It's like I said, it's like the testing will be incomplete. Maybe they didn't follow the security stuff. Maybe, you know, like there was a performance regression. And I think it's like you know assuring there's kind of constraints on the system so it's hard for them to do the wrong thing. I suspect that you know I don't know if it's going to be this year or next that we're materially going to stop looking at the code in the same way

1:28:16 that we don't look at assembly anymore like you will

1:28:19 we we trust the compiler to produce pretty good assembly. We trust the compiler to produce good assembly and most

1:28:25 unless you're one of those people where it's really important you're and you're maybe a game developer or or someone and you look at it and but there's fewer and fewer of those folks.

1:28:33 Yeah. And even then I think what we'll be migrating too is why why does the human have to look at the assembly? The AI should look at the assembly. like I'm doing something right now where I I want to have a zero overhead abstraction, you know, for doing something that is done at test time and when it's in production is compiled out. Babel set up for me a little system where it actually looks at the decompiled code to verify like that there's only a few extra instructions put in place. I never would have put that in place, but now it has this guardrail that every time it changes this, it can verify that there's no regression. And I I think you're going to see more and more of this where you kind of put guard rails in place. I think the model kind of likes it because it can work within that guard rail.

1:29:13 Now, for what, like 20 plus years, you were writing so much code like I'm sure you were in the zone. Do you remember being in the zone and just turning it out? You wrote wrote a lot of like production ready code now that you're coding with AI like do you get into the zone?

1:29:29 Yeah, absolutely.

1:29:30 And h how is that zone? Is it the same? Is it different?

1:29:34 Feels It definitely feels a little bit different. It's maybe a little bit less intense, but you're managing more things cognitively.

1:29:41 Like, can you describe me like what is it right now when you're in the zone?

1:29:45 Yeah. Well, I'm thinking of ideas that normally would have taken me a a week to experiment with and I think of multiple of these experiments and then I fire them off all simultaneously and then I'm kind of like reviewing like what else should I do while that's that's being complete. Sometimes I'm kind of reviewing like they'll be giving me progress updates of like, oh, hey, this is coming in. We're seeing this stuff. And I'm being like, well, that doesn't sound right. Hey, what about this? Or do we do this, you know, correctly? You know, maybe I had design and it's not implementing the design quite perfectly.

1:30:13 But I have this little feeling that, you know, I haven't been a college professor, but maybe I'm I was a, you know, if I was a college professor and I had a whole swarm of research assistants and they're all got off doing things and it's coming back, but it's coming back just really rapidly.

1:30:25 Really rapidly. You're not waiting months or weeks.

1:30:27 I'm not waiting months or weeks. And then I'm iterating. I'm like, "Oh, that one failed. That's fine." You know, you just got to let go. And this this is the nature of the the software I build. I think there's other pieces where it's like you just whip out a website. You can whip out something that doesn't have this level of kind of scrutiny, you know, very quickly.

1:30:42 But I I talked about this in I have a

Domain experts act as amplified sorcerers

1:30:44 whole bunch of analogies about like the the

1:30:46 the let's talk about analogies. What analogies do you have about about using AI or AI? the the one I was um you know advocating for uh I had kind of two that I was advocating for like late last year and this year which is you know AI is coming for us it's here right and it's like uh you know you're producing software you're like walking down the road and sometimes you know someone will pass you they're running they've got an efficient gate and whatnot but everyone's under their own locomotion and the these agent coding agents came and it was like a car pulled up next to you get into the car you don't how to drive you don't understand the controls but you got to get in start you might forgot the gas pedal and it takes off and crashes into a tree. But you have to learn how to drive, you know. I think using all these coding tools, it just isn't doesn't just happen naturally.

1:31:30 It's learning how to drive. I think we might be in the era of the F1 driver right now, which is like the really good people can drive these systems a lot harder, a lot faster than the people who just picking them up. If you've never used an agent coding tool, there's a vast difference between someone who's like really expert in them, knows where they break, can pay attention to that versus someone who's just picking up for the first time.

1:31:53 One analogy I've heard is we used to talk about 10x engineer. You remember like this this used to be a debate for a very long time. Is it or is is it not? But now what I'm hearing is the 100x engineer.

1:32:04 Yeah. And so you're saying that you are seeing some folks who maybe let's not use the term 100x engineering but like this like F1 driver who is just really really good at it like how would you describe a person who who you've seen this is is it just like rock solid engineering basics and they picked up uh they lean into using these tools or what are they like? Yeah. Yeah. I mean there there is uh quite a bit of a it feels like you know it's directional that if they're good at software engineering before I mean I sometimes think it's like you know everyone's in this kind of spectrum of capability of software engineering and this is just like you know taking that line and spread it out and it's not quite true you know I think it's helped some people more than others but you know feels like it's just stretched it out so you know your ability before is now amplified

1:32:47 I'm always interested to learn that for example Boris Churnney Tibo at uh uh open AI I they both have been really really good software engineers. Boris wrote one of the first Typescript books, the first TypeScript book for O'Reilly. He built some massive systems. Same same with Tibo who built it and now you're kind of seeing oh these people are building all these tools and innovating like yeah they

1:33:08 they were really good before as well.

1:33:10 Yeah. Yeah. Well, I mean this is I mean I have two other analogies to give you about like the what feels like an AI. You you've undoubtedly heard about the term pair programming and pair programming. Yeah, pair programming, you know, and the the idea behind pair programming is it's good to just like, you know, have one keyboard, one module, and two engineers at it, one at the keyboard and the other one sitting beside them kind of like looking over the shoulder and giving guidance. And I think there's an aspect of that feeling where I actually got chat to do a little image of this where, you know, it's like the Android is at the computer typing and you're just there giving instructions. But that was maybe the way it felt like a year ago. I think it feels a little bit different now. Um, the one I I I've just started recently saying is that I feel like the domain experts, the people who were really strong before are now massively amplified. Have you seen all this mathematical stuff coming out like the crazy proofs?

1:33:56 Well, I don't understand it, but I

1:33:57 I don't understand them either. Yeah.

1:33:59 Yeah. So, I told you I wasn't very good at math, you know, it's not complete like I just was like I I stopped in the freshman year of college, right? Yeah. But you know I kind of like wa watch along with these uh advancements and you know Terrence Tao he's like probably the most famous living mathematician you know super genius he actually posted this session there was this recent breakthrough I think it's called the Jacobian conjecture I don't even know what it meant but he posted this chat GBT session where he's interacting I think it was chat GBT and you could see him interacting with this intelligence and it was crazy because he's talking to it as a peer colleague and it's responding and l it honestly looks like I'd encourage everyone to go this up. It looks like, you know, almost a foreign language. It's like his domain expertise is getting amplified by the system he's interacting with. And you can see how he's like kind of learning and exploring ideas just really rapidly. Now, there's all this controversy about AI

12 Highlight · Why domain expertise is required for magical AI results

1:34:49 mathematics, but I feel like the the analogy that comes to mind to me though is uh the domain experts. They're a little bit like sorcerers in their particular domain. You know, you have the earth sorcerers, the earth wizards, the water ones, whatnot. And if you know the the magic incantations, the right words to say the right order, you actually get something kind of magical. And if you don't, you just get sparkle stuff that doesn't have anything there behind it, you know, you you would probably you don't know much about distributed databases. If you're going to ask Fable or Astra, build me a distributed database like Cocker DB, you will get something out, but it'll kind be ultimately hollow inside. If you're an expert in databases and you ask to build a distributed database and you can point out all the various things you

Using AI as an advanced personal tutor

1:35:28 have to know about distributed database, here's what you have to worry about. storage layer, the networking layer, here's the various data structures, runtime inside, you can actually get something quite magical very very rapidly out of it.

1:35:38 So what would your advice be for advice be for engineers with like mid-level to to senior level who you know who have been figuring out how to do coding to become strong engineers in this I guess age of AI? Well, the first off is you have the most amazing tutor kind of readily at hand. And I mean, one of the things that I would always do, you know, throughout my career, and now I've kind of stopped doing it, but it's like the reason why is going to become obvious, which is like I would always look at other people's code. So, I was at, you know, Google early on. You probably heard of Jeff Dean. Jeff Dean was amazing coder. His colleague Sanjay Gowat also just an incredible coder. And you know, I would be looking at their poll requests. I'd be looking at their changes. It wasn't called poll requests at Google. There's a different name for it, but I'd be looking at my CL, right?

1:36:29 Yeah.

1:36:29 CL. Um, this is P4. It's a different version control system. But I'd be looking at their changes, be like, how'd they do what they did, right? Be looking at their code. Oh my goodness, Sanj's code is really always very elegant. You know, Jeff is high performance. How's he doing that? You know, like how's he going about it? It's almost just like you acquire via osmosis. But now what you can do is not just acquire vossmosis, but you can literally like I mean I would be encouraging if you're a junior engineer and you know there's a senior engineer nearby like you could ask them how they're doing what they're doing but you could just ask the AI to dissect what they've done and explain it to you and explain to you at various different levels like how does this code work? What is it doing? Give me a diagram. Explain it to me like I'm five.

1:37:09 Explain it to me like I'm 10. Explain it to me in French. whatever like you want like I mean fundamentally the the to some degree AI is a translation tool they translate it from whatever you know kind of language you're understanding it's in and keep on interrogating it till it gets it you know increases your understanding

1:37:24 you were saying how domain experts are very much amplified I I guess one strategy as a software engineer is like obviously become a great software engineer and use it as a tool you can get a lot faster you can get good at distributed systems like I'm not a distributed databases expert at all but I I use AI to explain a few things for me to understand up front which was very helpful and it would have taken me a lot longer time beforehand. So I can use this

1:37:48 but I wonder if there's another part of like as a software engineer you can use it to become more of a domain expert wherever you're working if it's a payments company. I mean use it to learn about payments as well. So you can help the business you can help your team and honestly you'll just learn more right.

1:38:02 Yeah. Yeah. I I think you know I would encourage everyone you have to have a little curiosity right don't don't be bound in by you know the area you're working on explore outside of it you know I was working on Gmail but I was fascinated about how like the the main Google search engine worked I was fascinated by like how the internals of big table worked even though I wasn't directly working on big table just explore in look at those things and now it's so much easier because you have this super advanced patient intelligence there to explain to it like why do you think it was done this way and then Like I mean once you become an expert you can be integrity and I see it's done that way. What happens if we change this?

1:38:37 Would this be helpful and you know that's where you go from just kind of learning to actually contributing back. I think uh everyone has to have the personal agency to do this. You know if you're just sitting there waiting for someone to educate you on how to do this, it's going to be really hard right now because anyone who's coming in explaining how to use AI or explain how to be a better software engineer, they're going to be out of date, right? You just got to get in there be using the tools all the time yourself and using it like you have to learn. I feel like I've learned more in the past probably even year than the previous five years combined which is weird given

1:39:09 given your trajectory and given the environment you were working in. Right.

1:39:13 Yeah. I mean like everybody's been in the industry for a while. Like I'm a definitely a better coder. I was a better coder 10 years ago than when I first got in the industry. It's like I could look back every decade and realize like I got a lot better and I feel like I just got a lot better over this past year. And and this was also one of the reasons I was really excited to talk to you because when we started to just exchange messages, the first thing you wrote to me when I asked you like, hey, you know, how how are things going? You said like you wouldn't believe, but my coding output is insane and it's high quality and it's database quality level.

1:39:42 And those were things I don't really usually see it. I I usually see, okay, I'm now producing more code, but it's slo. But again, like to me, this is a bit of an inspiration like look like you can use these tools to just like amplify yourself as a software engineer. like you are one example, right?

1:39:56 Yeah. Hopefully one of many.

1:39:58 Yeah. Yeah. No, and we I'm not the only one in Cockroach Labs. We have other people doing this as well. I find it very exciting. You know, it's a little bit exhausting right now, but it's very exciting. Like I got into software engineering um because I like building stuff. You can build stuff faster. You know, the stuff you might have had to compromise on the in the past and you can take away some of those compromises. I mean, you see this in the UX of software coming out. I think the UX is a lot higher. You see all the fancy like web animations and whatnot. That's only just like the surface level. It just extends way way below that. Peter, this was awesome. Thanks for coming on the podcast.

1:40:30 Yeah, this is wonderful. Thanks for having me. One reason I was excited to talk to Peter is because he's been a very high-profile and productive engineer preai building some of the most resilient distributed systems in production. Cockroach DB is known for its resilience and how even if several nodes are destroyed, the database still operates without data loss. Basically, it's as hard to get rid of as cockroaches are. Hence the name. One interesting part of our conversation was how Peter built more efficient data structures than the standard coding libraries had. Thanks to him and colleagues paying attention to parts of the library that seemed slow. He did it for the C++ STL map and in Go for the Swiss table implementation. I found both stories a good reminder that you can improve the existing library or even the language especially if you measure which parts feel slow. Another part of his conversation that I liked was how Peter came a bit of a full circle. He used to write 100,000 lines of code per year being a very productive engineer and CTO. He didn't stop writing code aiming to coach engineers between 2022 and 2024. And then he started to code again because with AI tools he wanted to coach

1:41:31 his engineers better. But it's hard to do if you don't use the tools yourself. And now he finds himself being extremely productive. And this time the team around him is productive as well. And we're not talking about vioded software, but database worthy, highquality code generated and committed to production. Peter is convinced that AI amplifies existing expertise. And this is one reason why he probably learned more this last year building with AI than the previous 5 years combined. And I find it a valuable reminder that learning and building deep expertise in software engineering, this is very valuable. And as closing, I appreciated that Peter said that not only is he excited, but he's also exhausted. There's a lot to learn, but it's tiring and neither him nor anyone I know is immune to this. So, if you're also exhausted with all of the things going on with AI, know that you're not alone. Check the show list for more the pragmatic engineering deep dives on Google's engineering culture and on distributed systems. If you like this episode, please make sure you're subscribed in your podcast player and a special thank you if you leave a rating.

1:42:29 Thanks. And I'll see you in the next