The revised rules of engineering management with Will Larson

25,110 words · about 100 min to read

Early tech stacks and resource management quirks

0:00 One of the worst parts of being a engineering manager in the 2010s is when you joined the ranks of engineering management, one of the first lessons you'd get is like, hey, you need to never make a technical decision again.

0:10 I remember that's literally what I was told. And that's definitely not the ethos today. One of the ideas that I was really frustrated with at the time was this idea that staff engineers principally got to work on interesting, fun problems that they found personally rewarding. Some of my very best engineers were like, I want the more interesting problem. moment is like I only have suffering to offer but this suffering is what's going to make the business succeed.

0:32 People are wondering am I a bit of a career trap.

0:35 It's a hard time in middle management right what I found personally very rewarding is kind of getting back into the details of writing software this is kind of like a unique moment to actually reskill. I do think if you're in management I think right now either you have to like get back into the details or how has injury management changed in the last few years. Will Larson is one of the best people to talk about this. He's been an engineering executive for the last 10 years at four different companies.

1:02 Wrote some of the most widely books on engineering leadership and surplus levels. And I really appreciate how deep and thoughtful of a thinker he is. Today we cover how zero interest rates heavily influenced a decade of management habits in the 2010s. Many of these habits no longer helpful. How the role of engineers is changing and why solving problems that the business cares about is becoming much more important to thrive inside of companies. the path to becoming a CTO today and why line managers often make better engineering executives than middle managers do and many more. If you're an engineer or engineering manager and are wondering what your career path could look like with AI changing the industry and how the path to CTO looks like today, this episode is for you. This episode is presented by Turbuffer, a ridiculously scalable, fast and cheap hybrid search engine built on top of object storage by an engineering team that I've really grown to like after spending time with them. The Turbo Puffer engineering team is doing something really cool. They are completely redesigning their storage architecture from PUR principles to make sure it's faster, cheaper, and more reliable at scale. The new system called Turboper V3 is not yet in prod, but it

2:05 already passes CI. Still, it's not nearly as performant as the current production version v2. And so, the TBuff team is now perf maxing query plans until they are as fast or faster than V2. They're doing all of this in public, documenting how they increased performance all the way up to the V3 production launch. And it's so cool to follow along and see the different performance wins and how they did it. For example, we see how FTEX search the red line started out as 129 times slower than the current system, but now it's only four times slower. And in the development log, we can also see that the team used things like batch scoring or batch reads to increase performance.

2:42 A lot of these wins are basic foundational work right now, but the team will be sharing some very cool systems work over the next few weeks. But hang on, why is the team doing a full rewrite? It's because search is changing and so is turbo buffer. Tapuff started as a vector database, but it's no longer vector DB. It has evolved into high performance database for search. The v3 discards the vector primary index, and all indexes will share the same column or source system. Every query will run on vectorized query engines which is something you don't usually see in search databases but wishful results in both very fast search and just generally a very fast database.

3:17 The Turbuffer team is keeping a worklog of their journey and you can follow along at turbopuffer.com/v3. It's a blast to see how it's being built. Will welcome to the podcast.

Non-traditional paths into software engineering

3:27 Thank you for having me. Excited to be here. Finally chat in person.

3:30 Finally in person because we know each other for for so long. Both started writing. We will talk about that writing as well and also just hung out before but this is nice. I want to jump into the beginning. How did you get interested in computers? Were you interested in computers to start with

3:47 from the time I was five? We had like a Mac 2C or something like that. This was like a not not black and white, not the very earliest one, but it was pretty basic and I think it had like one thing that I understood how to use as a a four and a half or 5year-old which is like kid pics which you maybe I don't did you ever use see kids pics? I I I didn't I know of it, but I never used it.

4:04 It wasn't like a a super sophisticated thing. I don't even know if you could save files, but it was basically like a graphical program for for children and you could make squares and circles and little like graphics. But that was kind of the introduction to me and I remember using that computer a lot and like that this is an era when there really no internet to speak of, really no way to figure out how to do anything other than just banging on it, which is sort of the the fundamental of writing software in some ways is just banging until you figure out how to make it work. So, I think that was a great introduction, but to be honest, I wasn't doing a whole lot in like software other than, you know, multi-user dungeons. Did you ever like play a MUD growing up?

4:42 Multi-user dungeon. What's a no? What's a mod?

4:45 It's a textbased game. And so, basically, the the idea is that World of Warcraft is actually a graphical textbased game that we basically derive from these ideas of MUDs or multi-user dungeons. But when I was a kid in high school, that's when I started doing more software. So a little bit you start with IRK IRC internet relay chat and and Merc uh which I don't even know what the M stood for to be totally honest as a client could do a little bit of scripting with reaxes but then a lot of the MUD clients as as a kid could do a ton of triggers reaxes and that's where I got a little bit further into it learned my reaxes there but really I didn't start writing software at all until I got to college

5:24 and then at college what did you start writing? How how did you get into a more serious level? So I came into college not really knowing what I wanted to study and I thought I might do physics or English or you know economics. My father was professor of economics so there's some some family uh you know lineage to try to live up to but ultimately I kind of fell into computer science at my school which was a small kind of private school in Kentucky.

5:49 There were I think 13 people in the computer science program the year that I graduated. So and that was like the entire program. So I think there were four graduates my year and then like the the juniors and and sophomores in total were like 13 and so it was quite small. So I think there were four that graduated my year but um but yeah but really that was when I got into it. And why I just enjoyed we had an intro course that we required to take and it's kind of funny um with a lot of American colleges especially smaller private ones they have random kind of requirements.

6:20 So I was required to take a introduction to Christianity course like the history of Christianity then also required to take like a introduction to computer science or some other course. I forget what the other alternative was. So I really enjoyed the computer science one.

6:33 Did you decide that you're probably going to be a professional software developer engineer programmer whatever we call it back then.

6:39 So I think one of the interesting things when you talk a lot about careers is like a lot of folks go to like elite schools. So you talk to someone from Waterlue and they're like, "Oh, how did you get your job?" They're like, well, literally everyone at Waterlue is coached on how to go do software. They're literally trading the answers to interviews with each other. So, they know the interviews and drill on the interviews before they go in. They also like amazing program. They've been doing co-ops for years. Like they're great.

7:05 They're great candidates. They tend to do really well. But if you go to to Waterlue, you go to MIT, you go to Stanford, you really there's like this path and you just kind of put your foot on it and you do well and it carries you there. And these are amazing students. Almost

7:19 but it is an escalator if they're good enough to stay in the program. They tend to kind of get there. Going to a small school with again like fourish folks that graduated my year in computer science. There really was no path. I remember one of the uh one of the folks that graduated the year before that was working in software. He went to go work on a horse uh racing betting software company in Lexington, Kentucky, which I I don't think there were more than like five engineers there. And this was like a really talented, really capable person. And this is kind of what they knew how to find. And so for me, it actually was interesting. I wasn't quite sure what I was going to do. I actually ended up teaching English in Japan on the chat program, the Japanese exchange teaching program. And so I was in the middle of nowhere in Japan in the GEU Prefecture for for a year. But that's when I really started writing a lot. So I was like, "Hey, I'm definitely not going to be an English teacher for more than one year. It's a one-year contract.

8:11 I did the one year. I had no thoughts about doing a second year and and I came

Breaking into tech through technical blogging

8:15 back and started trying to figure out how to actually break into Silicon Valley from there.

8:19 And you had the goal break into Silicon Valley already. Why Silicon Valley? Can you take us back to what Silicon Valley was was like back then?

8:27 First, at the time I didn't know what Silicon Valley was, right? I just I just wanted I just wanted a job that that paid well. And I I think you know this was so I graduated in ' 07 and then the the next year I was teaching English and then I actually started working for Yahoo in I guess maybe early '09 something like that. Really interesting time though. So initially I was working for Yahoo from the east coast as a remote contractor. It was an interesting moment. So I started working with Yahoo because I wrote a series of blog posts on something called Yahoo Boss build your own search service. And this was a product that they had launched on this one team. And I think it wasn't getting a lot of kind of traction in the industry. But when it launched, I was writing a lot. So I did three or four tutorials on building boss things in Python. Um, boss things with Django.

9:15 Kind of made a Django, forget the name. It might have been Django Springsteen was like the name of the search library to use Yahoo Boss with with Django. And so I think they got excited because someone was writing about it, which also got me excited. And that ended up ultimately in this contract job. And then ultimately they had the policy where you had to move after a year as a contractor and go full-time. And the reason I was hired as a contractor though is they had a hiring freeze for basically that entire year

9:40 because this was right after a financial crisis.

9:42 And so they're just going through layoff after layoff. And so they could hire a couple contractors, couldn't hire anyone full-time, and then at 12 months they are like, "Hey, now you have to go full-time or or you're not employed because there's kind of the um compliance obligations to to avoid." And so they're like, "Hey, and if you go full-time, you got to move." And so basically, I had a job at Yahoo if I moved or I didn't have a job at all. And so I moved from the East Coast out to California pretty easy at the time.

10:08 Didn't have a lot of obligations other than, you know, employment.

10:12 Yeah. And back then, Yahoo, I guess, their their star was kind of fading with with Google, but they were still a pretty good company, right? And good engineering challenge. Like WhatsApp founders famously came from Yahoo.

10:21 I think Yahoo honestly is a really interesting company, right? And so I think one of the challenges for Yahoo is if you if you came into tech in 2007208 a lot of open source sort of made things easy. Yahoo though is a company that predated all these open source libraries. And so um how do you run a server at Yahoo in 2007208 you run Y Apache. So they had their own fork of Apache with a Y in front of it.

10:47 What programming do language do you run on Y Apache? Well, of course you would run YPHP, which was their fork of PHP. And what UI library would you use? You might know this one, but it's YUI. And then um how do you install things? So, if you think about like um you know, among you, you'd have kind of like a AP or whatnot. Well, you'd use Y inst which was this Pearl script that would navigate kind of the dependencies for installation. And so just like really really fascinating place to come up but every single thing there they made from from scratch even they had like a a precursor to to Kafka which I think was called Yrepple which basically would be like a read log to do replication kind

1 Highlight · Server black markets spawned by rigid allocation

11:28 of outside of a database and that that was like a core tool for managing state just like a fascinating place and I think if I just had like one quick Yahoo story that I loved so David Pho kind of famously approved every server that was allocated in the And so one of the founders like would review. And so the the theory was this would drive accountability, right? You had to prove you needed more servers. What it actually did is it created like the black market of of servers. And so on my team there was like one engineer that had 30 servers from a prior canceled project he never gave back. And so he would basically sell them to you effectively, right? And so if you couldn't get approval

12:08 for like favors and IU and stuff. And so it actually meant they had massive numbers of outdated servers that were like using like inefficient like power wattage because people would never give servers back. And so you had to find like someone who liked you with servers to loan for your projects to launch. Like fascinating how these incentives drive just like wild wild situation.

12:29 Well, I guess that's one thing you you remember for when you later became an engineering leader.

12:34 Yeah, I it was in very instructive, right? And I I'm sure I I'm curious like did David Pho know that people were literally had a black market of servers? Like I assume he probably did know but didn't didn't change the kind of the top

12:46 down mandate.

12:47 Then after Yahoo, is that when you ended up at Dig?

12:51 Yeah.

12:52 At the time very famous company for a very amusing story.

12:56 Can you tell us about like how how you decided to to switch because it was a startup, right? DIG was a startup back then. So depending who you talk to, there were going to be different

The existential threat of algorithm and traffic shifts

13:04 narratives, but I think the the dig narrative is basically it was like the before Reddit, it was the the the big community site, but it was centrally curated. And so Reddit had these sub communities where the communities kind of voted with the algorithm on what they wanted, the the Reddit algorithm. Dick was actually centrally curated on what what went on to the front page. And so it had like a very different experience, but it also meant that the quality was consistent and you had less of kind of the bias and subreddit problem. Instead, you had a bias and central moderation problem. And there were, you know, communities, there were IRC channels where people would basically trade favors trying to get things onto the the dig front page. And in fact um for dig before we changed the algorithm and and I had people saying they had come murder me for stealing their livelihood because their entire you know livelihood was selling access to the front page. The new algorithm didn't support that and they were were literally threatening our lives. And so it was it was a fascinating place but dig is um dig was

14:05 a really important website. And you know, when I joined Yahoo, this wasn't true, but a little bit before I joined Yahoo, literally um the front page of Yahoo was a static file on um a net a net scaler, which was basically like a a filer, which is like a a way to store files. And Yahoo Sports was just like a static page, right? And so dig in that era like early on was like a very important static page. But then it became not a static page like a dynamic of sorts community I think really was extremely important to folks at the time though it's faded into slightly less relevancy over time and it's been restarted a couple of times since but never really picking up the the traction once did. I think it was a top 10 website at one point.

14:48 Yeah, it was top 10 and that's why the story that I I remember is they did a big rewrite. I think it might have been a V4 one, right? And from the outside what I remember is well well dig was doing good. It was popular. There were complaints a bit like with Stack Overflow maybe a few years ago and then Dig did this massive right and disaster followed and it was like a very juicy story. Everyone was talking about it. What can you tell us like what actually happened and what did it look like from the inside cuz you were building this thing.

15:16 Yeah. So I was there I was there for that launch and I was there after that launch and so I joined DIG about 3 to 6 months before that launch went out and it was a fascinating time. So I joined and then um the Friday before I started the CEO got removed and then Kevin Rose came back um in as

15:36 he was founder right?

15:37 Kevin Rose was the founding CEO. He was there for a while and then he kind of stepped back and then he then he came back in but he came back in and we had this you know very hard deadline to to launch dig before which was a rewrite and so dig v3.5 was the prior version before dig 4 basically entirely a PHP software running on distributed MySQL and this was like a different era right so dig was running on hardware that we owned in like a a colo and so it wasn't like just spin up some capacity or whatnot. But DIG v4 was trying to solve a bunch of problems. But first and foremost, um the internet was changing at that time. So this dig at one point a top 10 website was starting to lose traction to Facebook. Um and it was starting to lose um search traffic as well. And so there were a series of large Google changes. And I think we don't hear about Google search algorithm changes that much these days, but back then there was like the Google Panda algorithm change and there were a handful of others. and they were starting to really reshape how traffic

16:38 flowed through. So for a company like Dig, a substantial amount of the traffic, I want to say like 60 to 70% the traffic was actually coming in via search.

16:47 Oh,

16:48 into something called the permal link. And so the permal link pages would be like say you write a great blog post. We that would then get posted to dig and then we'd have the reference to the permal link and also like a small link to your site u of course and comments about it. But we would take a ton of traffic um through these permanent pages that would be higher page rank than the original source themselves. And you can argue like now like maybe we shouldn't have been probably not. I think that would be kind of the perspective today.

17:14 But from the perspective of that moment like we were actually creating the discoverability and the conversation around it. This was before Twitter. This was before Facebook got like substantial distribution. This was really early moment. And so for us like that that was the core of where people came into. Um also it was really where we sold the most ads. And so losing that traffic was basically tanking the company's like financial kind of prospects. And so dig before from the outside DIG was working pretty well. It was you know seemed like maybe it was growing but really like what we knew is like dig was shrinking.

17:50 Um we had one beautiful month of profitability and then an algorithm change hit. Um and we started shrinking and we knew that ex this is existential for us. We were going to simply not exist not that far down the the path and we were when I joined I want to say like 110 120 people something like this but this was this was in you know 2010 when funding rounds were a million dollars or $3 million or something. It's not like today when people are raising, you know, $500 million and you can have a team of of however many or it it was a very different moment. So even with 110 people that was pretty substantial and we were we were running down re um you know the remaining uh coffers pretty pretty fast.

18:34 The the launch internally did it go poorly? Did it go well? Because from the outside again there was a perception that it went terribly and people there was a bit I think a lot of it was the community was upset and and then there was a bit of celebration as well like well they they messed it up like they they kind of deserve that you know the the the mob the the people especially who were very involved they didn't like change first there's like a really fascinating set of cartoons someone made which is like dig versus reddit and like dig is like blue and then like reddit's red and there's this like giant war that someone did like a four or five part it's like this very intricate comic someone wrote about around around this time and it's kind of interesting to see how much energy people put into it. But yeah, there there was a lot of passion and I think it's one of the challenges of like a really passionate community is like when you foster a passionate community then you change what you are.

19:21 Community doesn't like that because I mean there are literally these folks like Mr. baby man was like a really big one like one of the top dig kind of curators uh un unsolicited curators of content getting to the front page and and he you know he and many others were they lost kind of their their business right but the launch went pretty poorly and why did it go poorly it went poorly

The high-stakes gamble of the Dig v4 rewrite

19:43 for a lot of reasons so first the launch technically wasn't really ready and so we made a lot of really aggressive technology decisions and again I came in about 3 months before this went went live so I I don't personally take a ton of responsibility. I also again think about where I was in my career. I was basically was my second job. I'd been at Yahoo for for two years. So I won't own this one. I later in my career I'll own I'll own anything that happened. But this one wasn't really my fault. But basically we made the decision to launch on Cassandra. And there's like a joke at the time which is that Cassandra was a Trojan horse basically had launched to basically ruin competition in the industry because anyone who launched on Cassandra was basically screwed afterwards which I I think certainly was true for for dig and so running Cassandra to replace the sharted my SQL system. that sharted my SQL system was pretty complicated, but it definitely worked. And Cassandra, I would argue like basically didn't work. But I I remember um there was a a team responsible for load testing Cassandra

20:44 before our launch. And two days before the launch, uh this other guy Rich uh who's who was fantastic. He and I were p pulled aside like, "Hey, well, they never did the load testing. We need the two of you to do the load testing, you know, in the next two days." And so Rich was just like a software engineer like me. And so we're like, well, I guess we'll like install JMeter and try to figure out how JMeter works to do some load testing. And we had this like really interesting conversation with uh Chris, who is kind of the the head of the Cassandra initiative. I was like, "Hey, like I can't do the reads on Cassandra." He's like, "That's not true." I was like, "Well, I can't do the rights." He's like, "That's also not

2 Highlight · Surreal launch day chaos during the Dig v4 rewrite

21:17 true." I'm like, "Okay, Chris, if I'm writing to Cassandra, I can't read from Cassandra. If I'm reading from Cassandra, I can't write to Cassandra." He's like, "That is true." It's like, "Well, we're supposed to launch in two days." And then we launched and we didn't fix it. We We just launched. And so I think the company thought that we were literally dying and and the company was the company literally was dying. It was running out of money, but it was it was like the most interesting day I've ever had professionally. Can't remember when I got to the office.

21:43 Probably around 8:00 a.m.

21:44 When you say interesting, you mean stressful.

21:46 You you can let me like paint the scene a little bit. So um it was a launch, but they also decided to have a launch party. So the entire day there were caterers kind of in tuxedos with like sushi trays, champagne.

22:00 Oh wow.

22:01 And so like the entire office was there to have this like kind of like celebration. And then in the center of the room there was this like table this it might have been like a boardroom table or something and pulled out this giant table maybe three or four times the size of this where the engineers were sitting. And we didn't have enough hardware to run the new system and the old system concurrently. And again this was hardware we owned. So they were literally wiping the old servers and installing the new software. So we couldn't roll back. We didn't didn't have the money to or or the and because again we were running out of money but enough budget for the party but like not enough budget for for the servers, right? So we were wiping server by server

22:38 spinning it up

22:39 and ultimately like the damn thing didn't turn on. It was functionally what what what happened, right? And so we had this um you know like down for for maintenance page up for I think roughly 24 hours and then we were able to scrap while the champagne and the tuxedos are coming around. Surreal

22:57 and there were people upstairs getting massages. There were you know I remember certain groups who won't be named playing pingpong while we're trying to get this up and it was it it was just like it was sort of like a magical realism.

23:09 I see what you mean by interesting. This is not how I imagined a launch like this. You can never imagine a launch like this. It was just it was so it was so surreal truly like a magical realism moment where there's kind of all this normaly happening while this like site's literally imploding and it was just it was unlike anything I've seen but it's also like a moment I take a lot of pride

23:30 in. And so I've been there 3 months. I think interesting thing is we did get the site kind of working but there were a lot of issues and so people just gave up and like day by day like people just stopped showing up and like a small group was really the only group left of maybe like six people trying to keep it working and we did and so we in the first two days like the the same guy Rich and I wrote like a caching system from scratch on top of Reddus which helped the site stay on for hours

23:57 y

23:58 but we're still had huge memory leaks and so we're actually like restarting every process every you know like hour or two and the only way to keep it up was rolling restarts of every every

3 Highlight · Cascading memory leaks from Python mutable defaults

24:08 process. And so it took us it took us like a full month to debug and I I have like a blog post I wrote a long time ago about it but basically the core bug was we created an API service that could be used for like third parties and it was in Python. the people writing it didn't know Python very well and this was Python then calling like another Python gRPC that was hitting Cassandra and basically um in Python when you instantiate a default parameter if it's mutable I don't know if this is still true by the way I I hope it's not so true but at the time it was true where if you basically put a list as a default parameter and then you mutated it like appended ids into it it would persist in memory and so each time that function was called would have the same diction or the same list and So the memory of the user ids basically every time they call this API to look up user ids to get the user objects back. All the users that were ever passed into it would still be there. And so it just cascaded over and over and over until we restarted and we installed memcache kind

25:09 of in between these and the memcache would start falling over and then it would you know destroy Cassandra. Um so really finding that bug it took me I think four weeks to actually find it.

25:21 once you found it trivial to to fix, but we got it up and then our new CEO started I think that basically the same day that I I fixed that and then then we you know basically the entire company got laid off over the next year. So

25:36 yeah,

25:36 pure victory in the sense that we did solve it. It it was working but I think you can't really have the central site be hard to use or down for like a month. But like if you go to the counterfactual, what would the company have done differently? If they had simply kept up the old site, they also would have died, right? And so I don't know that they made the wrong choice, but at the time I certainly was pretty pretty salty about it.

Scaling up into hyper-growth engineering environments

25:59 Yeah. How did you eventually get into engineering leadership? I know you and me over overlapped a little bit at Uber. Actually, we both worked there, but we never worked together.

26:07 So the kind of the end of the DIG story was we got aqua hired. And so they sold the patents for DIG did a lot of like innovation. So like in ad or instream ad units basically like you see it on on X now where they'll have like an ad that kind of looks like the the stories from people that are posting like DIG owned the patent of that. They were the the first company to do it. They own like a variety of other patents and so the patent portfolio got acquired by one company. Dig.com like the actual like brand and the website got hired acquired by I think Beta Works out of New York at the time. I think they've sold it subsequently a few years ago and then the team got acquired by social code which is like a third company a subsidiary of the Washington Post company from the time when that was a holding company that own owned the post Kaplan and this company social code which basically did advertising on

4 Highlight · Overcoming the scale barrier in engineering leadership

26:53 Facebook and so the team went there and I was there for a couple of years um but when I when I left I was thinking about okay I I want to be kind of in engineering leadership but I can't get hired into a really large role because at that time I was managing maybe a team of 12 engineers or something like that. I can't get hired into this big role because I haven't done the big role. Like how do we get hired into this big role? And so if you're at a small startup, like you just there's the scale is not there. Even if you go be a head of engineering at this 15 person startup, there's just five engineers there. And so you can never get into the scale they're looking for for these large roles. And so my what I was looking about at the time and my thoughts on this topic changed a lot since then. But this just like honestly where I was. It's like, well, if I want to manage a large team, I got to join a company that's growing really quickly.

27:38 And I kind of looked around and like where is growing really quickly. And like Uber Uber was growing really, really, really quickly.

27:43 15 16

27:44 I think this is 2014.

27:47 This is when growth just started to kick off. Like this was very early.

27:51 This is pretty early.

27:52 You had to be in San Francisco to know that Uber was doing really well. Like outside of SF, I think I think news started to spread in 2015 and then by 2016 it was a global news.

28:02 That sounds right. But the the way I knew is that one of the engineers Jeremy Grosser from DIG who was on infrastructure there left and joined Uber. And actually this there's like a interesting technology adoption curve because DIG made all these technologies. is one of them is called clust which I think dig um Uber and then like uh Second Life uh that that company I forget their their actual name but whatever it is maybe the only three companies in the world that ever adopted this tool clust but it managed basically a key value store to manage kind of ster infrastructure a little bit like a console from from Hashi Corp is very very similar

28:37 just years before console was out

28:39 yeah it was probably made originally in like 2008 2009 something like that and so he he had actually brought like a ton of the dig stack over into Uber because he he was one of the first infrastructure engineers there and so uh you know his team was hiring and I I really was like a a generalist engineer like at at Yahoo I was actually hired to do front-end engineering which I I don't know if I super qualified for um and then at at at Dig I was doing you know frontend Cassandra work backend just like everything

29:09 y

29:10 um and then they hired me to kind of run this S sur team but it wasn't really an S sur team it was just like an infrastructure team that they called SRE. You know, I read like an SR book like the week before trying to figure out what the job was.

29:21 You probably read the SRE book that existed, which is the Google SRE book, right?

29:25 I don't think that book existed, Ben. It it might have.

29:28 What book did you have access?

29:29 I think I think there was a DevOps book might have been what I was reading. Um, and it might have been like the um the Phoenix Project might have been out around then. I'm forgetting if that the exact timeline, but there was something like the Phoenix Project. Maybe there was one other. and I I just showed up and then of course like uh classic interview I I remember doing the traveling salesman problem on a whiteboard in Python stuff stuff like that which didn't really evaluate uh my my DevOps acumen very well but but you somehow I think the company was was a little bit desperate on hiring at that point but but did did get the job and and started.

30:01 And then you were leading a team already you were growing a team. So the team when I joined I probably was managing like six people but it was managing like an incredibly broad range of responsibilities. So Uber when I joined was 2,000 people. It was 200 engineers and this team of about six was managing our databases our Kafka like our actual servers not well the the management of the servers. There was a tech ops team that managed literally racking them and installing them. But once they were racked, we managed that. Our service provisioning like we owned basically everything other than there was like a twoerson developer experience team.

30:39 There was this tech ops team that actually like racked and installed the servers, but we owned all the other infrastructure and it was um it was quite an experience.

30:48 And then in terms of just focusing on your your leadership journey, your your management journey. So at Uber, you started at this team. How far did it go and then what were the next steps between Stripe Carta?

31:00 Yeah, absolutely. So when I joined that team was I think roughly five or six folks. When I left two years later that team was 70 and that was all external hiring. And I think that that was like the most kind of amazing thing about Uber at that point is like Uber doubled the engine headcount every 6 months. And so you can't just like steal people from other teams cuz those teams are hiring a ton too and also really underwater. And so I I really learned how to hire at Uber. I I had no idea how to hire before I joined Uber, but they were just hiring so many folks. I think one of my favorite stories was there's this guy out of Eventbrite that I that I managed for for a year and a half that would give him an offer at like 4:30 p.m. on a Friday and he started Monday at 8. But I I have so many like Uber hiring stories.

31:43 Like some of them like funnier than than others. I remember someone who interviewed with Uber four times and I saw him interviewing a fifth time. and they got rejected four times and the team interviewed him a fifth fifth fifth fifth fifth fifth fifth time I was like we cannot hire this guy we've already interviewed him like four times like stop wasting his time but he he he got an offer for that team and he joined and then he he quit within six hours and took a different offer that because he got an offer from Twitter which was right down right a block a block away

32:10 a block away or somebody was also

32:12 he spent he spent more time interviewing for for Uber than working there despite accepting the job and there are just like so many so many wild stories from the Uber experience um there but that team 6 to 70 over two years and our scope increased a lot in that time but it was just like a a really wild time and it's hard you know some of the companies today I think remind me a lot of that Uber experience but Uber when I joined basically every Friday we were literally manually reallocating software between servers trying to keep the the servers up because typically Friday rush hour was our peak time the of the week though a Saturday you might have more volume but that would be like spread over more hours because people aren't all leaving work at the exact same time.

32:52 So many kind of wild stories of just like what a manic place that that was to work at. But I was there for a couple of years. Then I went to Stripe. I was at Stripe for about four years from 2016 through early 2020. Then I was actually at comm and a half, three years. CTO as that was my first CTO job. It was really kind of deliberate at that point. I think I was really worried. So I'd been at Stripe for four years. You know, Stripe had done really well. I think I joined at 5.5 billion and I think it was at 96 billion four years later.

33:24 Company valuation. Yeah,

33:25 it really came together. But the the also I think the quality of the work we were doing was going really well. And I want to say I was roughly someone that joined the day I joined at Stripe was an employee of 500. I don't know if that was me or or someone else in the batch, but I think by the time I left it was 1.5 to 2,000 folks, something at the company. And so Stripe grew about 70% year-over-year the period that I was there, which is a lot less than Uber, which doubled every 6 months. And obviously like Uber subsequently contracted much more aggressively than than Stripe ever did,

33:55 which is the natural consequence of doubling every every six months.

33:59 Y

33:59 but at that point I was just worried that like my entire kind of like savings were all in one company that was never going to go public, which I think has has been true. And so it was like a bit of a diversification play, but also the desire to what does it actually feel like to fully lead? And so for my my entire time at Stripe, I was basically, you know, one step away from the the CEO. So reporting into Jeff Winter for my first year, then into David Singleton, who was my manager for the the rest of the

34:27 Stripe's CTO later or maybe at that time as well.

34:30 He was hire he joined as the CTO. No, that's not true. He joined as the head of engineering

34:34 and then he became the CTO maybe a year or two later, something like that. But I was reporting to him from when he joined through through the time I departed in early 2020. So I was always one step away. But I was like, what if I could get into that seat? Could I could I do things some better? And that seemed like the the right thing to try for anyway and has been educational ever since.

34:52 And then you have had several CTO gigs then. So it was Calm three and a half year CTO. Then you were at Cardo.

34:59 So I was at COM for I think three and a half years. Then I was at Cardo for two years and then I joined imprint um a year year and a half ago something like that which is where where I am today.

35:09 Okay. So now I three times CTO I need to ask you what does a CTO do as as as an engineer? like let's like if if if you put yourself back as as being an engineer of course I think I myself had aspirations when I was an engineer like oh it could be nice to one day be a CTO because in your mind it's like well it has some management but it's also a lot of CTOs you see are technical and close to the thing how do you find the CTO I know you have a book on on on this as well on engineering uh executive careers and there's a lot of answers but what is your mental model how was it and how is it now especially because now with AI I

The convergence of technical and managerial roles in the CTO seat

5 Highlight · The modern CTO blends technical depth with management

35:44 I think your role has changed.

35:46 I think one of the the worst parts of being a engineering manager in the 2010s is I think like when you joined um the ranks of engineering management like one of the first lessons you'd get is like hey you need to never make a technical decision again. That's what it used to be, right? I I remember

36:00 that is that's like literally, you know, what I was told and I think you'd see people like the most important thing is to let go about technology decisions and this was the ethos then and that's definitely not the ethos today. But I I think the thing that I kind of aspirationally really loved is the CTO was like the only place where this like engineering kind of career, this technical oriented one and this managerial one converged. And so one of the things I've believed strongly in and continue to do today is having both engineering managers but also like individual contributor engineers report directly into me and CTO roles. And so today I have 14 direct reports right now and I think four or five of them are IC's and then the the rest are are managers. But having that balance is like really for many roles in management you're just not allowed to be a technical leader or you weren't for a long time. And so to me, actually getting to the level where you're they converged again and you were allowed to be a leader on the technology and the the management side was like one of the

36:59 big appeals. But also like there just there's so many like bad CTO decisions out there. And I don't I don't mean this to reflect any of the individuals I've talked about. I think David in particular was like really excellent. And the decisions at Stripe were generally decision quality at Stripe was pretty high. And so sometimes you could be like I disagree but you couldn't be like this makes literally no sense. But often like, you know, especially outside of the Stripe experience, there were just decisions that like just didn't make sense and I thought like maybe I could do it better. But I I think the thing that I've really loved in the the most recent kind of era is like you're really like not just allowed to be in the details again, like you're really sort of required to be. And and that to me has been like the the very most fun part of being back in in software in in this era is I think I you know I've done over 500 pull requests in a year and a half at imprint going all the way to production which is substantially more than I did in my prior career combined.

37:59 Um and that's pretty cool and that's

38:01 and then you did imprint as CTO. Are you more than CTO at imprint?

38:05 So I joined as the CTO and right now I'm also kind of leading the product design teams. Right. So product design and engineering all reporting into one person.

38:13 Yeah, it's like a unified EPD structure.

38:15 Is this a new trend? Uh one person because CTPO I started to hear about two years ago chief technology and product officer 2 three years ago and now I'm starting to hear design also going into one person a little bit.

38:28 I don't know if it's a trend or not. I I do think that for one challenge I've been seeing with like the the like agents kind of getting more mainstream is you actually have to be able to make decisions and make them quickly and h how do we do things? How do we not do things? Um how does agent adoption work for product? How does it work for design? Hey design if they aren't using the you know new tools like cloud design or the Figma MCP or whatnot then they're really the bottleneck. How do you kind of split the responsibility there on who owns design now? And so imprint for example has been experimenting with this developer-led design model for kind of second tier features and it's created a ton of friction like it it's solved this bottleneck. It's also created a lot of like who's responsible for these design gaps that show up when engineers do lead design and sometimes like don't follow the design system properly. And so having a unified kind of point that can actually make these trade-offs I think this moment of like immense change I think is pretty powerful. I do think just in my core I believe the industry

39:29 is changing more in terms of how we create software than it ever has in my career and you know I started in you know late 2008 I've never seen a change like this a lot of the changes I've seen were were more subtle or more kind of cultural or or social just how I create software is really different than it was even a year ago and having a centralized kind of mechanism during this moment of immense change I think has been really helpful for us but I I don't personally think it's like a long-term trend But I don't know. I'm trying to avoid predictions about too deep in the future at this point. Everything's changing a lot.

40:01 Will just said how he's never seen the way we build software change this much in his whole career. One thing that is changing is a stack that we use for get hosting which brings us nicely to our seasonal sponsor, Entire. Git hosting has a problem that starts with AI and ends with agents. Engineers like you or me are creating more code with agents. Those agents are pushing even more code and many of us are running parallel agents which results in even more code. GitHub is clearly struggling to keep up and has frequent outages. But there's a solution. Entire was founded by GitHub's last CEO Thomas Dumpka to rebuild Git hosting for the Agent Cara from scratch.

40:36 Entire was built to be very fast and to have your repos regionally close to you to reduce latency allowing for fleets of agents to push in parallel. And Entire's performance is next level. Their repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market. When GitHub is down, you can still keep working and you don't even need to migrate away from GitHub. You just sign up to Entire and the platform mirrors your repo. Also, have you ever wondered what prompt resulted in this code being generated? I find that the prompt and the conversation with the agent carries more information than the PR itself. Entire captures all the prompt history with your agent right in the repo, easy to check back, and has a pretty innovative UI to show all of this. If you're looking for Git hosting that works even when GitHub is down, head to entire.io/pragmatic, install the CLI, and monitor your repo with a click. I've already done it. Oh, and did I mention it works with any agent and it's open source? Oh, and one more thing. Something tells me they're releasing a full gitch force soon from a CI to replacement for the pull request

41:36 and code hosting natively on entire get in early on what could turn out to be the next GitHub for the agent era. I also want to mention our season sponsor antithesis. Agents are writing more and more of the code that we ship to production. We know that. So how can we trust that our system works correctly and without bugs? Code review is an increasingly naive and ineffective approach to do so. Which is why antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes both targeted testing and fuss testing. By running this simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it much easier to fix issues. The first thing I thought when I heard about antithesises is that automated bug discovery and fully deterministic statistics sounds like science fiction, but actually it's hardcore engineering under the hood.

42:31 Jane Streetfly.io and the Etsy community shape agent printed code with full confidence because they know it's been verified by antithesis. To see more case studies and details, head to anticysis.com/pragmatic.

The stark evolution of engineering management since the late 2000s

42:42 And with this, let's get back to Will and what changes he's observed across the industry. But why don't we go back to like from from when you started in the industry? Can we talk about the changes you've seen in software engineering? Let's just focus on your experience. So this to talk about the Silicon Valley companies. They we know they're a bubble, but they're useful bubble to talk about. How did building software change leading teams change over you know like starting from like what like 2007 all the way to today?

43:08 I don't know what your first experience was like kind of as an engineer but when I was first an engineer first as a contractor I was like 22 for Yahoo working remotely. No one else on the team was remote. And then when I went full-time, I had the same manager. So I had the same manager for two years. I had two one-on- ones with that manager over two years.

43:27 I I had no one-on- ones or like once a year or or for performance, not even performance for pay conversations. And then they would say, "Yeah, good job. Here's your raise." Or like, "Oh, and then promotions used to be like, "Congratulations, you're now promoted." And it was more about like, "Yeah, you're making more money. Cool."

43:43 And I didn't have any expectation that I'd have more than than two. And the first one was basically he wanted to talk about another employees performance trying to get like some signal about whether they were doing poorly. So it wasn't like feedback from me or something. Um and the second one was like oh you like gave notice you're leaving like uh what's what's going on right? And so and that was just kind of normal. And so his his job the way he saw it and I think he was like really good at it was his job was to go find an opportunity in the company for the team to have impact. So he was going around the company searching for something we could do that was really valuable and he kind of like brought that back and then our job was to implement it and make it work and that was just like how the work was done and there was no idea that he was like helping me have a rewarding career or something like that that like I think was irrelevant to him.

44:31 We're talking pre200 pretty much. Yep.

44:34 Um and then when I you know got to dig I think all the managers basically got fired. I actually became a manager at DIG because I was the only one dumb enough not to say no when they asked me but but you know I had I had like two and a half three years of experience tops at that point

44:51 and I knew nothing about management right and again I never interacted with my manager at my prior job. I interacted with like a tech lead to kind of set technical direction and then got upset if I screwed something up. But that was kind of the entirety of like the the leadership model there, right? So I came in and I I was a manager. Um but I there was really no management at at DIG other than you know me for the last year and I I didn't know what I was doing whatsoever. So really I learned a lot at Uber, but again at Uber everyone was so busy there was no management there either. It was really a sink or swim and either you figured it out or and your team really struggled because if you didn't get the decisions right the scale of the growth around you would just crush your your team and so again like really really no no support. It was interesting because by 2017 when I became manager Uber put together its first apprentice manager program which actually talking with people. So when I became a manager there, I had like a few days of getting together, mentors assigned, all that. Most engineering managers never ever had that. And Uber,

45:53 it was brand new at Uber. They made it more formal. So they actually did some internal DIY training, honestly. But yes, before that, nothing. Oh, and they did it cuz I think they realized that bad managers were becoming a really big problem.

46:04 Yeah, which makes sense. I think people were just figuring it out. And I think the Uber I worked in was extremely kind of myopic. So one of the kind of like values I might get it wrong was basically solve it yourself. That's like not literally one of the stated values but people didn't go get blocked in other teams they just did it themselves and maybe let builders build might have been like a variant of this.

46:28 Yeah there was the let builders build be an owner not a renter and when you combined the two there was something about principled thing but yeah it was like just just roam your way and don't ask for forgive just get it done.

46:40 Yep. And there was sort of like a no dependencies culture which I found really empowering where it's like oh I'm blocked it's like you're not blocked you need to go figure it out

46:48 and that that was really painful sometimes but it really meant you could always take responsibility and just figure it out your yourself sometimes you know I remember signing like purchase orders for Uber China where we couldn't get anyone to sign it and I ended up signing like a multi-10 million purchase order and I was I was probably like 27 or there was no one to approve it and it had to get done. And so, you know, Amir, the the head of Tuckups and I were just kind of staring at each other and was like, I guess we'll just sign it. And like the thing that I loved about Uber of that era is like no one ever talked to me about it. Like there was never there was never like a follow-up. Probably not good either.

47:25 The search in a corporate governance perspective that's like maybe a little bit like ad hoc, but we really didn't get each other's way and we're able to do like amazing things. Also extremely challenging place to work in its own right.

47:37 Yeah. People left and burned out for a reason. a lot of them

47:40 100% but but really so I all the way through then I finally get to stripe around 2016 and stripe sort of famously

Aligning senior engineers with business needs over personal preferences

47:46 didn't have a management culture either so when I joined there were I think three managers one of them uh who who I really loved working with Railene I think had somewhere between 40 and 50 direct reports um because Stripe had read the uh like the the paper from Google when they used to have basically no managers for some period of time

48:05 and how it didn't work

48:06 initially managed no they initially followed the the model where they there were no managers for for a long time.

48:12 And so there I kind of joined in that the wave of managerization as like one of the the early managers in that wave but I think there were three or four managers Railene Jorge or there were like one or two others. Jorge was managing a team of I think 30 with maybe no one one manager underneath him out of for those 30 folks. Railene I think had no managers. It was just like a very it was a very chaotic time as well. But again, kind of coming out of the Uber experience, I was very very familiar with chaos at that point. But but also the business was growing really really fast. And so I think really that period at Uber though is when the the cultural change around what is the role of manager I think started to like cement itself a little bit and kind of this role of manager not as kind of a core doer or a decision maker but as a facilitator really started to take hold

6 Highlight · Directing senior engineers toward business-critical problems

49:03 and and that was like pretty pretty fascinating to watch. And one of the reasons why I wrote Staff Engineer, which I think came out in 2021, was actually like collecting feedback for some of the staff engineers I'd worked with,

49:16 because one of the one of the ideas that I was really frustrated with at the at the time was this idea that staff engineers principally got to work on interesting, fun problems that they found personally rewarding, which wasn't what I needed for my staff engineers at all. like I needed them to go work on the business problems and the scalability problems that I as like a leader and when I left um Stripe I want to say my team was like 250 260 something like that I had all these problems and so some of my very best engineers were like I want like a more interesting problem and it's like I I only have suffering to offer but this suffering will make is what's going to make the business succeed and so how do I refocus them and I think many of them at the time felt like that was a vi

50:00 violation of the management con uh contract like my goal and my obligation was to give them interesting work they wanted.

50:06 Oh, interesting how how when we look back just how it changed from like 10 years before.

50:11 Totally right. And so I think um that that was like a a fascinating time to to be in but I don't think that perspective was ever really right. If you look at the people who are really senior at Stripe, but also everywhere I've worked those folks were doing the really valuable work that the company needed. you never saw that perspective like really take hold in the executive ranks but a lot of the folks coming in um really had that belief and I'd say like without naming names certain companies that they came from had very strong versions of that where kind of the obligation of the manager for senior hireers was to give them fun work and I think the industry's really um reverted on that one for sure was this something to do with zero interest rates we often come back to this which is so curious I I wrote about this as well how an

Zero Interest Rate Policy shaped a decade of management bloat

50:57 interest rate which feels so abstract and we sometimes hear about it. how it probably impacted funding in tech for a very long time. probably hiring startups, startups that were funded that maybe didn't have a business model but they were still funded and ultimately software engineering archetypes manager types management decisions and then the music stopped sometime around 2022 where efficiency became more important and now we're kind of seeing this and now I think a lot of us are asking and I'm also asking myself the question like everything that we did during that time was it right at the time

51:31 did you ever read I I'm going to get his name wrong but I think it was like Ron John Royy's like uh Zer explains everything blog post from the margins.

51:39 Yes, the margin. We'll also link that. I love his writing. Yes.

51:42 Well, one of my colleagues at DIG, who then I hired at at Uber was John Durk, who's the other writer on on the margins along with uh I think it's Ron John Androy, but um so I I read that piece and it it um it really resonated, right? I do think that Zerp has explained the entire like 2010s in terms of how management styles changed, salaries went up, then competition for hiring became your ability to hire and close and retain became the differentiator between these like successful companies, the less successful because you simply couldn't retain great people. Um, and that like pulled kind of leverage back into the engineers hands and created like a very different environment. And so part of the way companies were pitching themselves was like more relaxed, more focused on you, more career development, and I think in ways that I think were often like kind of self-defeating. So I think there's this kind of like broad judgment of like the Google Perk culture. And I don't think any company that did incredibly well other than maybe Google and and Meta um had a that level of perk orientation.

52:44 But I do think ignoring the perks, there are many other ways that companies try to focus on rewarding work, interesting work. you know, companies like, oh, engineers maybe shouldn't be on call. That's for like the the second tier SRE to be on call. And which was interesting because that was actually a place where every single person I've worked with knows engineers being on call for the work makes the software better. Y

53:06 and yet companies were still kind of like, oh, maybe for the enjoyment of it, we should let folks not be on call for the work they're doing and instead page other people that aren't involved in the creation of the software. And that orientation really reveals I think that the shift in in priorities in those companies.

53:23 Well, I also wonder if that was a time where managers were rewarded for having low attrition on team meaning few people left being able to hire on the team to fill headcount cuz often there was a time this might sound crazy but filling headcount was difficult. You were given let's say you had a team of 30 as a director or senior engine manager. you had 10 headcount this year and if you couldn't fill that meaning you couldn't hire the right people and this was like let's say senior and above so it was like you couldn't just get new grass cuz that would be too easy

53:55 then it kind of counted a little against you

53:58 or someone else would get that headcount which might help their promotion because it was kind of obvious if you had 40 plus people you're now you might be senior manager that's your director title and so managers who did well in this environment became a bit of people pleasers and also they started to shield shield their engineers from the business for example and I wonder if the shielding from the business has bitten back a lot of people because now as engineers you're increasingly expected to be savy about the business understand a business that's when you're going to be in demand that's when in a startup or other places you get to hit hit the ground running and I still hear stories from people who worked at this big company the let's say a big observability company at any of them and they went to startup 7% startup and they say it's terrible like I'm uh they're not letting me do things And uh I'm like, "Hang on, you're a seven person company. Like, how is it?" And you're an experienced person. How is it that you're not asking what does a business need to do? Let's do it. Boom. And they've been socialized in a different place for a while.

54:56 Yeah. I think I think that is like a huge part of it in terms of the incentives. I I do think the orientation around like headcount, filling, team kind of management, like the employee kind of survey results. These are really valuable tools. I I get like a lot out of them, but I remember spending a lot of time at these faster growing companies like analyzing the quality of manager based on the survey results. But survey results are like you have to really interpret them like high scores or low scores don't mean good or bad manager. There's like context to ask other questions about like hey why why is your team all upset about this? That doesn't mean you're a good manager or not, right? And I think these became a little bit a goal of their own rather than an interpretation mechanism which you know in a lot of ways is just like lazy kind of middle management in terms

7 Highlight · Management incentives during the zero interest rate era

55:42 of how these are deployed. But I I definitely definitely saw that and I think in general I think I hate headcount conversations and right I think they're they're almost never the most interesting conversation but I think in a world where managers are not allowed to make technology decisions the only lever they have to actually execute faster is headcount or prioritization. Oh, so are you saying that at that time during ZER where managers were the conventional wisdom was do not make technical decisions this forced sav and smart managers to put way more focus on the non-technical side potentially underinvesting in the technical things maybe just not making decisions that should have been done and overindexing on the soft side or maybe shielding their team from the business or all these things because they put their energy there. Oh, 100 percent. I think that's very true. Like for most like what I tell my teams that I work with now is like the hiring is the least interesting way to solve every problem.

56:39 Like we almost always should never talk about hiring. But the correlary to that is like if you have open headcount, you ask for more headcount. We shouldn't talk about headcount either cuz like what matters then is like the hiring throughput and you actually like can't use more headcount anyway. But yeah, I think we basically told managers they couldn't think about technology solutions and then what do they have? They have like process solutions. they have scheduling or or they have people and so I do think this was the natural consequence of the bifurcation of these two roles which isn't to say we shouldn't bifrocate at all but I do think we need effective leadership to have both or they overinker on kind of hiring and that drives a lot of the bloated orgs we saw in the past let's

Staff engineer progression centers on navigating ambiguity and trade-offs

57:18 talk about careers career development let's start with individual contributors how do you think career opportunities are have changed for software engineers working at startup of midsize companies that you see uh because we we used to have a pretty clear career ladder one two senior staff it was clear on how you get promoted to start with you know AI schools are out you know they're they're writing the code which used to be like nge1 and n2 they're now getting better at like architecture we're talking about loops do we have a career ladder should we have a career ladder if if you are a manager at a company should you be worried about putting together a career ladder right now

57:55 it's a good question right and I do think the expectation is certainly from the teams that we should have career ladders, right? And that is um so the the belief which I I think is like backed up in in research is that like having rubrics drives more fairer decisions than not having rubrics. And so I think this anchoring on career ladders is like something that's here to stay. And I do think if you don't have that then you do undervalue certain things, right? And so maintenance work, glue work, gel work, whatever you want to call it, I think gets like underemphasized and flashy things that actually work get overemphasized. I I think that's that's definitely true. So, I'm still a believer in career lading, right? I think it's still valuable, but what should be there, I think, is like complicated and I think it's like much less clear about what should be it. I think I don't know which consultancy has been selling um some of these career ladders. You kind of small company by small company, you see the exact same career ladder from some like kind of like I don't know like packet you can buy somewhere. And so, I think all of those are just really outdated, but we don't have like the the next version of

58:56 it. That said, when I think about folks getting to like staff or principal these days, like what I really care about is I I've been thinking about doing like an add-on to staff engineer. I'll get to it in like I don't know the next decade sometime, but it's really about

8 Highlight · Staff engineers must navigate ambiguity instead of getting stuck

59:09 decision-m and complicated situations. And so, first this idea of navigating ambiguity. And so at Stripe, I remember working on this project where we wanted to go multi-reion. And I kept getting from the engineers trying to lead it like we need product to decide how um cross region replication works or what like what's the product behavior that we support if there is like a only the consistency and availability but no um partitionability. They they wanted the product team to make these like cap theorem trade-offs which um the product team can't possibly make cap theorem trade-offs, right? And what they were doing is they were finding a way to get stuck when instead like what you need like someone at staff or principal level to do is to navigate an ambiguity by making some trade-offs, building conviction, figuring out how to like make a prototype that like tests like whether the the predict the assumptions they made are are valuable and and workable or not. And so I think these ambiguous problems are still a really challenging one and to me like a strong

1:00:09 marker even today. Even with agents, you still have these like complicated trade-offs that you have to make. And some of the trade-offs to your point about kind of understanding the business earlier do come from not just understanding like how the technology works, but the product implications. And to understand product implications, you have to understand what your users are actually trying to do. You have to understand the unit economics of like the product that you're offering and the cost to to service it and to actually what you're charging. And so increasingly I think you have to think be quite broad to actually work these ambiguous problems. But I do think decision-m has always been where staff and principal engineers have to to make the the impact. And that hasn't changed at all. I don't actually think agents are exceptionally good at making these trade-offs. I think the agents often don't even recognize the trade-offs because they're not in the context window. kind of subtle implicit things that live kind of outside of the the software. What about NGE12

Rubrics cannot replace managerial judgment in career development

1:01:02 careers? I I just had a senior engineer ask me saying so you know like AI clearly has gotten as good as an angel engine writing code because that's what we expected a lot of times from them and then this person was saying that well some point it'll come to senior but it's not there right now but I still remember this one cuz obviously there's truth to it but I in your or how do you think of Nge12 new grad or or people who are under the senior they don't yet have the battle scars or the expertise or the judgment that you can fully trust them both in terms of hiring but also developing them.

1:01:34 So one of the things I try to stay away from is I call like artificial like specificity and so like what's the difference between an edge one and nge two and you know I think it's like actually pretty hard to be like super concrete about it. I think a lot of companies successfully say the difference is like three years of work experience and they say is that is that equitable and like I don't know if there's any like super equitable way to evaluate nge ones versus nge two and again a really bad nge one you can identify but there was like a really good nge one and nge2 often is time and role in terms of just like building more kind of context over time but it's not like it's not like super like they can do this or can't do this especially with you know the the coding harnesses like they do anything but actually deciding whether they should be doing it increasingly increasingly tricky and in the wrong situation we're getting a lot of direction you can't even perceive their judgment in a lot of cases right so I I actually like care less and less about some of these things where I think just general things like time and level

1:02:36 stuff like that is probably good enough for a lot of these I do worry that getting incredibly detailed on these criteria is like a bit of a like a fake like victory where it's like oh it's made it really precise but often I you've been in like calibration meetings where there people who play to the rubric. I remember at one company I worked at we had to get to like staff there was like a mentoring requirement and so we'd have people that literally went out and like would talk to someone um and then be like report them as like a mentee and they were like collecting mentees and then you talk to that person like I didn't know why I was talking to them. they just talked to me once and and so like this is where if you actually create like the you know like you you get what you measure. So I I really think for promotions there's always been a lot of judgment. Whenever you try to remove the judgment I actually don't think you get good outcomes there either. And so having the rubric to help guide people I think that's useful. But ultimately it's like pretty subjective. And for a lot of these things I think it actually having

1:03:37 kind of constraints is more interesting than having really comprehensive rubrics that are used to make judgments. So here here's an example. So something I did at Carta. Well at Carta there's a lot of cost pressure. cost pressure meaning that like to reduce costs of the let's say or import manage costs exactly

1:03:55 and so I actually had like a ratio can't have more than this number of staff engineers relative to the number and there just you was not possible to exceed that ratio that was very annoying to a lot of folks right but I think it actually is more honest where the industry was like 15 years ago 15 years ago I think there wasn't a role past senior in most companies like senior really was the ultimate role for almost every company. And so, you know, as people have gotten like further along in their careers and we have more people with like 20, 30, 40 years of software experience, they have opened up more of these levels. But like a lot of these levels are like ultimately more like time and roll kind of titles rather than like really like a definitive like obvious like change in the capabilities, the individuals that are that are doing it. I try not to worry quite as much about like the details. And I do think having like a hard block of like one or one to eight or whatever it is is really

1:04:55 helpful if like what you're trying to do is manage costs because when you're managing costs, what you have to be honest about is that you're managing costs. And so at at Carter, that was kind of the situation we're in. Now at Imprint, like we're not trying to manage costs, we're trying to manage velocity and and quality level. And so we we haven't thought about that at all. But this is where I think um part of leadership is knowing what you're doing, explaining how you're doing it, and then letting people like accept the consequences of that. And so because of Carta, we were going to have like a a fixed ratio. That meant we had to start shifting um hiring to be less senior. So we could actually have more of a pipeline of folks coming in and filling those levels. From that fix, a lot of things came from it and which worked which worked really well. But it wasn't if we had just tried to fix it without figuring out the the other parts of the hiring earlier in the funnel wouldn't have worked. But I I don't know. I'm less of a believer in kind of like a a perfect description of things that are true than I was um earlier in my career.

1:05:52 Just so many exceptions to everything. The thing with like the rubrics and and with expectations like it used to be you know when both of us worked at Uber you later worked at stripe it was pretty

Software engineering is shifting toward business constraints and rapid experimentation

1:06:01 well understood what was expected of the role and you know with the levels like yes I I agree that one two senior these can be murky but there was a pretty clear expectation at some point for example a senior you could trust them typically I think the ind industry started to understand at some point for like closer to a decade a senior could trust them with a project if they had junior engineers they would mentor them a staff you could trust them with uh to set the direction of a unit or multiple projects and so on and and the nge one and n2s they could execute well coding without too many issues took feedback they they were kind of like almost like they were like apprentice levels like I think nons2 it wasn't a serious promotion in terms of it was just it wasn't a destination it was just they're growing some companies even had the expectation as you were that you need to grow to certain level but but they were like almost like apprentice honestly like they were just kind of getting getting to that that to have good judgment and all these Now that we're all using AI harnesses a lot more, including for reviews, code reviews are slowly going away as well.

1:07:01 We're doing a less of the the what we used to do, which is coding and the very visible feedback between each other. How would you define what the role of a software engineer is? And is it different from before? Is it more ambitious? Is it how do you think about it? Because it is it is changing. like I I I feel it's getting more ambitious somehow, but I'm not sure I'm able to put my finger on exactly how, especially from a CTO perspective.

1:07:26 And is this like the general role or is this like the level three or

1:07:29 No, this is the general software engineering role. How how because I think I feel each role has has changed a lot. Maybe the staff role the staff engineers never did too much writing code at like midsize and above companies because they were busy making sure we're building the right thing. So maybe it's hard to separate because because now no one is writing too much code and even the smallest rules they will have they will need to decide are we doing the right thing am I doing it right first I think we're so like early days in terms of where we're going and there's certainly people you're going to talk to they're like hey like the the software teams of the future are five people for any size company and maybe maybe that's true maybe we'll get there at some point we're definitely not we're definitely not there right now and I

1:08:09 I don't know if we're going to get there soon

1:08:11 but there certainly are companies that are you have a small number of engineers spending just a staggering number of tokens in productive whis and that's like a pretty a pretty cool thing to to see where is the role of software going I think we're going to find out and this is actually why it's such a fun time to be in the industry right now right the the most fun thing is like I think our profession is changing in a much more extreme way than anything I've seen since I entered the the industry and for me like someone who's always been very oriented around the outcomes of the work I'm doing. It's just been like the most fun time I've I've had. But I've also spoken to folks who got basically joy from, you know, being in the zone, being in that flow state. And they like have fallen out of love with software because what they loved was like the actual art of doing it. And for them, like you know, a few folks like one person I've spoken with recently, he he actually he's retiring. He's like, I just don't enjoy writing software anymore.

1:09:08 And he's lost this flow state. he's like I'm more productive but like they just don't enjoy it and so I think the role is shifting but I think it's shifting in a really interesting way and I to me I've had this kind of idea of software that the time to actually implement is approaching zero these systems are still very slow in terms of like you know okay let's say you can instantly implement stuff you still have to make all these decisions it still doesn't work in all these ways it's still it's going to scale poorly in all these ways still going to break the user expectation still going to have like the wrong user unit economics to actually be like a profitable business. And so I do think even if software were like literally free to to write, it's still really really really hard. And so like today um you'll see this like if software's free then all of a sudden engineers are running ahead of the PRDS from product and running ahead from the the high fidelity mocks from design and that's creating a ton of issues for design teams trying to work with this engineering model and something we're still trying to figure out at imprint to

1:10:09 totally honestly right that also could create opportunities where like well if if an engineer also puts on a design hat or a or a business domain hat and now and now like runs an experiment experiment full on to and obviously now you get to like there's only so many experiments you can run without confusing you customers which now it goes back to strategy someone having to say no and and also like who are you building with like if it's for people like they will have the neat number of attention like Netflix has I don't know how many new movies they bring out but I know that when I open Netflix they don't overload me I mean they do but they they they have they have they don't show me all 8,000 movies they they figure it out and so I wonder if If software is becoming faster like as engineers we will need to discover the constraints of the other parts of the business the domain the the customers the the reality like you say unit economics how how money can be made

1:11:04 I think something I thought about a lot

Complex domain businesses retain defensibility despite AI tools

1:11:06 working at Carta so Carta is a lowcale business by which I mean there's not like hundreds of thousands of requests per second coming in there's no like massive infrastructure problems there but it's an incredibly detailed business. How cap tables work is really, really, really specific. Yeah.

1:11:24 And it's very complicated and there are tons of exceptions. And then doing

1:11:28 it's important, right? It's a little thing, but you need to get that right.

1:11:31 Accuracy is really important from like an audibility perspective. Um, you're also working with law firms that have like a real strong point of view about how everything should be done very particularly as well. And you could be in a legal battle between two founders who are arguing about how much equity was actually vested or something like that. Um, similarly, the other businesses Carter went into like the fund administration also incredibly detail- oriented. And so there are a lot of businesses out there where the domain context has always been the hard thing.

1:12:00 And it's not super clear to me if businesses where the domain context is really the hard part have benefited that much yet from kind of all the AI tools. It's still incredibly hard to do cap table management because of the the details, the the laws, the relationships with the law firms in terms of training like the parallegals there. There's like all these little pieces that I think domain like complex domain businesses still have like a lot of defensibility in in this day and age. And I don't know if that's how that's going to change. I think it will change. Every two to three years there's this kind of like the product engineers are the new king.

1:12:35 Product engineers are the new superhero kind of like meme that comes around. Now it's the AI engineer.

1:12:40 Now it's the AI engineer who really are like product engineers that use like AI techniques, right? And so the question is

1:12:45 which is pretty much every product engineer these days.

1:12:47 100%. Right. So it's like it's unclear. I mean I've never really quite understood what AI engineer means. I think it means someone who uses harnesses and wants to get paid a lot of of compensation is I think what it means. But it probably means something else too that I I've just sort of missed the the nuances on. But like can you actually get individuals to hold all this context in their head? And so even today, you know, people used to talk about like the front-end engineer who's actually a good designer or the designer who could actually write production front end code. That person's always been rare. And now a designer with like a little bit of taste can write production code, right? And and vice versa, a front engineer with like good design taste can do do design. Why aren't we seeing more of that yet? And so I I think that's like a a social constraint.

1:13:30 Well, it's a really good question because I was asking the same thing. I saw some data from linear showing that PRs for designers and product managers are up and I was like oh actually some someone messaged this to me uh a fellow engineering manager said like hang on I've been asking around like we should be seeing more designers push to production right or product managers asked around 10 companies I asked around another 10 including the likes of linear ramp openai I have yet to hear except for like some exotic edge cases or like I Just one example of designers pushing production code. They will find bugs.

1:14:06 They will identify them, but they will still get an engineer involved who confirms this this approach works in the codebase. Especially when we're talking about outside of prototyping. They do prototype more. But even we're not there. It's I wonder if it's an accountability thing. It's on, you know, like when uh even though like I I can change a light bulb, I will still call the electrician when I need to do something a bit more on my house because it's electricity, right? And and I can read the books. I I wonder if it's this or specialization. I don't know.

1:14:38 You know, one of the things I've been thinking a lot about is like there's there's one worldview right now which is like if you have a great design system and you've actually used that design system, you don't need product designers anymore. That's like a viewpoint that a few people have. I haven't found that to be true though. Um, even with a good design system, understanding how all these different components fit together, understanding the missing components in your design system, figuring out um, when you have like trade-offs across different screens, how you manage those trade-offs while presenting like the unified experience for the user, these are still really hard things to do. And so, you know, I I know attention is all you need is actually not about human attention a at all. But I do think there's some dimension where like people are still constrained on attention and great designers and great product designers and great design system designers as well are all so constrained on like the details of what they're doing to do it well that they just don't have a ton of time to also be branching out. It does that inherently have to be true? I I don't think it's inherently has to be true. And I do think in theory

1:15:39 if you have a great design system and you have a ton of consistency and a ton of llinters in terms of like it has to be used this way all of a sudden your design system folks your product designers could be just looking at the exceptions much more aggressively. That is where we're trying to get to at at imprint for what it's worth where the the base case is just done by engineering using the the the stock design system and product design is more working on the really complicated kind of versions of it along with extending the design system of the design system folks. It's not working that way yet though like it's still it's still a little bit tricky but you know as as like the folks deep in AI say like I kind of do assume this is like a quote unquote skill issue and that we will like get get to the other side of it.

1:16:20 But I it's still a skill issue for for us. We're still figuring that out.

1:16:23 One thing related to this is you you wrote uh revised rules of engineering

Durable high-ownership teams outperform reactive headcount surges

1:16:27 management. A really nice article. We'll link it in the show notes below. In one of the rules that I want to highlight, you say we wrote durable high ownership teams. I'm quoting you. Durable high ownership teams. So domain context are even more important. One of my biggest lessons at Uber was that persistent durable teams work magic by accumulating domain context building a sense of camaraderie and feeling increasingly strong sense of ownership over an area as they continue to work on it. Now this was from Uber like what like 2014 15 so like more than 10 years ago and you're saying that you feel this is very relevant right

1:16:59 I think this is the only way to build scalable organizations and it's interesting though I think this is this seems kind of like obvious but it's actually a bit of like a a non-conensus take if you will and so a lot of times you're at a company say there's 50 engineers and you have a really important project for a partner that project's late there's still the mythical man month thing they're like hey we're moving 15 more people to work on this key project. And you know, the mythical man month I I I want to say was written in like 80 84

1:17:27 7575 and the second version 95

1:17:30 and it's that that's a while ago. Um and so

1:17:33 more than 50 years

1:17:34 we we know this doesn't work and so like first like when people do things like this a lot of it's like optics and creating the perception of doing everything possible.

1:17:42 Yes.

1:17:42 So that when you miss it's not it's not

9 Highlight · Why durable pods beat throwing headcount at crises

1:17:44 your fault. But actually, not only does does it not work, it also destroys the thing that does work, which is creating little pods that are able to own areas. And so when I joined Stripe, I had, you know, I took over kind of the team Jorge was leading at the time and he moved on to a different part of the company, I think, to to lead connect. And so I had 30 people and one manager was managing like four or five of them. And then I was directly managing say like 23 24 people. Um, the first thing I did was like, all right, what are some pods? um and figured out like some areas I could put together who could own areas and then once I saw some progress I just stopped thinking about them entirely till something went really wrong so I could focus on the bigger priorities um for for me which is like the problems you couldn't solve this works so well and so at at Uber did the same thing when I came into imprint did the same thing as well same um at at comm card was like a really well kind of scaled organization by the time I joined it so I didn't didn't need to do the same

1:18:44 there. I think this just works incredibly well. I think if you do this well that people actually do better work than you will from top down dictation simply because they understand it really well and it gives you the ability to actually put your attention on the highest leverage or most painful parts of the company. And so yeah, I think this is a strange non-conensus take, but it just works incredibly well, but optically it almost always looks wrong.

1:19:09 And that's like one of the reasons it's hard to do where you have like a partner launch and you're like, "Hey, I'm not going to I'm not going to move these folks. I'm going to let them stay on what they're doing. It's really important. Instead, the team that's on the the critical path, we're going to work hard as hell together." But like, we're going to burn to get through it. We're not going to pull everyone over cuz if you don't do that

1:19:31 or or what what you might have done in the past is especially in Zer where it was you had meetings, recruitment, all that. You would say like for this team on a critical path, we remove them from all of the BS. Totally. The recruitment, the university trip, you don't need to attend all hands and all these things. So, you can be hesitant. Which actually, by the way, works so much better than adding people.

1:19:52 And and also sometimes telling people like, look, this will be a twoe push. Anyone who can commit to it, please do so. uh including putting extra hours and in return you can take time off afterwards or or you know like you you create incentives and and you just treat them as adults and you're saying like can you and of often times people will like I know in my life outside of work sometimes you are okay taking a thing for this one thing especially when you

1:20:17 know that I will repay you in kind somehow and if you don't do that the thing that happens is actually at a at a real growing company there's a crisis every Yeah.

1:20:27 Um, and that means if you rotate people to crisis, there's a crisis for everyone every day and it creates this sense of like devastation. And so I remember talking with someone at Stripe one or two years in. We're talking about this and he's like the the way Stripe had run up to that point. His metaphor was like it was like people on a wall trying to defend a castle and they were always running to the place where they're about to get like overwhelmed but they just couldn't be everywhere because they they were getting faster and faster running but they still we're not everywhere all at once. And what we were trying to do was switch to this metaphor was like we had many people many places and so it the people working in one spot had to solve that spot but it meant they didn't have to be everywhere. Those are kind of the two trade-offs. But to be a successfully scaling company that's growing quickly, you do have to get to a mode where it's not the same people running everywhere at once because those people just completely burn out over time. And you there's a certain point where they just can't manage the coverage anymore no matter how amazing they are. Although I think the the question with AI is is that point much

1:21:28 further than it was before. But I I'm not sure if it is. I still think their attention runs out before their their

Executive leadership requires pushing through unpopular decisions and managing incentives

1:21:34 ability to write code does. Also recently something that I was thinking about this with the durable high ownership teams is just an example with Slack on how you have Google Google chat you have mattermost you have Microsoft teams and slack let's say these are four big competitors for the chat market and it's pretty well understood was there and oh and you have AI uh your your startup you could also in instruct an AI to like copy Slack and here's the interesting thing you know if if you do it the AI would copy Slack's UI and or behind open source things. First of all, there's a lot of decisions that have never been documented that are behind the scenes that people at Slack will know and they take it takes years which if you build it, you'll figure it out.

1:22:13 If you're big enough from the concurrency, deliverability platforms, yada yada yada. But taking that aside, Slack launched Slack agents, which was a super sweet integration with coding agents and that do the work inside of Slack. It just works beautifully. It's been well thought out. Clearly like this is a team that has a domain context and they've clearly been working behind the scenes with teams partners. I'm kind of like hang on why has none of the other competitors launched it. Why has Google chat who have amazing people as well?

1:22:44 These are all people that have the domain and I think this is where I'm starting to this one example just starting to realize like there is probably value in like Slack having all these people who are working. I'm sure there's we would be horrified of how many people work there. I I know it's from Uber like why does Uber have this many people but they're working on this but they're they're not only able to operate what they have but they are they keep reinventing what customers will want as the world changes you know the other companies seem to have not been responded even though agents have been here for a while they just haven't responded Slack has not only responded they did something that is objectively good and now it's copyable but how did you get there right

1:23:21 there there's some paper about the the size of the bureaucracy of managing like the the British kind of empire and the size of bureaucracy, the number of bureaucrats kept growing even as the territory getting managed kept shrinking and there's sort of this inherent kind of thing where companies do just expand and create like bureaucracy and create like more kind of overhead. So there's something there where like sometimes companies have just kind of like lost the plot on their ability to actually move. But like my assumption if you I think this is true for the the muse work on for meta. My impression is that when really interesting good products come out there's one or two highly empowered people that are connected to the the the founder, the CEO, the executive team that are pushing it through and that are able to make painful trade-offs that are not wellliked. And those can be like legal versus like company trade-offs.

1:24:11 Those can be product trade-offs. those can be there are you know the the sacred kind of elements of how things have to be done not doing it that way and so like my my guess is like for for Slack

10 Highlight · Pushing through unpopular tool migrations

1:24:23 there was some sponsor and look people hate overemphasizing executives but I do think executive sponsorship to actually empower folks to do something difficult Imprint moved over to linear you know like three or four months ago from Jira we did it in like less than than four weeks and there were basically zero people that thought it was a good idea when I decided decided to do it. We still did it in less than four weeks and now people are really glad we did. It solved a lot of problems we had about having centralized issue management so the agents could actually know where tasks were, pull them down, work them, actually do kind of task unification, etc. We're having a really hard time managing workloads of tasks without that. But there were like maybe three people out of like 230 that thought it was a good idea before I did it. And that one I just, you know, pushed through. And I think now people are like, "Oh, that was a good idea, but they just didn't think it was." And sometimes you do have to have someone who's willing to take the risk. And then, you know, the flip side is you then need to judge execs and other senior leaders on are the risks they

1:25:24 made us take good.

1:25:26 And so I I think 3 months into the linear migration, I think, you know, no one talks about Jira anymore, but certainly no one's upset about the change. People are really glad with it. But conversely, if if people had hated it, like they they should be like, you

Executives require both technical depth and bureaucratic skill

1:25:39 know, rallying to get me fired, right? But I think a lot of times people don't get held account to this. And so that's the the flip side of that highly empowered executive sponsor that does make these great products. Sometimes we let them do incredibly boneheaded things and don't hold them to account,

1:25:53 which is great segue to talk about executives. You've been a CTO, which is a former executive. Can you tell me what in your interpretation executive is and how what is the career path today to get to that level from an engineer like okay like I like the idea of executive I like that to have be able to make decisions including unpopular ones helping people helping companies what was the old path and what's the new path if there's any

1:26:21 so I think the old path you're you're a software engineer for some number of years could be pretty short then you get into management and then you climb through middle management and then you get to the CTO role management, edge manager, senior director.

1:26:34 Exactly. And so I think there are a lot

11 Highlight · Executives need both technical depth and bureaucratic skill

1:26:36 of problems with that. First, it means that often the people that were selected for management initially weren't like the most focused on the technical pieces of the work because they're trying to get promoted to management and then the selection bias is really towards people that are able to operate well, which doesn't necessarily have a whole lot in common with technology side. Often they happen to be really good technology, but there wasn't like a strong kind of selection bias for it. But you really do have to be a good bureaucrat to be an executive. And this is like one of the interesting challenges where a lot of my work is incentive management. It's knowing, hey, if we roll out this sort of performance review, here's how it's going to fail. And so we just shouldn't do that. We should do this other thing instead. And so if you haven't done the reps of being like a bureaucrat, you actually like can't run a function because a lot of running a function is in fact being a bureaucrat.

1:27:20 So how did you learn you learned to be a good bureaucrat?

1:27:23 I I was a bureaucrat, right? And so I mean out of necessity. So at at Uber, there weren't like really like rubrics. There wasn't like an interview. We didn't have any structure how we interviewed when I was there. So you had to make it all up yourself for your function. And again, we were doing tons of interviews and each time we didn't give an offer out, I was like so pained because I was doing so many interviews. So I had to figure that out. Um and then at Stripe, Stripe was like a much more structurally cohesive company and so we did have like a lot of structured process and then I got to work on a lot of that and learned a lot there.

1:27:54 You kind of got pulled into it. you picked up the pace

1:27:57 100% and a lot of the structure I thought wasn't working well at the point that I joined to kind of like work to try to iterate and improve little pieces of it because we were you know growing growing pretty quickly and so really you know working these middle management jobs I think did educate me on how to be a good bureaucrat which is part of being an effective executive because you can just allocate processes that work and avoid processes that don't work think about incentives that work and don't work structure that does work and doesn't work you know the jeopard seeing like a state like you're not allowed to be in Silicon Valley if you haven't read seeing as like like a state but it's basically um a book about how states see citizens and accomplishments and how that shapes how policy works and it's the same thing like if you've never been a bureaucrat you don't know how to see things as a bureaucrat but also like if you can only be a bureaucrat you don't know how anything actually works don't know how software actually gets made and so it's like very damaging to you

1:28:50 or am I getting this right that you're saying to be a good engineering executive So VP of engineering or CTO, you want to be excellent at technology, knowing how it works, being hands-on if you need to be or just be hands-on and also be a good bureaucrat where you can manage and you want both. And you're saying that if you go the middle management route, often times it it might happen that you either never were a good technologist or you stop becoming that by the time you get up there. So you're a great B crab but not technology. But if you just go from like an engineer, great technologist straight up here. Let's say like you're a startup tomorrow. Congrats, you're a deitio, you will not be a good bureaucrat.

1:29:26 I think that's exactly right. And it's it's interesting where often times I think line managers are better kind of instinctual executives than kind of middle managers because they're used to working the detail.

1:29:37 So they can make better calls for the team or the company.

1:29:40 Their judgment's just better on all these like non-bureaucratic dimensions of the job, but then their judgment's terrible on kind of the bureaucratic dimensions of the job. And so this is why I think being an executive is hard and why it's a hard career path is that it was you know in the 2010s in particular you had to climb these like layers of middle management to learn this kind of bureaucratic component of it and that was the only way to get scale to actually understand that. Now those layers are kind of gone but the interesting thing is like the other half

1:30:06 the job is now really emphasized in terms of better decision-m understanding the actual product understanding the actual business understanding how software shipped in a way where we kind of deemphasize that. So I think you know people are like is it easier? Is it harder? I don't think it's like any easier or any harder. It was it's a lot harder to become like a bureaucrat now which again like negative connotations right but this is like learning how to actually operate a function is really difficult work and as like a C level like you'll often end up running three or four functions often one you don't know much about yourself. So, for example, at imprint, I was running the customer operations team for, you know, much of the last year, which is not something I'm incredibly detailed on. I have like some exposure to and had like an amazing leader in place, but I I've had to learn how to do that and I've had to do that, you know, a number of times.

1:30:56 Um, at at Stripe, I was managing support engineering, which is not something I've done personally myself. I was running data engineering, which is not something I've personally done myself. And so, you know, you learn how to run that way when you have to. It's really hard to get the jobs to learn how to like operate a function that way where you don't actually know the details of the work, but you get to be a much better more detail- oriented leader for longer. And that's that's really where I think the

Middle managers face a stark choice to reskill or step back

1:31:20 best future executives are are line managers because middle management has historically selected for totally different set of skills. How do you get both? It's incredibly hard now and it was it was always hard in different ways. And what's your take on middle management slowly dying? Do you agree? It

1:31:36 it's definitely a hard time to be a middle manager right now, especially if you came up in the 2010s. You were kind of selected and trained in this more of this empowerment model, more of this like functional like bureaucracy model. And now people are like, we don't want it and we don't need it. And so you've kind of risen to this pinnacle, but you can't get to the next step because of these other skills that are more kind of in the details execution really deemphasized and kind of people have changed the the rubric on you, which is like super super frustrating for folks who kind of get trapped by that. That's definitely the the challenge that folks are in now. And I I don't know I don't know how this wave of execs are going to happen, but it's always it's always been a hard role to get. Well, and I I do want to ask you, two people asked me for advice, and I'm I'll share both of their stories, and you can decide if they're similar or different. One is a head of engineering who has about seven people reporting to them at a company and was thinking, hm, I wonder about my future because I'm seeing about this middle management. The other one was a story from another head of engineering who has

1:32:38 three EM reporting to them and three IC's reporting to the VP of engineering. they've been shifting to AI transformation and in the end department also doing similar things. uh the VP of engineering is satisfied with their work but they're under the VP of engineering and this person also wondering like hm both people are wondering am I in a bit of a career trap and they asked me to ask you well you you have a lot of experience what tactics would you be able to suggest to them to figure out what career path could be for them to figure out what they might want what is possible and again I think the tricky thing in both cases both people have spent a bunch of time in middle management. It's not that they're not technical, but you know, they they had this path and and it was pretty clear where they would go next would be VP of engineering for both of them, but in their current company will not happen or unlikely. So,

1:33:32 it's a hard time in middle management,

12 Highlight · Reskilling by stepping back into line management

1:33:34 right? There's a lot more competition for these roles. And so, I think what I found personally very rewarding is kind of getting back into the details of writing software. And so when I left Carta my team was 250 maybe something like that. When I joined Imprint the team I was leading was I think 25. And so for me this is kind of like a unique moment to actually reskill. Well if you're in management I think right now either you have to like get back into the details or the world is changing so radically that like you the profession you know will not be valued in three years. And I think you sort of have to decide like am I retiring now or am I reskilling? And I I thought about that too, like I, you know, a card I like I could just I could just be done, right?

1:34:17 Is this is this is what I want, but it's not what I want, right? I I really want to still be participating. And I I really genuinely believe the way that we work has radically changed in the last year. It hasn't changed everywhere. There's like a long tale that's going to take a long time to catch up, but in the places I work, it's changed a tremendous amount. And for me personally, it's a lot more fun. So first I think there has to be some like openness to like taking a backwards step or a lateral step. Just what people want is very different and you have to be open to like getting a different like look at it. And so I think going back into like a line management position for example is actually much more educational than a middle management. The tools for line management and writing software are so good. The tools for headcount planning or doing performance reviews are are basically terrible, right? like all LLM applications for these things are not good. So, I do think you're missing a lot in middle management right now, I would I would try to find my way out of

1:35:18 it. But thinking a little bit about some folks that I hired at at Imprint. One of them that comes to mind is Jasmine, who I worked with at Carta came over and she was uh she she was hired as a VP, but she was a VP that was hiring or managing a team directly,

1:35:32 but she was a line manager.

1:35:33 She was a line manager, a really experienced line manager. was running engineering at a Mux some some time ago and she came in to to line manage and she's doing just a tremendous amount of of software which she would have never done at Carta because she could have never gotten time to do it and at MX like she was also like way too busy like running the function to actually be in the details of writing. Those people that are willing to actually go back and get their hands onto it are so valuable.

1:36:00 And if that is who you are, you have to let people know cuz otherwise people would think you would never take that opportunity. But I would hire more

1:36:07 people like this all all day long. You know, we were talking about in the 2010s uh on the reserve how engineers got got a little bit too used to like, oh yeah, my my company and manager is here to do what I would like to do. I would like to do platform. No, no products are even if the business needs it. H just just not for me, you know. I'd rather quit. And sometimes they quit like they did. I wonder if there's a little bit of this with management where like look again I don't mean to imply too many things for these folks but generally some I have been I have this path I I thought I will be VP of engineering soon and I I want to get there but is that what the industry wants? Is that what the other companies want? No. Right now they want really good let's say line managers.

1:36:46 Yep. And either similar to those engineers, the best engineers I know, they kind of put their ego on the back and when the business said like, "Look, we need product to build." They're like, "Look, I'm great at platform. You know what? Let me be the best product engineer I can be." I felt with you as well that you, you know, you had 250 people, I assume, you know, compensation, equity, status, and you went down to lesserk known startup with not as a well-known brand yet or at the time. uh taking a lot smaller team, smaller responsibility at like you let the ego at the door if you you know what I mean and then you learned so much more.

1:37:21 Yeah. I think in in a lot of ways it was just like a much better situation for me and so like there are a lot of pieces that weren't like kind of oh like this is a hit that's not really how I thought about it. It was the perfect situation for me cuz what I knew what I wanted was getting back into the details relearning how to do software in the current age. That's what I believe is going to be the unlock to kind of the longevity of my career. Also though, it's just like much more fun. And so like to me, it's just like much more energizing. And I I've always been someone who kind of got tricked into management. Going back to the big story

1:37:55 where they just needed a manager and so I I said yes.

1:37:58 You've been tricked ever since.

1:37:59 Um and then they made a director so it looked like they had a director and acquisition conversations. So they they was more palatable. But I didn't know how to line manage, let alone direct. I just they kind of needed one on paper. Um so you know uh very kind of them though. So for for me this has been kind of what I wanted and this has been like a unique opportunity but I think for a lot of folks there is this sense the industry has betrayed them and that they were owed something that they've worked for. And the thing is like they're right like they have worked incredibly hard but the rules have changed. And so I think that at certain moments the world looks very stable and our industry looks very stable. And I think in the 2010s we forgot that our industry is actually very volatile. Our industry's changed just a staggering amount in the last every decade things are just radically different than they were in the decade before. And we've sort of like lost sight of that. And I think that people that kind of came up in an era of a lot of stability don't remember that

1:38:59 sometimes. And so I I do think we have to be willing to kind of take lateral moves. We have to be willing to to be a little bit humble. If but if if doing the actual work that's like thing that matters to the company is humility, then we've sort of like lost lost track of what we're doing here because we really do just get paid to do things that are valuable to the companies, right? Like they don't there's no like entertainment orientation to why these companies employ us.

1:39:24 Yeah. What about the TLM role, tech lead manager? This was a role that we declared dead. You and me both declared

The tech lead manager role remains a high-demand trap

1:39:31 dead as a as a dead end. It's is there but this was back in you know like the the tech leads who gave who were given a very small team to manage as well and we we were clear like in in Zer that it's just a career rut and if your experience is great I think you you also made it clear that it's a trap. I think you have that article. What do you think about this role now? It's funny. It's it's still a trap, but it's a highly demanded trap. And so, and I think in many cases there there's no way to actually there's no alternative. So, kind of it's it's less a trap than simply what's there.

1:40:01 And so, I think in a lot of companies, management is getting flattened.

1:40:05 So, we're starting to see like a director, let's say, have a bunch of like tech tech leads and like a or of like, you know, 30 people and a couple of tech leads here and there.

1:40:12 Yep. I think you're either seeing kind of two models. You're seeing that there are engine managers, but they're managing much larger scopes. And so they're actually like less kind of hands-on the individuals than they were before because they're managing say like 12 13 14 people before instead of like you know like four to four to eight. And then we're also seeing kind of directors are then managing tech lead managers instead of managing kind of these line managers. And different companies are trying different things. But I think this is where I see the industry kind of asking for folks right now. And I do say like some of the best folks we've hired at at Imprint have been folks that were in like more pure management roles that came in but really wanted to be writing software and have just been doing an amazing job of kind of like doing a blend of both. It is hard to find those folks where it's almost like

1:40:59 interview them to be a great EM and then like do do you just get frustrated you're not writing more software? It's really that person is kind of what what you want. But it it's hard, you know, not how we've been interviewing them so far. I think we we've just found a few great folks, but that that is the role that we really want. And again, it kind of goes back to this thing where you want people who have the capability to do the bureaucratic parts, but don't want to like the the capability but but not the desire because those are really

Authentic writing builds lasting industry leverage

1:41:25 the folks that I see super effective these days. They can use that when they need to, but they default to actually writing like software, which is really where most of the value is.

1:41:34 As a closing topic, I I wanted to ask you about how you write. You've been such a prolific writer. You Still, I I I I really enjoy your blog. Whenever there's a new article, I I check it out. There's also uh you can subscribe. There's an email that I subscribe to and I actually get it in my inbox. How you approach writing and what you think about AI and also what is your take on there's a lot of AI writing going on, not just for social media on LinkedIn, we know that, but also for some authors who before AI, they didn't write with AI, but now they're starting to a little bit. So, I've been writing for about 20 years um on online in my blog and I founded Irrational Exuberance in 2007 and then, you know, in next year I actually might have been like May or June or something when I started writing it. So, I'm getting close to this 20-y year kind of anniversary.

1:42:22 And the when I first started writing, I was simply trying to figure out how to like network my way into a job basically. But I did a ton of tutorials just like here's how I did this and this was a different era. now the sort of thing that you just ask you know an AI like how how do I do X and it would write a great tutorial just just for you or maybe just do it for you directly right then as I started working at Uber um I started writing a little bit more of like the things I've been learning in hypers scale and then it really took off when I was writing at Stripe because I think Stripe was a place that both had a lot to learn but a real like love of writing and that to me like really energized my work and then um wrote my first book kind of the culmination of the things I had written and stripe an elegant puzzle and then got a lot of momentum. think a total of four books.

1:43:07 Probably not writing another book for at least a decade. M maybe not ever. But I think it's just been it's been so transformational for for me. I think like so much of like when people know me, which is not that very often really, but like when they people know me, it's exclusively from the books that I've written or or or the blog. And I really do believe writing is like unique way that we as individuals can actually change the shape of the industry. And I think when you've done that, there's like little things that kind of pop up.

1:43:37 So for you, like the the trrimodal kind of salary piece, right? And that is first like a memorable name that that helps, but also like I think was like a really important observation that a lot of people didn't understand. I think you've you've changed a lot of people in how they approach compensation. And then Patrick McKenzie has his work on negotiation. so impactful, probably created so much wealth for people in the world.

1:44:03 I think he might try to track how much he's he's created on it, but I'm not I'm not sure if that's still true. But for for my books, I I really think I've changed the the industry um with um and I think staff engineer I think in notably I think like the staff plus kind of concept that that that book created staff plus as like a concept, right? And there's other things that I think are like less like quotable but have that mattered. And there's just like there's nothing I could have done in my work in any company that could have impacted the industry in this way.

1:44:31 This many people and and so so positively or I I remember the both the

13 Highlight · Why AI writing slop destroys two decades of leverage

1:44:36 elegant puzzle and sap engineer. I read it and I took I took a lot of guidance from it

1:44:40 and that's just so powerful. And then I think about um writing with AI and like I've spent tens of thousands of hours over 20 years creating this writing this stuff. I'm not like super craft oriented in my writing by which I mean I really trying to get ideas out more than I'm trying to write like the best blog post. I've never been craft oriented in terms of writing great blog posts and we're just trying to get the ideas out written somewhere.

1:45:05 I think I like what I like about your writing is there are very punchy ideas. It's it's not about the writing per se. Like you're you're I would say you're a decent writer or it's enjoyable, but it's not it's not like I don't like it because it's enjoyable to read. The idea comes and is there and it it it makes me thinking and I'm not thinking because of how you presented it. And again, the presentation is fine, but the idea is what brings me home.

1:45:29 Yeah, it's pretty minimal writing and that

1:45:31 and some of your articles are short. Yeah,

1:45:33 it's not even about the length. I'm way more wordy in my writing than than you are, which you will know, but it's just my natural style. But in yours, it it doesn't have to be like what I like about yours, you don't waste a breath in your writing when there's not much. And that and that's complete.

1:45:50 And and then I think about that um all the stuff I put together, all the things I've tried to change people's minds on. And to me, like writing with AI would just be like throwing that away. And so I think, you know, I like I've never done a newsletter trying to collect like a huge number of folks. It's more if you want to read what I'm writing, this is what it is. It's not like my business like it like it is for for for you. But still like, you know, I've gathered like some number of folks and like they I think they read what I write and that's really powerful. And I I know I know if I write something like often there'll be hundreds of companies talking about the idea there and the next week and that's just such like a privileged spot to be in. If I just wrote flop, I would just throw it away and it it it's I could never get it back. Like I'll never I'll never spend another 20 years doing this, right? And so when I think about the advantages I have that I've kind of carefully built over like two decades, why would I throw that away? And so if I don't have something to say, I just don't say something. I'm not trying to

1:46:51 manufacture content or something like that. I think it would be such like a devastating moment to me to throw that all away for nothing. Like if I was throwing it away for something, I could maybe get there, but I think it would just be like an uncompensated loss if I were to write AI slop articles. Like I don't think anyone would read them and then I would throw away this this thing that I've spent so long putting together. And and I assume that you feel

Prioritize deep ideas over high-volume content mills

1:47:12 the same way. Is that true?

1:47:14 Yeah. Well, I I I feel that to me writing is how I think and when I write the drafts I often like a lot of my social media activity actually and I never talked about this before but sometimes people ask me what is your social media strategy I have a large number of followers which is just a byproduct of doing things consistently and in my case honestly it's researching and thinking about like I I almost do full-time or close to full-time thinking about stuff that's happening in the industry and that I'm talking with engineers usually what happens I start writing a draft about something and I had this idea. I put it in a in a tweet just like in another window and I keep writing it and I later look back it's now got it's gotten traction because I guess it was an interesting idea.

1:47:57 Uh sometimes it's controversial. Uh sometimes I actually test some up. I'm not quite sure. I'll just put it out there. I call these a little bit like brain farts which is not entirely true cuz I am thinking about things but I don't really care about them and I I'm actually more interested in the conversation. So sometimes it then distracts me when I go back but it then makes my ideas form and and a lot of my ideas are about observations. Is this happening? Are you seeing this? Which just generates conversation. So, and to me that is that is the writing like and what I found with AI there was a time where it was popular to like feed all your writing and have a bot that you can talk to and some people wrote a pragmatic engineer bot which I I didn't like the idea to use but anyway I still tried it out and it was almost hilarious because for example it would my opinions change all the time my views change and it was kind of spit back something that I said you know like years back and I I know that we talked about TLM if if you had a Will Larson bought it would say TLM is a trap and now we're talking about like well

1:48:57 actually there's way more to it. I I I I tell people like don't put your thinking whenever I see someone write on LinkedIn especially a fellow professional or they message me I'm they often comment with their real voice and I'm like why don't you just write that imperfect that's you I I I like the you and I don't like the sterile thing and I also don't like how with AI it's got the same voice for millions of people using it whereas millions of people talking have millions of different voices. So, this connects to something I think is really related to this, which I think is um a lot of people believe to be successful they need to be writing content, but they don't really want to be writing content.

1:49:35 And so, they're like, "Ah, I really hate writing content, but to get my promoted, to get my next job, I need to be writing content. Like, how do I do that?" And it's like, obviously AI, right? And I think something I've been trying to push back on folks on for for a while. I wrote a piece. I forget the exact name, but how to be like a a tech influencer or something like that.

1:49:53 That we're going to link that that post. It's such a like not catchy, but almost a clickbaity name. It is the single best post about how to be influential in tech. And I I will link it. It's one of

14 Highlight · Prioritizing high-impact ideas over content volume

1:50:05 my favorite articles. I I forward to so many people. So, by the way, thank you for writing that.

1:50:10 Of course. And and the reason I wrote it was like these people were like, "Oh, I I need to write 200 things, 300 things. I need to write every week. I need to do And it's like, well, if your goal is to be visible, what you need to do is write one thing that gets traction. Spend a lot of time on it. Test it. Like, figure that out. You don't actually need to write a ton of stuff. There's almost no value in writing a ton of stuff, unless you are, you know, running a business that is based on a subscription. If you're just trying to get your ideas out, if you're just trying to be a little bit more visible, if you're trying to show proof of your ability to write like cogently, you can write like two things. You can write three things.

1:50:44 You can write one thing every two years. And you don't have to write a lot. I think about Tanya O'Reilly who has not written a lot on her blog, but when she writes something, it says something really interesting, right? And so she's not like a volume writer, but she's been able to have really impactful pieces. I think No Glue is like the most obvious one that have really been transformative. And so I think even for people that do write a lot, people don't remember the vast majority of they write. There's like a few pieces that stick out. So I really think that if if people reoriented around the real goal, which is maybe to have some visibility, one or two good things matters so much more than volume. And so I I wish people would stop creating volume if they don't actually I think it doesn't serve their goals. And in fact, like I I've seen some candidates that I've interviewed, you know, over the last year where they have just like AI slop content and like a huge volume of it. And I I don't know.

1:51:41 I'm pretty conflicted on those candidates cuz it's like hey people aren't reading it. You keep doing it and it's sort of like why are you continuing to do something that doesn't serve any purpose cuz it no one's responding to it. So it just feels like people who are like running like a recipe but aren't tasting the product and aren't it's it's cargo culting, right? They don't

Mental models and systems thinking drive modern engineering

1:52:00 actually know why they're doing it.

1:52:02 And so to me it actually implies like poor judgment rather than showing like a sign of like oh this is an ambitious person trying hard. Yeah. N clothing. What's a book that you would recommend or books that you enjoyed reading?

1:52:16 Certainly, I think uh I would recommend kind of my my books to to anyone who enjoys spending money on on things.

1:52:21 Your books are all excellent. Really?

1:52:24 Thank you. And and certainly I think crafting engineering strategy came out last year and I think is you know the the newest one but I think there's like a lot of interesting ideas in there. Um for for sure I think you know thinking in systems of primer by Don Meadows is like one that I think about a lot. read it based on your recommendation.

1:52:40 I I think it's it's honestly the book that kind of formed my view about management and kind of operating in and companies. I think that it's not really about operating companies, but it's like a mental model you can you can use.

1:52:53 That book I think is incredibly incredibly helpful. But but really like any book that forces you to have like a different perspective on something. And so I spent, you know, a decent amount of time thinking about typing a couple years ago, like not not typing like on a keyboard, but how to have like better type systems, which is not that relevant to to my work on a day-to-day basis. And then there's another great book whose name I'm losing on TLA plus and using that.

1:53:17 Is that Hello Wayne?

1:53:18 Yeah, exactly.

1:53:19 Yeah, he was on the podcast. Yeah, he he has the famous TLA plus.

1:53:22 Fantastic book. Um, and and look, like I'm not using a lot of TLA plus on a day-to-day basis, but I think just the ideas there are incredibly powerful in terms of using languages to increase the verification or verifiability of your software, which is one of the biggest challenges of our time, right? If all of our software is getting written by someone else, which is an agent, making verification cheap is is really critical. And even if TLA plus is not the tool for that in many cases, thinking about that problem and this is the same reason why typing is really interesting, really, really valuable.

1:53:52 And so I think right now those those are some some useful ones.

1:53:56 Thank you, Will. It was awesome to have you and finally have this super nice conversation.

1:54:00 Oh, likewise. Thank you for having me. I've known Will for so many years and it was great to sit down in person and we did it at a time when there's a lot of change happening with traditional best practices when it comes both to engineering management as well as engineering career progression. One really interesting topic was how far managers used to bend backwards to find interesting projects for their engineers. Will recalled how at Stripe some of his best engineers asked him for more interesting problems. Back then he already pushed back saying that the only thing he has is suffering to offer, meaning not particularly glamorous problems, but that suffering is what's going to make the business succeed. But back in the 2010s and the early 2020s, lots of engineers saw this approach as not particularly great management behavior.

1:54:43 And yet today it's kind of a default for career stability and promotions. As an engineer, especially at the senior plus levels, you will want to help the business solve their most pressing problems. And this includes having empathy for those problems. I also found it eye opening how Will thinks about what it takes to be great CTO. He thinks of it as you need to be two things. one, a great standout technologist who knows what to build and how to build, as well as a bureaucrat who can run a whole function, including a function you might not be an expert on, like product or design. In the 2010s, lots of managers learned mostly about the bureaucracy part. And by the time they became CTO's, they lost the technology side. Today, with smaller teams and less middle management, many CTOs will be technologist first, which Will thinks it might be better overall, and I have to agree with it.

1:55:30 Finally, I appreciated Will's advice to middle managers. Middle managers being angry managers who have teams that under managers report to them. Will did not beat around the bush. Yes, today it's a hard time to be a middle management. The two paths to seize looking forward are either you go back to being closer to where software is being built, work on a smaller team or as a line manager role, or just accept that your expertise of mostly being a Barack but not a technologist could not be valued that much in a few years time. Will himself took a backwards when he left the 250 person engineering team at Carta to set up a 25 person team at imprint but he did it consciously to learn how to build with AI changing the engineering profession. I do find it inspiring that will not only talks about the importance of learning the new rules of software engineering but also walks the walk. Do check out the show notes on related the pragmatic engineer deep dives on zero interest rates and engineering and EM careers. If you enjoy the podcast do subscribe in your favorite podcast player and a special thank you if you also leave a rating on the show.

1:56:25 Thanks. And I'll see you in the next