Build the Right Thing: Product Engineering (Part 1) — Kent C. Dodds, EpicProduct.engineer

10,819 words · about 43 min to read

0:13 Okay. Hey everybody. We have like the line Stephen. Hey. Uh, we have a line that goes like all the way over here and a bottleneck right here that we're all engineers. We should be able to figure out how to solve this problem. No, just kidding. I I don't want to step on anybody's toes. Um, but if somebody wants to figure out what app he's using and help him get them through, that would be sweet. My name is Kent Seed Dods and we're just going to get started because we have two hours. I think we actually have plenty of time. Um, but uh I I don't want to have you all just waiting around. So, as people are filing in, be friendly and if you've got a seat next to you, like wave them over and let them in because I I am not certain that we're going to be able to fit everybody who is in line in this room. And so, we do want to get every seat filled up so people um are able to make it in here.

1:04 Um I am just thrilled and honestly I don't want to say I'm shocked uh because that reveals some uh level of um uncertainty on my part but um so we won't say shocked but I'm thrilled um that uh you all are here uh to talk about building the right thing um product engineering before we get into that to give people a little bit of time to come in uh I want by a raise of hands I want to understand who uh I'm talking to so how many of you would consider yourself at your company, you are like individual contributor. You're um maybe not writing the code anymore. Who's doing that? But you're like directing the agents to write code. All right. I kind of figured most of you would be that. What if you were somebody who leads those people? And you can be both.

1:48 Okay. Awesome. What if you were only somebody who leads those people? Okay. Okay. We got a couple managers. Uh all right. Uh what about like product owners? Anybody consider themselves product owner? Okay. Sweet. Uh any CEOs in here? Who's a CEO? I was just going to be like, "Hey, that's cool for being a chief." Um, uh, all right. Sweet. And how many who has traveled the farthest? I talked to a group of guys over here traveled from India. Anybody travel farther from India? Where did you go?

2:18 Melbourne.

2:19 Oh, Melbourne. Nice. That is one continent I have yet to set foot in. Would like to one day? Um, Melbourne is nice. My sister actually lived uh in Melbourne for a while. She liked it. Big bugs. Uh, anyone else come from really far away? Uh, yeah, over here.

2:36 Germany.

2:37 Germany. Nice. Uh, I, uh, I was going to be in Germany next week, but I'm in a play with my kids and so I had to cancel. Uh, Finding Neverland. I'm not a main character. That's for next play, maybe. Uh, anybody else from far away? Hey. Yeah.

2:55 Brazil. Oh, whoa, dude. What do you Oh, no. The game just ended. Who won?

2:59 Brazil. Oh wow. Okay. Good for you guys. That that was an interesting game. 1-1. That was uh for a long time. Very cool. How exciting. Congratulations, Brazil. Uh okay, let's Dang, this stinks that um that everybody's not in. But I'm It's so exciting that so many people are interested in learning what uh how to build the right thing. But let's get right into it. Here is um my clicker not working. Let me plug it in.

3:30 Okay. Um, oh, there's an on switch on this clicker. There we go. All right. Here's

Existential crisis of coding in AI era

3:40 my thesis. This is the reason that um I am talking about this. Well, actually to back it up here, I'll I'll just back up since we have a little bit more time to give you a little backstory on myself. Um, I have been developing software for over a decade. uh professionally I graduated from my uh university BYU in 2014. So it's been 12 years since then. Um and then I'd been developing software as it like you know part-time before that uh a little bit. So yeah, about a half a decade uh I've been doing software development and um through all of that I very quickly got into teaching whether that was uh teaching on the side uh or eventually in 2019 I went full-time teacher and that has always been for me about teaching experienced software engineers how to uh or rather accelerating the acquisition of experience for uh experienced software engineers. So you don't know how to write tests? Well, great. I'm going to

1 Highlight · AI agents make implementation details commoditized

4:37 teach you how to write tests. You don't know React? Great. I'll teach you React. You don't know full stack, I'll teach you full stack development. And uh something happened this year that uh gave some of us a bit of a fright. I know for myself, I had two existential crises this year. How many people can relate to that? All right. Yeah. We all get back from Christmas break or maybe on Christmas break you're using your agent and you realize, oh my goodness, this this thing is better at coding than I am. or okay, maybe it's not quite, but it's gotten a lot better. And so my existential crisis was not only am I a software engineer actually building software, but I'm teaching experienced engineers how to get experience in technology they don't have experience in yet, but now who needs to learn React?

5:25 Like that is just not a like if you're an experienced software engineer and you're like, "Oh, I've done some Vue before or whatever. Oh, this project uses React." You don't need a React course for that. And I me just saying that means I lose money because you could go buy my React course. But I'm telling you right now, you don't need that because you're an experienced software engineer and you can direct the agent to go build the right thing. Like you don't need to know what used state is like who who cares about any of that anymore. As long as you have the right building blocks, you can build really anything now. So existential crisis for me, like who needs to learn implementation details anymore? And so I came down to Yeah, I kind of maybe I should just pull up I have this other talk while everybody's coming in. So you're going to get a little bit of that one

Choosing the right target over execution

6:11 um that is uh sort of related. So uh we we'll we'll just do the first part of this one. So um this is this is the work. It's standing up on these two stands. If you don't get the work all the way across, then it's going to fall over our tools. We your IDE, your CI, whatever. and then here's the work that you do. And over time, um, our agents have just been eating into the work that we do, which is wonderful because it means that we have less work that we have to do. We can accomplish more to, uh, get more done. It's awesome. Um, but, uh, yeah, it makes you ask yourself, what do you actually do? And here's just total overwhelm of what's prompt engineering and reasoning and and loop engineering and flow engineering.

6:54 Like, what? all of these different things that we're supposed to learn, right? As an educator, I look at that and I think, yes, I could make bank on just every other week I do another course. You come and I'll teach you what the how we do things this week and then I'll see you in two weeks because it's going to be different and I'm going to make so much money off of this. Um, as much as I love making money, uh, and I do have six kids I have to take care of, um, I yeah, I'm not into that. I would much rather teach you durable skills that you can uh learn and um and use long term so you don't have to keep coming back to your uh dealer or whatever.

7:35 So we don't know what the future's going to look like. It could look like that. It could look like this. We really just have no idea. Um but uh the the fact is that there is a possible maybe we can argue this but there is a possible future where the work is actually completely done by AGI right like that is like if you think that that is impossible and I'd love to have a conversation with you about predicting the future. Um but I think that this is definitely possible but it's not very useful to plan for this because who knows what the world is going to look like if we eventually get to that point.

8:12 uh well like what is uh humanity at that point. So let's say let's take it a step back from that. So it's doing almost everything but not everything. Okay. So it's going to fall over. It can't fill in that last little gap. So there will be a piece that is us filling in that last little slice. So my as I'm thinking about okay what do I do as an educator? My interest is that okay let's fast forward time all the way to the future. AGI is here and is doing everything.

8:38 Take one step back from that. what is that slice that humans are still doing as software engineers that is still valuable and that is where I came to judgment being able to tell uh and and not even just judgment but product engineering in general being able to tell what to build what's the right

2 Highlight · Hit the right target instead of perfecting aim

8:55 thing that we should be working on um and the metaphor that I like for this and gosh okay the metaphor that I like for this and I'll I'll actually bring this up in the talk uh the official workshop later uh is uh an archer with arrows So you've you've got your arrows and you've gotten really good at like your product manager tells you hit that target and you're like okay great and you aim you take account the wind everything and you hit the target you're like this dude who is aiming to hit the target through that ring which is just and he does it I just like wow can you imagine being that good some of you are that good as software engineers but see here's the problem the problem is that we have gotten our arrows has changed and now you don't have to be that good to hit the target because the and like I realize I'm doing some handwaving. Yes, we all know that agents make mistakes, yada yada, but like look at the trajectory. They're like homing devices now. And so now not only can you hit any

9:56 of those targets with ease, but you can actually hit way more targets than you ever possibly could have imagined before. And the the trick is the fact that because of uh well the actually this has always been the case t all the targets are not created equal and so knowing which one of those targets is the valuable thing to hit that's the differentiator. Okay I think that's as much as I oh yeah this is if if you have seen uh Robin Hood men and tights you know this reference. If you haven't then um I'm not going to show it anyway so

10:29 don't worry. Uh, all right. We've got enough people in the room. I'm gonna get officially started. So, thank you all for uh joining. Um, we're going to talk about that skill. This is what I call the durable skill. It's the last skill that the last software engineer needs. Um, it's the last thing that you need to learn. And if AI learns this skill, then I don't know what we're going to do. Um, but it's not this. So, um, this is the thesis. When AI agents level the implementation playing field, which they are actively doing, then the differentiator becomes building the right thing. Okay, I I'll I'll stop. I see some people taking photos. This is the slide to take a photo of. This is my thesis. The entire workshop depends on this slide. I'll bring it up a couple times uh later. All right, I hear some of you chuckling. How many of you know what this slide is? Have you seen this slide before? Really? Oh, how Okay, got a couple. This is very exciting. Um, I do this in every one of my talks. I'm going to invite you to please stand. If

11:29 you're physically able to join us, please do. Your blood needs to flow in your body for your brain to operate at peak efficiency, which you will need for the next two hours, hour and a half. So, I want you to put your arms out in front of you like this. Don't hit anybody, please. Squat down like this and come back up. This is called exercise. This is air squat. We'll do 12 of them together, and you need to count out loud with me. If you're not counting loud enough, I will start over. Ready? One.

11:58 Oh, wow. Two. Great. That threat really worked. Three. Four. Five. You're doing so great. I see smiles. You love this. Seven. Eight. And you can go really deep if you want. Nine. 10. You know, this is so fun. Let's start over. One. No, just kidding. 11. And 12. And then stretch over your head and then over to one side and over to the other. All right. Sweet. Go ahead and sit down. Thank you so much. Your body needs blood flow. Yes, that's right.

12:29 Uh if if you're starting to feel a little sluggish, it is after lunch. Um so if you're starting to feel a little sluggish halfway through, just stand up. Do it. We all know what's going on. It's fine. There are some seats up here in the front. So if you're standing in the back, it's not awkward. I already told them to be friendly to you. So um they will be friendly. All right. We are going to for the workshop, we're going to be working through an example product, an example app. And so, um, this is a QR code where you can submit your ideas for what that should be and upvote other people's ideas. Um, I'm running a big risk because it means I wasn't able to practice the specifics of what we're doing. So I'm going to need a lot of feedback from you on um the different ele how we apply the frameworks we're going to uh be talking about to the example app that we come up with and then also this is the same QR code and uh this is how we're going to do Q&A. Uh so I will be doing Q&A you can upvote each other's questions uh all there and I will put this same QR code

13:30 at the bottom of every slide. So uh if you are uh think of a question later then you can scan it. Um and I'll come back to the example app uh that you come up with feel free to uh to if you think of any ideas or actually as we're working through this I want you to be thinking about your own application taking notes about how these frameworks

AI accelerates bad practices without oversight

13:51 apply to your own application as well and I'll be asking you for um examples of how you've applied it to your own application. So I'm expecting you to do that. Okay. So the the premise here is AI changes the scarce resource. Uh it's no longer scarce to do implementation. That is a um a commoditized thing. Now it's again I'm being a little hand wavy. I realize that agents aren't perfect and they make mistakes. You still but um we are in a a place where agents are getting really really good and they're only getting better. So the more valuable thing is deciding what to build, whether it's worth building in the first place. And that applies not only to entire apps. Like if you're here and you're like, "Well, Kent, I work at an enterprise and I'm it's like internal stuff. Uh I'm not working on new apps every day." That's not what this is about. In fact, I'm more interested in your use case than the startups. Um I'm interested in both. But uh these all these principles we'll talk about apply to both of those. So we're moving from

14:52 can we build it to is it worth building? What's interesting about that is that has always been important and the best software engineers, the most valuable software engineers at any company were the ones that could answer this question, could help the business get to this answer. We'll talk about that a little bit too. So, we already talked about the arrow metaphor for those of you who missed it. We gotten really, really good at uh hitting targets uh with the agents that we have now. There are tons more targets we could possibly hit. And so the real question is um whether we which one of the targets we should actually try to aim for because maybe you do have infinite money but I don't and I can't aim for every one of those targets and even if you tried you would probably overwhelm your users anyway and that would be a like a failure mode to use the AI. Like I never used the words failure mode until AI started saying that to me but that would be another failure mode. Um, so I'm going to actually be referencing a podcast that I've been running for the last couple months uh quite a lot in this workshop because they're just

15:52 really really smart people uh and I really like their takes. So this has been like this is an upcoming episode. He said it's once development becomes cheap and easy and enables anybody to be able to do it especially non-technical people then it's really about the ideas that you have. Now, I wouldn't say that we can take our tools that we have today and give it to a non-technical person and expect them to build like real software that can be maintained and like last for 35 years or whatever. Um, but it's possible we get to that point and the the fact is that the implementation playing field is leveling. It's in the process of that. So, ideas have always been u one of the most valuable parts of it. And then of course there's execution. Um, but ideas are becoming more valuable relative to that execution. Um, Julius, um, I'm not sure how to say Julius's name. Sorry, Julius.

16:39 It's, uh, Marming, I think. He works on T3 code and and, uh, a bunch of other T3 universe stuff, but he said, uh, that the hard question is deciding whether a feature is worth having or what the long-term consequences are. That is an important part of ownership that we'll talk about here as well. Um, so it's easy to build to spec and think that, okay, yeah, I'm a software engineer. I build to spec. That's what I do. uh hand it off and then say my job is done. Let me go grab the other thing. But guess what you look like when you're just taking a ticket and turning it into an implementation. If that's you, you look a awful lot like an agent to me. Uh oh.

17:18 We're going to play some music, I guess. Um, turn that off. I think there's a little play button on this thing. I've never seen what that does until now. So, uh, very replaceable to just turn a ticket into an implementation. Uh so a product engineer um I is able to recognize that you can still fail after finishing the implementation if it doesn't produce customer value and that's not just your PM's job. Uh okay and then Wayne Allen brilliant guy this was an awesome episode he said that the product concern is building the right thing and the engineering concern seems to be building uh the thing right. Um, however, building the right uh right thing is downstream of building the uh sorry I I said this wrong on the podcast too when he I repeated it to building the thing right is downstream of building the right thing. So it is very much in your interest as an engineer who wants your company to be successful so that you can continue to receive a paycheck and take your kids out to get milkshakes that night or whatever you

18:19 know with the money that you get. it's in your best interest to make sure that the upstream activities are coming down so that you have uh something that's actually going to be valuable to work on. Okay. Uh wh there we go. Um Dax Rad uh creator of uh Open Code. Uh he How many people use Open Code by the way? Yeah, Open Code's pretty cool. Um he says that the default place for our new coding agent abilities to go uh to is work on the wrong things. Products go from good to bad faster than ever. How many people have noticed some the products you use getting worse? Yeah, I I relate to that. The products that I build are getting worse. No, just kidding. Hopefully hopefully they're not the products you build. Of course. No, nobody in this room is doing that. U it's just everybody else. No, it's just so easy because the agent's not going to say, "Hold up, hold up." Like, I think that we're expanding the system too bad or whatever. They might in the future, but no, like it it is our responsibility

3 Highlight · AI accelerates bloat and technical debt

19:15 to slow down and be intentional about what we're adding to our product so it doesn't bloat up to something too ridiculous. And the the problem is that agents accelerate um bad practices throughout the codebase and really do need to be wrangled in uh as Shandai is per uh uh talking about here as well. So you can move bad faster if you're not careful. Um and Rita uh Koslav, she I just recorded this episode last week with her. uh she is a the um oh shoot what a director of product or VP of product uh I think at Cloudflare uh she's a higher up and she's wonderful wonderful person but but um she said that it's very easy to get the look and feel like really quickly and just decide you know what why don't I just ship this so she's on the the management side of uh things and it's really easy to feel like oh I can just ship this but then she says you can build something that on the surface layer feels functional but it's not actually scalable doesn't address edge cases. And uh the the real key and this is where we come in as product engineers

20:17 is to actually take ownership of that to look at the prototype and be like yeah I'm going to throw this away. We'll have cloud or cursor whatever reimplement this in in two hours and it'll uh you know with my framing my systems thinking and everything um so that I can take ownership and accountability over it. The reason that matters is because if you know that you're going to be paged at two in the morning to deal with issues, then you're going to take a lot of care in the system, in the playground that you create for your agents, uh, to make sure that they're successful when they're implementing things. And Aaron Francis, you can do the wrong thing

20:49 incredibly fast and feel like you're making a ton of progress. How many people relate to that? Yeah. So, so bad. Because like you get so far into it and you're like, I can't stop now. Sunk cost fallacy is not a thing. And yeah, so how to do the right thing incredibly fast or at least directionally correct rather than 10,000 lines of code a day on the

Product engineers bridge users and code

21:09 wrong freaking thing. It's so irritating. So, uh, a question that I get often, we're we're going to get to the interactive bits here in just a second. I just want to establish things so nobody leaves the room. Um, that we are not I'm not saying now it's time for you to all be product managers. That is not what I'm saying at all. So, let's talk about what where the line is between a product engineer and a product manager. Um, okay. So, your job as a product engineer is to connect the customers uh the understanding what the customer needs to the technical choices that are being made so that you don't paint yourself into a corner and you don't overbuild. So, uh you're going to decide on the data model and the uh workflow shape observability constraints failure modes, and the smallest slice.

21:51 All of these things are technical decisions that you make and you make them better by understanding the upstream uh where requests are coming in, where the ideas are coming from. Um yeah, that and we'll dive into each one of these a little bit um more here in a second. So um this is one of my favorite the smallest slice thing. Um I just I love this. Um so how many of you have seen this before? I'm not I'm not sharing anything new, right? Okay, so you got your waterfall. Okay, well by the end we'll have something useful. In agile it's like we got useful use, but it's different every single time. We got to rebuild it from scratch almost every time. In AI it's like here's the whole thing and uh your job is to whittle it down to um the actually useful thing that you're trying to uh to accomplish here. And what's interesting actually this image also makes me think of uh how Instagram started. Instagram was originally an app called Bourbon uh which was basically a foursquare clone.

22:46 Uh, so they were having people check in uh to to things. I think it was uh alcoholrelated. I don't drink alcohol, so I don't know that world very well, but I think that's what it was all bourbon. That's out. Yeah, I I'm just kidding. Um, but uh yeah, so um they realized after watching you user behavior that the only feature anybody

4 Highlight · Building software requires seeing users in action

23:05 really cared about was sharing photos. And so they just gutted everything else and just made it about sharing photos. And they had a pretty nice exit. So good for them. Uh, okay. Uncle uh Bob was also on the podcast just last week and uh he had this experience uh as a software engineer. He was working making these little mini computers for uh a company that did like telephone lines and stuff. And so he'd make these uh software for the uh guy who would go up the telephone pole to fix stuff. And his boss said, "Uh, have you ever been on the truck?

23:37 Have you ever like seen what it's like?" He's like, "No." Okay. Well, you're getting on the truck, you're going out. And it he said it totally changed the way that he thinks about building software for people because that guy's up there trying to use his software while he's hanging off of this pole and he's realizing okay so there's like 30 things I can think of to improve this software to make it easier for that guy to use this. Um so seeing your uh users use your software um is humbling uh to say the least. So in his mind a product engineer lives half in the technology and half in the customer's house.

24:08 There's a deeply human side to product engineering. His podcast episode is coming out I think tomorrow or next week. So if you're interested in hearing more from him or really any of every one of these episodes, it's really good. You should definitely uh check it out. Had

24:21 Grady Buch on a couple weeks ago and he also feels that engineering judgment comes from technical experience married to human issues. U so being able to attach what the human actually needs to technical expertise you have. That's what makes a product engineer a product engineer, not a product manager. Um, okay. Like I said, we've got a ton of these. I'm probably overindexed on the number of quotes, but Ronin is awesome, too. It's about launching a product. It's not about the cool API.

24:51 And I, in fact, like there are so many um so many instances that I can think of where somebody will come to me and just talk about their solution all the time. Like, oh wow, like it does this and it does that and it does this and it does that. And I'm thinking, none of those things matter to me whatsoever. like it. Uh, okay, that's kind of cool, but I already do this one thing and it's not uh so much better that I want to switch from my current workflow to that thing. And it's just so clear to me that this these people are so excited about their solution that they've totally lost the plot of the problem that they're trying to solve for users. So, um, it's really easy to fall into that as an engineer.

25:30 And therefore a great way for you to stand out as an engineer at your company is to be a product-minded engineer. So you have the technical capabilities but you also understand what the user is ultimately trying to do and what your company is trying to do to solve uh their problems. So um a a lot of the decisions that we make as engineers um

Architectural primitives dictate long-term success

25:52 shape what the product ends up being and um and the reason that is so important is because changing architecture is really expensive. You might think, well, no, Ken, like it's literally just like, I don't know, a million tokens, and I can be from, you know, one database to another, or I can be from monor repo to microservices or or or uh mono monolith to microservices or whatever. Uh it's depending on the size of your app, maybe that is uh the case, but um it is really expensive on the way that it impacts the rest of the team and on the way it impacts the user. Um, change is expensive and and if you say, "Okay, well, it's just a couple million tokens.

26:33 What if I could hire you or I could hire somebody who made the right choice the first time because they understand the product?" Yeah, I'm going to hire the person who can uh do it right the first time uh and cost me less. So, it's all about the primitives. This is Ree. Uh he works uh actually he just made a startup uh and really cool dude. But, uh he's he's saying it's all about having the right primitives that you can build upon. It's really difficult to build a really great solution on top of those u wrong primitives. So your engineering expertise matters. Hopefully that uh uh that you know what dang this is going um I got to talk about Michael. Who who here knows Michael at work OS? This guy rocks. Uh and work OS is really cool.

27:15 But he uh work OS is an authentication platform and he was deciding okay so with this new platform do I make it like a on-rem thing like I sell you a license now you can run this uh authentication thing on prem or do I make it like an open source thing with like a commercial ARM or whatever and after interviewing and talking with a lot of customers he realized oh no like if there is some sort of security problem then we need to push out a fix immediately and we can't wait for people to upgrade some npm package for that and so therefore I will make it a hosted solution and so your technical expertise and understanding married to the understanding of the um the user and the actual problems is going to make a significant impact on the direction that you take the uh product um oh gosh you know what I'm going to it's been way too long since you've actually done anything so we're going to skip over this you need hopefully you get the point there is a line between product manager and product engineer and both of them need to really

28:15 understand the uh the actual problems

Designing systems to prevent user error

28:18 that the user has. So uh one of the one of the things that you are doing as a product engineer is um building a system for your underlings to be successful in your uh even like years ago before we had AI agents you would have a team lead and their job was u to make sure that that system that you're working in is very efficient for you. So you got good testing, you've got good architecture. It's very obvious which Lego blocks to put together to build out uh different features. These primitives are very important. We are now all team leads over however many agents you're able to run at once. And it's your job to make sure that that playground is a really nice playground and they don't make a ton of mistakes. Okay, I'm going to skip uh ahead. Here we go. Um, and uh gosh, I really want to get you you all doing stuff here really quick. So, I don't want to miss anything. Oh, okay. There's so many things that you miss if you don't have product

29:19 understanding. I just I got to tell this story. Who knows Don Norman? Who ever heard of Don Norman? This guy legend. He's 90 years old. He comes on my podcast. Like, that first of all is pretty wild. and he has so much he this is the guy who invented the term user experience when he was at Apple. So like yeah kind of a cool dude um and just delight to work with. Well uh how many of you are familiar with the three mile island um incident? Okay.

29:49 1979 in Pennsylvania, there's a nuclear reactor. Something terrible goes wrong and um and like there's a cool issue and and whatever. Uh it was a technical issue and these operators made some um bad decisions based off of the technical problems they were experiencing. So they bring in Don to come and be like, "What happened? Why did these operators make such a terrible decision?" And he looked at the problem totally differently. He said, "You know what? Actually, it was not their fault. They're smart. Uh they're competent. The problem was the system. The system was wrong. User error does not exist. That is his assertion.

30:25 So if you want to learn more from Don, the design of everyday things is like the canonical book. You absolutely should read or listen to that. Uh and he also wrote design for a better world. Also a really u great initiative there too. So I sorry I had to jump like this guy is fabulous and really understands users. Okay. So before we get into um talking about the specific app and and the frameworks that we're going to get into, I wanted to double check if uh there are any questions. So let me pull those up and um we'll we'll get to our

Hiring for product sense over coding tests

31:02 example uh software here in just a second. Um we've got one question in here. How many features are too many for an MVP? Um, that depends on like tons of factors, but I would I would actually say that that is the wrong question. The the question is, and we'll get into some of the the frameworks here in a little bit that will kind of help answer this, but the question it really is uh what is the minimal thing that you can do to demonstrate that you solve uh a problem for uh your target audience?

31:32 Uh and in particular, if you're entering a space that's already overcrowded, you want to niche down as well on that target audience. So, we'll talk a little bit more about this. Um, but since there are no other questions on here, does anybody have a question that they didn't put on here, but they want to ask? Go ahead. You can just yell it and I will repeat it. Oh, we've got a couple. Is there a link to my slides? There will be at the end. Um, right now it's local host, so I can't um but but at the at the very end there is actually a link to uh to the slides. So, haha, you have to stay uh or watch it later. Um uh how do you evaluate software engineers in the hiring process for product engineering? Uh okay so I this is a great question hiring is really really tough to say the thing that everybody knows already. Um having really good technical expertise as I explained is still very important.

32:31 Uh I would not be doing coding challenges at all. Uh that is like I certainly wouldn't be doing any coding challenges where I say no you are not allowed to use a an agent to do this. Like what are you hiring them to do? Like how awful is it to work at your company that I can't use an AI agent? Goodness. So no instead you're going to be talking about systems. um you're going to be talking about how to uh in fact I would probably take a couple of the things that we're going to do here in a second and I would just ask them like here's an example what are the questions that you're going to ask to figure out what the the core problem is and then and then once they say okay yeah you got what the core problem is now from there what are the system implications from the um that discovery of the core problem that would be how my my interview for that would go that's a great Um, I lost it. Uh, okay. When do you stop talking to customers and start building? Uh, I think that you do both.

33:28 Constantly. Always. You do not stop. Uh, it's it's a feedback loop that continues to go. We'll get into this a little bit uh as we go um further on in the workshop here as well. Uh, not only what to build, but how to get users to use it. Traction is the new moat. Um, okay. So, I don't actually talk about this specifically. Distribution is is definitely a difficult problem. Uh, you just get Theo to talk about it on his YouTube channel. That's that's the answer. No, just kidding. Um, yeah, that's a that actually is a really difficult one. Uh, I have heard that people will just spend an unreasonable amount of money on u marketing and advertisement and that seems to work for them. So, um, yeah, this is not something we're uh really going to uh to talk in uh talk about and I'm not prepared to answer. So, sorry to not be uh all that helpful with that one. Uh, okay. And we Oh, did I just mark one that was that I didn't answer? What did that question say? I think it moved on me. There's a bad user experience right there. Somebody who works at Slido can go fix that. Um, okay.

34:29 Uh, I'll I'll answer like three more questions then we'll move on. Would you recommend product managers stay in product teams and product engineers within Oh goodness. Uh, as far as organization is concerned, um, often applications end up uh, looking a lot like their org chart. So um I would huh uh I I definitely feel personally that your product u manager should be very integrated with engineering. That communication layer should be very tight. Um and your engineers should also not be siloed from other engineers in the organization because one important uh thing for a product engineer to do is not only look at their slice of the application but also step back and see at least their neighbors because if you don't then you'll end up building things that are already solved for you by other primitives that are within the organization and uh and so you're just rebuilding those same things. So you need to at least go one neighbor out and one neighbor up and down. Uh and this is

35:30 actually the same with um programming languages like if you're a React developer then you need to understand how JavaScript and the browser work so that you can use React effectively even though you're not necessarily using all of the features um that are included there because the framework is doing that for you. So it's the same sort of idea. Um tip highlight the questions to bring the focus. How do I highlight? Do I just click on it? All right, we're doing a product lesson right now. That is a a useful tip. It seems like a good idea. Oh, is that what that is? All right, there we go. User error. No, it's not. It doesn't exist.

36:07 Uh, okay. So, if I highlight it. Okay, that's cool. Should all engineers be product engineers? Um, well, in my estimation, if you're not a product engineer, if you don't have product sense or design sense, it's that's going to be really easy to replace you by a a as the agents get like continue to get competent. I don't want to be like alarmist or anything like you know, whatever. But it does seem like if all you're doing is listening to somebody tell you what to do and then turning that into a code implementation, that part seems like it's pretty replaceable.

36:40 If you don't understand the product, uh, then the system you build will not solve the user's problem. Well, okay. I think we will come back to these questions uh as they're voted up a little bit more

Validating ideas by asking about past behavior

36:52 later. So, after um quite a while into the workshop, we're going to finally talk about what the workshop is going to um be talking about. So, um what I want you to leave with is um frameworks for early idea validation. How many of you have heard of the mom test? Okay, sweet. So, we'll be talking about that a little bit. Framing user request jobs theory. Anybody? Okay, sweet. And prioritizing software changes. Kano model. Okay. Yeah. So, I one of you raised your hand three times.

37:22 So, I invite you to come up and you can talk about this. No, just kidding. Uh, okay. So, here we go. This is the dangerous thing. What is the idea that we're going to be framing things around? And this is a risk. I've never given this talk before so this might go really really poorly. Um hopefully not but uh let's look at what we've got. The top voted with fire is an app that gets people into conference workshop sessions.

37:50 Uh uh yeah. Okay. I I literally am going to do that. Um okay. What should we call it? What's what's our short um how about Lightning Lane Workshop? You know, call back to uh to Disneyland, right? Lightning lanes. Yeah, lightning lane.

38:08 Just get in.

38:09 Oh. Oh, there. That's so good. Just get in. Oh, I love that. Just get in. I don't know. Like, are are some of you thinking like weird things? The workshop. And sick people. I just heard everybody laughing like why are they laughing? Oh, okay. Gosh. Okay. So let's talk about early idea validation for the just get in the workshop app.

38:38 How do we validate this? All right, throw out some questions. What are question? You're at a conference. This is the perfect place to talk about the just get in the workshop app. So you you've got all these people. You're standing in line and you're like, go this is so awful. What are the questions that you're asking people to validate that your idea is a good idea? So just throw them out. Raise your hands if you want to. Yeah.

38:57 Why do we need that app for that?

38:59 Why do we need an app for that? Okay, that so ask them, is it a problem? Is this a problem? That's Yeah.

39:05 How long have you

39:06 How long have you been waiting? Yeah. Okay, that that's getting really close to a really good question. Other questions?

39:13 Sorry, what?

39:16 Why are you waiting? Oh, okay. Yeah, that's a good question. Yeah.

39:21 Oh, could there be a pre-registration? Like, yeah. Wouldn't it be better if this was a pre-registration thing? Yeah. When was the last time you were stuck?

39:29 Oh, you've read the mom test. All right. All right. He said, "When was the last time you were stuck?" Yeah, that's that's okay. Okay. What What else do we have?

39:40 Sorry. Say that again.

39:43 Oh, what's the max time we're expected to wait? Yeah. Okay.

39:46 Are there any open seats?

39:48 Are there any open seats? Yeah. Okay. Okay. So these are some questions that you might think okay if they what you're looking for or what we naturally are looking for is somebody to say oh yeah this is a huge problem for me I definitely want some I would I would spend just untold sums of money to be able to get in the workshop really quickly and easily. I I think you had another question.

40:11 I got another one like what is your what's what your experience while you wait

40:17 oh yeah how's your experience been while waiting? So you're trying to like get a sense for uh for that overarching experience. Yeah. And that can be a good input into the solution. Is this a solved problem? Oh yeah. Like have you heard of an app that could really have sped this process up? I saw somebody holding up the mom test book. You literally have it in your Unbelievable.

5 Highlight · Why standard user interviews fail and what to ask instead

40:39 That's awesome. Uh I I also carry my mom test book with me everywhere I go. Uh yes. So this is the mom test. You don't want to ask users to evaluate your idea or diagnose the problem for you. This actually this is so common and it's so hard because when you think of an idea, you start thinking, okay, so we will we'll make it uh a mobile app. Obviously, they have to install it on their phone. Oh, wait, no. Uh maybe like it's a one-time use thing.

41:06 So, we'll also make a a website and then we're going to uh integrate with all of the different uh providers of all these different things. And uh and so then you start talking with people and you say, "Would it be easier if it was already integrated with the app, maybe the conference had a link to it or maybe it was integrated?" And so you're talking about the solution to your specific solution with this u potential possible user before they've even agreed that this is uh a problem worth solving.

41:36 Okay. Um so here are a couple of the weak questions that may sound a little familiar. Would you use this? Is it a is this a problem in the first place? What is the problem? Help me define the problem. Some better questions are tell me about the last time this happened and what did you do instead? And what did it cost you to do that? Now for ours, none of us I think would actually I don't know you all paid to come here to the conference. So there is some amount of like I'm waiting I'm missing the first 20 minutes of this talk that I actually paid to attend. Don't get mad at Sean.

42:10 This these things happen. Sean's a friend of mine. U but uh but yeah, like the the questions of what did you do instead? What is your workaround? And how much did it cost you? Because a solution like this doesn't exist. You do not talk about the solution at this stage. It's so hard not to. But you don't. Instead, you start with um what did you do before? What what is the past? You do not want to ask them future questions. Nobody can tell the future.

42:37 You don't want to say would you pay? how much would you pay? Whatever that often what that turns into is somebody's like, I kind of want to get out of this conversation and I'm just going to say yes to everything that they say. And in

42:50 the book they call this complimenting. So that you're just you're you're asking you're fishing for compliments. Uh so Don Norman said don't ask somebody what's a problem because they'll tell you the symptoms and often they will actually tell you solutions as well and their solutions will be just as ill-formed as your own uh and maybe even more so. Um, so you don't want to just ask generally what is the problem. You want to ask about specific behavior. So tell me about the last time this happened, what happened, give me specifics and who was involved, what made it annoying enough to remember. Now our example is kind of um kind of funny because you if you're in line, you're literally experiencing the problem right now. So, it's a little visceral, but most of the time you're not um like your idea to solve somebody's problem.

43:35 They're not actively experiencing that problem right at that very moment. So, it's not necessarily going to be so much, well, tell me the last time it happened. Well, it's happening right now. Like, maybe it is. And that's a pretty good signal. Then, you're going to be able to get some really um direct answers to that. But often, you're going to have to let them think back about like, oh yeah, why why was this so bad and who was involved? What made it annoying enough that I can even remember? If they can't remember, that's a signal and it's not the one that you want it to be, but you want that signal, that's for sure. So, uh, you want to know what they did do instead. So, what are the workaround? Oh, whoops. Ah, back up. Uh, how much does it cost? How often is this happening to them? Uh, how much effort are they putting into solving this problem already? And that is gold.

44:20 That is, um, a really strong evidence that this is a problem we're solving. if they're already putting a bunch of effort and paying a bunch of money to solve this problem with some workaround um that's a really good opportunity if they are like yeah I I don't remember the last time I know it's happened yeah I don't really remember u and I just I kind of gave up that that that's a pretty good signal too like that's a signal that they're probably not going to pay uh and and change their workflows or whatever uh to get a solution now um the uh yeah u the the point here is that as software engineers, we've gotten kind of used to just like sitting in our corner. Well, some of us I'm kind of an extrovert, so I I like to talk to people, but uh some of us are just like, "Yeah, I got into software engineering because I never have to talk to anybody and I can just like take my jar ticket and turn it into implementation." I that I'm sorry, that's not the way that software engineering ever was. The best software engineers were not that way.

45:16 Um, I mean, you could turn out some pretty amazing code, don't get me wrong, but the software engineer who really talks to people and understands their problems is the one who's going to build a system that can solve those problems better. Uh, and uh, one one thing that Michael did actually in our conversation, he was talking to so many people validating his idea before he uh, built work OS. Uh, and he would ask them what are they losing out by not having a solution to this? What's the cost of not doing that? And that worked out for him.

45:44 So the output is uh like actual evidence rather than somebody just saying, "Oh yeah, that's a good idea." And this is why it's called the mom test because your mom, of course, she loves you and she's going to be like, "Yeah, of course this is wonderful. I would absolutely use it and I would and so now you have

Avoiding costly overbuilding through early validation

6 Highlight · The heavy financial cost of building without market validation

45:59 two users, yourself and your mom and uh or dad or whatever." And and that's wonderful. Good for you. Um so um here's a a fun story from Wayne. Uh he he lives down in Australia. He worked for a real estate uh company down there and they were building a product uh that was kind of an experiment in in this bigger uh organization. They he was on a team that's building this product and in the process of that they ended up building a really highcale thing and they in fact decided what if every single person in Australia logged in at the exact same time we need to make sure that we handle that. And uh they spent a year working on this. It was like microservices the whole thing. Um, and after it was 1.2 million u Australian dollars that they spent on this, which is like almost 900,000 US dollars, um, they uh ended up like nobody used it, not a single person. So like not only did they not have to handle every person in Australia, they they didn't even have

47:01 to handle one person in Australia, nobody ended up using this thing. And so he told me that uh something he learned from that is we probably could have launched something in two weeks, not a year and done the integration manually. The integration is always the most painful part anyway. Uh and we could have tested the market, learned a lot of things very very quickly and not invested a whole team to work on this. So project validation really important and it's again not just the PM's job to do this. It's your job to really

47:29 understand okay so this and and we're not just talking about startups here either. this is a product feature that you've been asked to do. And if if that feature request doesn't come with some sort of validation of why this exists, then you as a product engineer need to go back to the PM and say, I need to understand a little bit more about why this exists for not only to validate that this is going to be worth engineering time um but also for some of the other questions that we're going to talk about here in a second. But part of that is um understanding what like where this request is coming from will help you shape the system that you use to build it. Is this just like a one-off experiment or is this something that's going to be core to our business? I'm going to make different technical trade-offs based on that. Like how much investment do I need to spend? Should I spend a year working on this? Is this like the business will go out of business if we don't have this? Okay.

48:23 Okay. Well, I'm going to uh spend uh a little bit more time and attention to detail uh while still like balancing the um MVP style like let's just get some validation. Okay. I'm really glad I grabbed that water. Uh yeah, this is something Michael did as well. Very easy. Yeah. Gosh, this is

Closing feedback loops with early prototypes

48:44 so relatable. It's very easy to convince yourself that something is a good idea just sitting in your room by yourself hacking on it for weeks and weeks and weeks. Uh, so you need to show it to people. Uh, I How many people relate to that just like Yeah. Uh, it's fun, right? Like why you got to rate my parade? I'm having fun doing this thing. And then you go show it to somebody and they don't get it. So you don't want to talk about it. You just want to work on it yourself. That was a signal right there. Like if they if they don't get what you why you're so excited about it, then that yeah, you want to pay attention to that feeling. Um, and then we need to close the loop with user feedback. is part of like shrinking things down. So like the validation of the idea that the and the mom test questions we talked about earlier that is like telling you whether it's worth um working or solving the problem in the first place and now we're getting into how do you know whether your solution is doing that effectively and uh Ruben says that user feedback is how you do that.

49:41 So uh and also what Michael was doing and and and um Wayne as well. So you try to get something some prototype to those early users. Early adopters are like totally fine especially if if the solution that you build has a ton of gaping holes then yeah they're going to be like pretty um their reaction to that is going to tell you a lot. If there are a lot of gaping holes but like it gets them like almost there or like just gets them over to what they're trying to do and they're still excited about it. That is very significant. if that doesn't quite get them there and they kind of give up early or something and they're like, "Ah, I don't know. I could just, you know, I had a couple of these, you know, paper cut issues and so I stopped."

50:24 Okay, that's a really good signal, too. The the U problems that are really worth solving are the ones where people will just run through the glass cutting themselves and they're like, "We made it." Now, that's what you're trying to uh to go for. And and getting early user feedback um through prototypes helps a lot with that. So, why product engineers need the MOM test? because it helps you translate what the PM is saying um into the shape of the system that you build uh and select the appropriate architecture um especially understanding what do users do now or what what do our potential customers do now to solve this problem um can help you determine the architectural shape okay so um there's a lot of opportunity here and um there's going to be a lot of downtime so we need to pay attention to how we do migrations or whatever uh okay and yeah it's very easy to end up building the wrong abstraction without that context. So, I'm going to pause really quick. We'll see what questions you all have. Um, so far the most upvoted ones are the ones I'm going to focus on. Um, and back here, here we

51:27 go. So, three main things that, uh, let's just highlight that. There we go. Three main things that distinguish a product engineer from product manager. Do we need both? The three main things.

Technical expertise distinguishes product engineers

51:37 Um, so yeah, kind of talked about this a little bit, but I would say that the product engineer is the one that's I'm not sure if I'm going to get three, but I'll just say a bunch of things. Um, product engineer is the one that is actually in the code. You may not necessarily be looking at the code anymore, but you are thinking about the system and you're thinking about the uh data types that are available, the primitives that you have available. Oh, we do have like a cron job system or Q but there are some problems with that and I and you understand what those constraints are and and what the limitations of your current infrastructure are you understand what the available uh primitives are that and so when a request comes in and you understand what's trying to be solved here you can categorize that into whether it fits in the existing architecture or if you need to expand beyond that and understanding um that um the ultimate user goal or the job to be the foreshadowing. Um that is going to help you decide whether we should actually expand the system with a couple more primitives or if you can reuse existing ones. Um that's probably the

52:39 the number one thing that distinguishes a product engineer from the product manager is just that technical expertise and understanding of the system. Uh which a product manager could probably just talk to an AI about over and over and over again to like kind of ramp themselves up, but they should actually be doing other stuff. Um and so yeah, I would say um the to answer the question, do we need both? This is actually interesting. So um I am Oh, sorry. I I am self-employed. So I do not have a product manager or a product engineer or a CFO or what? Like I just work for myself. I am all the things. And you start a startup often it's just like three of you. Sometimes you all three of you are technical or maybe one of you is the the business person or whatever. And as the company grows, you eventually start divvying out those responsibilities and you have multiple jobs. So yeah, um do we need both? It highly depends on where you're at in the process of uh your scaling of your business and and whether you ever want to get there. So um I think that now uh you can actually build a much bigger

53:40 business without splitting out into a much bigger organization uh than you used to. Uh great question. Okay. How small do you think teams Oh, literally just said that. Um, pretty small. I like I don't know how I can give you a more specific answer to that. Uh, okay. How would people get promoted from junior to senior in the age of a AI? Um, well, huh, I'm not sure how to answer this question. I I think uh juniors and seniors are both um like we're all ultimately trying to do the same thing.

54:16 even all the way back before AI um uh the junior was just trying to emulate the s senior as much as they could until they s suddenly or over time become a senior engineer and I think that nothing about that has changed uh as far as like actually getting promoted that's just going to be a conversation with your uh with your person uh I should actually stop really quick because I I'm just noticing people walking out and my timer went to zero this session is a two-hour session it's just back toback and I'm planning on just going straight through But if there are other sessions you wanted to see, you're welcome to stand up and leave. I will just take a mental image of you and frown at you in the hallway later.

54:54 Just kidding. I won't do that. You're you're welcome to take off if if there was another session you wanted to go to. Um because there are so many good ones.