Use AI to Make Product Decisions

4,858 words · about 19 min to read

Using AI for high-level product decisions

0:00 I think we are aligned. So, let's go. All right. So, what did I just tell it to do? Well, this is better with Kent, where you and I learn durable skills to get better together. And in this episode, we are going to talk about how you get alignment with your agent. And I'm not just talking about agentic coding or whatever. In fact, that's not really the durable skill. Agents are going to continue to improve everything. But what I'm talking about is how to make a product decision using your agent to help you make that happen. So to figure this out, let's go all the way up to the top. The first thing that I did in this lengthy conversation was I gave a link to a post on X to my agent. Now agent has Cody and in fact we're working in the Cody repo, but through Cody it's able to resolve this and find what this is all about. This was from Ashley Peacock and this is Cloudflare agent week which is going to impact Cody significantly because Cody runs on

1:00 Cloudflare and so I'm keeping a close eye on what is going on with Cloudflare and this Cloudflare computer thing caught my eye especially and if we take a look at the blog post it is pretty interesting. Your agent needs a computer not a container. Okay, that is interesting. I feel like a computer is like a bigger thing than a container. But if you look at the blog post, it's actually um a very intentionally uh scoped thing. So there are parts of the computer that don't need a full-on container. That's kind of what they're talking about here. And it's pretty cool. So Cloudflare is all about working hard to provide the right primitives.

1:33 There is that primitives work. Again, you should be thinking about primitives all the time. And this entire conversation, that's what we're talking about here, too. Product decisions revolve around primitives. Uh so that matters a lot. That's why I highlighted it. Uh and then here we see the uh cloudflare computer package provides an agent runtime where the details and mechanics of what code runs in an isolate uh a container sandbox or a web browser are handled by the platform.

1:59 Okay, so drastically simplifying things. Now this is interesting because Cloudflare computer is probably something that anybody could have built as a package that uh runs on Cloudflare and just find like kind of routes the agent to do these different things. But Cloudflare just has these amazing sets of primitives that they can then build on top of themselves and then expose a a nicer cleaner set of primitives for us to simplify our lives. Uh which is one of the things I love about building on Cloudflare. So yeah, we need to look for solutions beyond traditional containerization. Okay, so that sounds interesting. give the repo a star and I hand this off to the agent uh to look at not only that but the other things and see if there are opportunities for uh possible uh built-in features and capabilities of primitives we could add uh for those new platform features. So after just a couple minutes of working then it was able to uh determine what these features were and um what Kodi already supports by way of primitives and it's saying here's kind of the TLDDR

3:01 of what this is about. It's day one rundown of Cloudflare agents week and here are the thing the opportunities available. So it calls out Cloudflare computer as the strongest fit uh which I expected as well and it suggested a new workspace assistant primitive. So like basically a computer so that through MCP the agent could work on this remote uh computer and not just um when you're talking to the agent but like through triggers and everything else that the agent can uh or that Cody can enable the agent to work in. So having like basically a little computer that uh you could use at any time seems like a reasonable thing to do. It's saying hey we're going to add this to your available primitives and on top of that we can use this to improve the repo sessions. So let me just give you a a quick rundown of what uh how Kodi works with repo sessions. There actually two tracks for this. If you're using uh a coding agent like cursor or VS code or cloud code or whatever uh then that is able to clone a repository, make some

4:03 changes to it and push the repository. That's the preferred mechanism for working with Kodi when you're building out your packages on Kodi. And then um you can use any agent to actually execute those utilities or or the things that are exposed by those packages that were built. And in fact, you could even use like cheaper models and stuff like that too because it's pretty simple at that point. So it's a building the packages is the tricky point. But sometimes it's useful outside of the context of a coding agent to make updates to your packages and you make those updates uh through these repo sessions. And so you can communicate through MCP and we parse out the uh commands that you're issuing that are git commands and we try to simulate those the best we can. Uh we don't do a super great job of that. I don't really use this feature a whole lot myself because I just use coding agents when I need to work with packages. And so the agent here is calling out that we could actually improve that considerably. So we don't have to parse the git commands and try to simulate it. But we could actually use real git inside of a computer. Okay. Yeah, that makes a lot

5:04 of sense. So, this was a another opportunity here. And then it said you could go full-on container mode and have like npm install and running tests and stuff. And it's saying, hold on. This is like kind of a pre early preview package. Maybe not something to bet the ship on just right right now. And I kind

1 Highlight · Using agents for exploratory product research

5:20 of uh agree with it at this point. But here the point is using the agent to go and gather as much information as you can. So, and even make recommendations so you can start thinking through these. And this is actually something that Will King brought up on the become an epic product engineer podcast. He said that you want to use agents for gathering information to use your judgment on. And so that is what we're doing here in this case is providing the agent with the necessary context that it needs and then it can go out and do additional searches or whatever and then like synthesize that information for us so that we can make our own judgments. And what's interesting is these recommendations are not necessarily the direction that you want to go. And so you do still need to use your product mind and your systems thinking to be able to make sure that you're making the right decisions for your primitives. So the agent made a couple other suggestions and I kind of honed in on the Cloudflare computer package because that one's the most

Protecting systems from feature creep

2 Highlight · Guarding primitives against feature creep

6:17 interesting to me. And I said, "Hey, couldn't the Cloudflare computer package just uh be a community package or is there a real benefit for making this a built-in primitive?" And this is a really important thing that you need to remember is that once you've added primitives to your system, it's really difficult to get those out. Now, not technically difficult necessarily. You can just delete the code, but you have users who maybe rely on those features or you have uh other primitives that are kind of built on top of or combined with those primitives. So, you really need to be careful about this. And especially during my time as an open source developer, I I still do open source, but when I was building packages that were used by many people, the decision of whether I bring in an abstraction was a really really big one. And I had some projects early in my career where I just said yes to everything and eventually the package just became too difficult for me to maintain long term. And so you have to be very protective about what you invite into your system and be very intentional. And so that's why I'm asking the agent here, hey, isn't there

7:16 an existing primitive that we could just use and provide all these same capabilities and ultimately the answer was no. And but that's why you ask these sorts of questions. Another thing that product engineers do is they're not just thinking about the feasibility like is it possible, but we're also thinking about the business and whether this actually makes sense from a sustainability standpoint. So here I said, how would this affect entitlements? I'm thinking about cost.

7:41 What does this cost? How does it change the way that we're uh our users are using our system? Would it be possible for somebody to use this and abuse this to the point where they are costing us more than they are paying us? Because that will not be sustainable either. And if you're at a big company, you probably are not tasked with answering these questions yourself. But if you receive an issue from your product manager or whoever and it doesn't have some semblance of like how this affects the long-term uh sustainability of the business or something and you don't just know then you probably should ask because it is going to change the shape of the system. And in this case it's saying hey like for most things this isn't going to change anything. However container mode that would actually change things considerably. That's a new cost class. we need to gate it before we actually build it because containers cost a lot and it gave me an idea of how much this could possibly cost and I ultimately decided hey we're going to postpone that people can actually use containers uh as a separate thing as

8:43 kind of a community package that is not a primitive that we necessarily need that's built in and so we're going to kind of wait on that one but some of these other things they don't actually affect entitlements or or the cost of things nearly as much and so we can proceed with that. But again, you as the systems designer, you need to take these things into consideration because you should care about the business and the sustainability nature of things. And I would say that the product engineers who

Driving decisions with usage data

9:09 do that the very best are probably the most valuable members of the team anyway. And then another thing that I asked it to do is here I said look at past usages by myself. I'm the the biggest user of Kodi currently because it's still invite only right now. Uh, and I said, 'What things have I done that would have been better if we had the Cloudflare computer package? Or what things am I not even considering doing because I don't have computer and I'm not uh in the right frame of mind for this? So, this is another thing where you're saying, "Hey, agent, get me information, but not just like speculating things, but I want you to use real data." And this is why it's important for you to instrument your code and your entitlements, your usage.

9:49 You need metrics, you need logs, all of that information that the agent can go and grab for you to bring this actually datadriven u information in front of you so you can make a data driven decision. And so yes, it gave me a bunch of uh really useful bits of information. Um some of them were incorrect. Uh so this uh chunked CSS uh encoded or base 64 encoded thing that was wrong and I corrected it later. A couple of things are just like suboptimal ways to do things. Um, but all of this kind of coming into my mind as the decision maker is really valuable for me in making sure that I'm making uh judgment based not just on my gut but on the actual data. So ask your agents to bring you the necessary data to help you make a right informed decision. And then when that data comes back, you don't just take it for what it is. I pushed back the chunkedbased 64 thing that was a limitation of MCP. you know, Cloudflare computer is not going to help us with that. Um, that said, like the what the

10:50 agents are doing for cloning repository locally and stuff. That's that's something to think about. Uh, and then this u the reason I was doing things this way is because of predated stuff. And here's something that is interesting. Uh, I say I would actually

Delegating side quests to secondary agents

11:02 prefer that file be in a git repo anyway. Can you spin up a cursor cloud agent in the use cod repo with grock 4.5 to make that shift? So the use cod repo is just a repo with a single file. It's an agents MD that instructs the agent to just use Kodi. U and I use this a lot when I just need to have an agent that uses Kodi and has a file system and a full environment uh along with it. And so I'm saying, hey, um I I need you to fix that package, but do that somewhere else. I don't want to distract our current conversation. And I do this a lot. This is one of the really nice benefits of having an agent that has some really good primitives that it can use is because you can have it go off on this side quest um where it's not distracted and it can like spawn a separate agent to go and do that. I actually did that multiple times in the course of this conversation. And then taking it further, I said, "Oh, you okay, you got me on the sta stash and skills packages. I need to know a little bit more about how Cloudflare computer can help with this. Uh these are probably also just git repos. um but I

12:02 don't want to have to clone them every time I want to make a change to republish everything. Uh kind of made them mini apps with CRUD capabilities. Does computer make this easier? What are the cost implications here? Thinking again about the sustainability of the product. Uh and then sending a lot of other things. But one thing, the reason I want to call this out specifically is as we get down here to um like thinking about stuff that I'm missing. I'm saying, hey, yeah, you're current, you're right about artifacts uh co covering this, but there are two gaps.

12:31 Now, that should be a signal to you that it's uh a gap is not necessarily a missing primitive. It could also be a primitive that needs expanding. So, here it's saying first, artifact repos only exist attached to packages and jobs rooted at a package json. There's no just a repo of documents entity. So, a better with Kent episode draft folder has to masquerade as a package to get a repo. And that right there, it kind of sat in the back of my mind for a little bit and it took me a while to really get this. But that should be a strong signal to you that there is an opportunity to not make a new primitive, but expand an existing primitive. And so that's what we're going to see here in a little bit.

13:10 So then I had another uh thought where okay if we're going to be using git repos for managing some of this uh larger files then maybe we need to do this in a uh like instead of a durable object we need to put this in R2 because durable object storage is probably a lot more expensive than R2 more than a rounding error which is what it called

Applying human engineering judgment to costs

3 Highlight · Applying engineering expertise to cost estimates

13:31 out uh earlier in its uh message and I was like I don't think that's more than a rounding error. I think that's probably considerably uh much more. This is where having expertise in this industry is really valuable. I'm not saying that like you could hand an agent over to a nontechnical user and expect them to produce software of the caliber that an experienced person could develop. And maybe one day that's the case, but currently your expertise as a software engineer is highly valued here.

13:57 And if I had just gone with okay, yeah, that sounds good. go. It would have stuck large files in durable objects and it would have cost me 13 times more than in R2. And so anyway, that's uh an

14:10 interesting thing we're going to look at here in a little bit. Uh also spinning up another cursor cloud agent to do another side quest sort of thing. So we're not distracting the primary agent. Um but we can still solve this problem while I'm thinking about it. Uh, and what's nice about this too is you might think, well, Kent, I could just literally open up a new agent somewhere else myself and type in the prompt myself. And I'm saying that sounds just fine if you want to spend your time doing that. But this agent has all the context necessary and it can prompt other agents really well. So, if you don't have that sort of system set up, you absolutely should look into getting that sort of system set up. So, uh, it also said something I needed to ask a question about. Um I I also wanted to know like what do we how do we work around this problem getting large files into the computer and um and then also starting to think about the product impact and the way that we shape this and what how users are going to experience it. And so here saying should each user get their own computer or do

15:10 they get many computers that they can interact with with is isolated file systems. So we're going to take a look at what it does here. So here I caught it being sloppy. You know they are sloppy sometimes just like humans. And uh it realized, oh yeah, this is going to cost way more if we use uh durable objects or R2. Uh and then as we kind of answer some of the rest of these questions, but now I'm starting to think, hey, like maybe we just do data repos. U if we're going to just do data repos, like the agent was thinking, oh, we'll just have a workspace with a repo inside of it. But I'm thinking if that's what we're doing, then wouldn't artifacts be a better primitive? Like we already have this and in fact coding agents would just use git to communicate with the computer. So it feels weird to have a computer that's just a git repo when we have artifacts. And then the large file solution seems fine. Um and I'm starting to think, okay, like do we make this super easy? Um do we want to redirect the user to use R2 or do we do

16:11 this for them opaquely? Like they have no idea. And in fact, this is how the uh Cloudflare shell works is that it stores things in SQLite and R2. And then I don't know how it decides which to store it in, but it does so opaquely. So I'm thinking, oh, maybe we can do the sort of same sort of thing. Uh and I'm saying, hey, based on what you're telling me, the actual prim missing primitive is an artifact repo that is not a package. So we just a repo that stores data and is not actually published so that it's importable by other execute calls and stuff. And so now if we have that primitive what's left for um this computer uh thing and so this is the sort of thing that you get from having a conversation like this with your agent is that you get that implementation decisions and their impact on the product and the consequences of those decisions. So that's what a product engineer does is they're not just thinking about the ticket they've got, but they're thinking about the system as a whole, the available primitives that they have, and how we can reduce the negative impacts on the business and also improve the

17:12 positive impacts on the business. Not just implement the thing as it sits in front of you, but thinking about the broader picture and maybe solving many problems with a single simple primitive. And here the agent decides, yep, you're right. A artifacts is a better backing conceded without reservation. And here we're talking about large files being opaque with an entitlement gate, not user restriction. And the more that I thought about that, the more I thought this is going to violating the principle of lease surprise, which is not just a programming principle, but it is a product principle as well. You don't want your users to be surprised. You you want them to feel anticipated. You want them to feel that kind of surprise like, oh yes, that is what I was going to do next. But you don't want them to feel like, "Wait, why did it do that?" or "It's doing this." Like, "That's confusing." And so, right about this

Verifying technical constraints empirically

4 Highlight · Empirically testing technical limits with agents

18:00 point is what I'm thinking. Uh, doing an opaque thing probably isn't the right way to go here. And so, that's actually what we get to in the next uh prompt here is I say, "Hey, do some research on large file support in Cloud Fi Cloudflare artifacts." And it did. It actually did some really interesting research. And instead of just like searching the internet, which it it did do, it also is connected to Kodi. And so it's able to just figure it out for itself. And so ultimately what it ended up doing was it ran a test. It created a like throwaway repo and tried to upload a 30 megabyte file which succeeded. It just took a long time. The 31 megabyte file uh that failed a after 4 minutes with a 500. And then even larger files got 413s out of that. And then the get LFS push that failed completely. And so u the conclusion was no there's no LFS support in uh Artifacts and let's take a look at the launch blog post. It mentions that could be an extension eventually and so I reached out to my

19:01 contacts over at Cloudflare and said hey let me know when you're ready to test this but right now we've got this practical limit at about 30 megabytes and I think that's actually pretty reasonable. Most people probably won't be going that much bigger. Uh and unfortunately there's no nice error

19:15 message from Cloudflare. And so what this means for me is I have to add some sort of progressively disclosed useful error message that I can send to it when I detect that it's uh that users are trying to upload something that's much too big and kind of redirecting them to other places that they can host these larger files and then there can just be references in their repos. Other thing that I think is really interesting is that the agent is actually now being price conscious. And I think it was only price conscious because uh I was being price conscious in my prompts over to it. And so the product sense that you kind of feed into the agent just by the

Converging on unified primitives and execution

19:53 questions that you're asking to the prompt will change the agents behavior. So keep that in mind as you're talking to your agent. Don't just tell the agent what you want it to do, but explain to it why you want to do those things and it the output will be improved from that. So, we're getting toward the end of this conversation and I realized, you know what? I think I I've got a good idea of what the shape of this primitive needs to be. Instead of making a separate new primitive for uh a data repo and then we have a package repo, let's make a single primitive that is a repo and then there's just a marker on whether that repo is a package. And so, I I gave it this. I said, "What do you think about that?" And it of course agrees with me. not syphantly because sometimes it will disagree um but based on the actual conversation that we've had so far and the way that things are structured and the way that I've actually used it all of that context feeding this actually good product decision that we're making together and then along with that it gave me a number of things that I should be thinking

20:53 about and um make decisions on. And so I say yes, yes, yes, all of these things, that sounds good. And at this point, I'm thinking, okay, we've got a pretty good, solid, shared understanding of how we want this to work. Uh but then I had one other thought, hey, I'm concerned that agents may push up a repo and forget to mark it as a package. How do we mitigate this in a progressively disclosed way? Now, this is the sort of thing that you really only get from having experience in working in the product and um being thoughtful about the way that this product is going to be used. And so, um, we h have this conversation and we get to the point where, yeah, let's fold that into the appro, uh, appropriate work. Wait till I say go. And then at the beginning of the video, I said go.

21:37 And now it's still working all on all of this stuff, all green, and it's validating. It's going to push the PR and market is ready, all this good stuff. And we'll see this in production before you see this video. So, if you have Kodi, then go ahead and give it a try. And I want to mention something. I actually don't use a lot of skills for this sort of thing. Maybe eventually I will turn this into a skill, but right now I feel like it's just much more natural to talk with the agent and ask it the right questions and that sort of thing. So I do use some skills with the actual like orchestrating the work and conducting and and shipping the PR and all of that stuff. But as far as that's concerned, I actually don't really use a ton of skills for this because it's just kind of a natural conversation. So that's going to do it for us today. Let me tell you about your homework. So, I want you to ask for one research deliverable that could change your mind on something about your product. So, you get a ticket or you've been talking about this product decision that you want to have. I want you to involve the agent in that conversation. ask it what it thinks and uh it's going to come back

22:39 at you with maybe good ideas or maybe bad ideas and you explain to it why those ideas are bad and it maybe will come back with other things that you need to think about uh as you're going through that. Maybe it'll surface some sort of tech debt that you need to fix or something and have it spawn an agent as kind of a side quest to go and fix that. uh or if you're not really prepared to just have the agent go and do it, then you can have the uh agent create an issue so you remember to go through it again. And once you've had this conversation, you come to a point of alignment, then just say go and see what happens. You've been watching Better with Kent, where you and I get better together by learning durable skills. Hopefully, you improved your product sense and and ways that you can think about using agents to help you build better products from this video.

23:24 And if you did, I would please invite you to like, comment, and subscribe, and share this with the people that you care about. With that, that'll be all for today. Thank you so much for getting better with me.