Agentic engineering with Claude Code and Android CLI

5,279 words · about 21 min to read

Embrace agents as a mindset for development

0:05 Hello, I'm Ben Android, developer relations. I'm Stephen, product manager. And today, we're going to talk about making apps with Cloud Code and Android CLI energetically. So I think it's fair to say our role as developers is changing a bit. And things are moving crazy fast. So fast, in fact, that let's be honest, the content in this talk is probably already out of date, especially if you're watching online.

0:33 So we took a forward leaning stance with what we wrote with the info we're presenting. But the demos. The demos they run today, they're grounded in reality, so they should be reliable. Now the other thing is we've used Cloud Code for this talk. But the information, it's agent agnostic. It's more of a mindset. So it should apply to whatever you prefer to use when you're building your apps. So we want you to start thinking about a world where you're managing teams of autonomous agents and how you set them up for success when they fail to implement a feature or something like that, instead of just giving up and going, I can't do this yet.

1:10 We need to step back and think what piece of information was missing and treat it how we treat bugs today. And if you've read anything coming out of the teams that make all these agent harnesses Cloud Code, Codex, anti-gravity, you'll see that this is how their teams actually develop those harnesses. And if we take them as a glimpse into the future, that's probably where we're all heading. So it's probably never been a more exciting time to be a builder, to be an Android developer.

1:39 But if you're like me, you might find it hard to know exactly what to be focusing on. And we as developers, we need tools, and we need workflows that really scale with intelligence so that as a new model extends the frontier, all of our hard work doesn't become redundant. We want each model improvement to lead to more intelligent workflows for agents, which ultimately culminate into better outcomes for our users.

2:13 Today, we'll cover four topics. First, we'll explain why it's so important to set your agents up in environments for them to succeed. Then we'll provide an overview on context and how best to manage it. After that, we'll touch on some of the best tools that we have available for Android developers working inside of agents.

Provide agents with proper environments and tools

1 Highlight · Equipping agents with real developer environments and CLIs

2:37 And finally, we'll make a case for adding loops to your workflows. All right let's dive in. So environment. But what does that actually mean. Well, it's like I was saying we're trying to create a space where our agent can succeed and can succeed without you having to sit there and constantly reprompt it and put it back on track. Because when you do that, that's when you unlock being able to run parallel agents.

3:04 And that's where the real productivity gains come in, when you can have these teams of agents all doing your work for you. But that goes way beyond just having access to the code. They need access to accurate documentation, which includes project specs and rules for how to work in that project. But that's the obvious one. Some other things people are doing. People are hooking their agent up to their team chat space now, not just to kick off agents, but to actually provide the context of how decisions were made that led to the feature they've been tasked to do.

3:37 Because you as a developer, that's the kind of subconscious information you're using when you solve problems. And so it can be helpful for your agent to also previous decision history and pull requests, they should be able to see the comments you're getting on your pull requests, the kind of things that they get pinged for things like that. And then you'll find they can work and fix that up. And also wouldn't expect to be able to write Android code without access to a device or an emulator.

4:04 And so you shouldn't expect your agent to be able to do that either. I know they seem like they're amazing and can do anything, but an emulator really helps. Same with profilers. If you want them to work on performance, they should be able to profile, which involves having access to a real device and any backend services that your app uses. Ideally, your agent would be able to talk to them too. And that's where Android CLI comes in. If you haven't heard of it, it was in the What's New in tools talk as well, but Android CLI is our new CLI for interfacing just with various Android tools.

4:39 It's not an AI product in itself, it's just a command line interface. So it provides you access to things like the Android SDK manager being able to start emulators, a whole bunch of things, and it also connects to Tools and backends like Crashlytics. It works across whatever agent you're using. It just sits in your shelf space. So yeah, you install it at this link. But I think it's easier if we just see it in action.

5:10 All right. So rather than simply explaining why it's so important to set your agents up for success, we wanted to really apply this principle to a production application that we built called sideline. And sideline is an Android application for tennis players that provides them with real time feedback on ball location, player location, court location.

5:36 It provides offline coaching feedback through video replay, and it provides fitness data for things like fastest speed, distance traveled, and longest rallies. It's all powered through custom TFLite models running directly on the Android device and first off, I just want to say we obviously cannot solve problems that we don't know exist.

6:01 So using Android CLI we added Firebase Crashlytics into our developer environment, which allowed for us to find some problems that we didn't even know were occurring. And the first one was that there was an application not responding error or an anr happening on a Pixel 11 Pro Fold device, which didn't reproduce in the emulator.

6:28 So fortunately, Android CLI has support for Android device streaming and what the agent did once we told it to go fix this problem is it created a connection to the device that the anr was taking place on, which was particularly useful for me because at the time, I didn't have a Pixel 11 Pro Fold to even validate the fix on. So by giving the agent the ability to go and get access to the device that it needed to validate the fix, we were ultimately able to expand the capability set of what's possible and to save time.

7:05 I ran all the experiments ahead of time, and I've summarized them here. And without getting too into the details, the issue that we were having was an affinity call having to do with Android 17. It was a quick two line fix which looked good to me. So I created the pull request, merged it, and updated with a new build submitted to play. So we not set up the coding environment with the capabilities that it needed to succeed, we wouldn't have been able to identify a real problem, nor would we have been able to validate a fix.

7:43 So with the 1.0 build can see that the problem has been resolved. However, new issues have arisen and we'll get to them later. All right. Next up, context. So about 1 and 1/2, two years ago,

Optimize context windows and structure workflows

2 Highlight · Managing context windows and avoiding token bloat

8:02 which is not that long a time, but in the AI space, that is a lifetime. There was this race to have larger and larger context Windows. And the theory was the more we could throw at the model, the better results we would get. And we were doing crazy things, we tried to include the entire developer.android.com in the prompt. It didn't really work. But even though context Windows have largely plateaued, model capabilities are still rising.

8:29 So this is Android bench 1.0. Android bench is our benchmark that benchmarks the ability of AI to write Android code. The tasks in it are realistic that the kind of things that you work on. And we can see context Windows plateaued at that Mark. But model capability continues to rise. But context is still a fixed budget and you want to save it for what's important.

8:55 You don't want things in there that aren't helping that exact problem you're working on. And a lot of the agent harnesses aren't actually exposing that full 1 million context window to you because they find it's pointless. The results just get worse when they do that. And there's actually a sweet spot. And it does change depending on the model. But if you can keep your conversations to about 150,000 tokens, that's where you're getting the full power of the model without any context rot.

9:22 But context isn't just what you're typing in. There's actually quite a lot to it behind the scenes that you're not seeing. It's built up of a number of parts versus a system instruction. The system instruction is provided by your agent. It's kind of just a prompt that tells it how to work with this agent, the kind of behavior it should do, but also Workspace information. And like your current working directory, what SDKs you have, any installed skills. We'll talk about that in a second.

9:48 And MCP tools, any tools provided by your agent as well. The model needs to know it can call them. So that takes up context. But the big one is the full message and tool call history is sent with every prompt, every new message you send with your agent. And that can really add up, especially with Gradle. I think as we all Gradle logs are really long, really, really long, very verbose. Most of that is not required.

10:14 So there's a quick tip to watch out for that. You can run Gradle in quiet mode or the bigger agent harnesses. Do truncate this for you. So Cloud Code and Android Studio, we're not sending the full Gradle log up, but if you're using one of the minimal agents because their whole philosophy is send everything to the model. Model knows best. So mini-suite agent sending that full Gradle log and that can get really expensive really fast.

10:41 But how do we stay within this 150,000 magic window. Well, the key is to use a workflow and there are heaps of workflows out there. The one I really like is by a guy called Matt Pocock, and he has a series of skills to help you manage these sessions.

10:57 So it starts with this skill called grill me. What you do is you run this skill when you're first starting either a whole project or just a new feature inside the project, and it gets your agent to ask you a bunch of questions. So you might just say, grill me. I want to build, I don't a new time alarm manager in my app. And what it'll do is it'll start asking questions back to you about what should we implement it in, where does it belong. And it gets all the ambiguity out of the problem and out of your head, down into an issue.

11:28 We then run the next skill, which is kind of writes a big spec doc. There's another skill that then slices it into issues, and we file those in whatever issue manager you prefer to use. And then the idea is each one of those issues should fit roughly in that 150,000 window. And then from there, you pick one issue. Either you're working in the terminal or your Ide, whatever you prefer. Build verify. Loop we clear the context at the end.

11:53 No more conversation history. And we've stayed. We've stayed within the window. All right, next up, let's talk about agents.

Define permanent architecture rules for the codebase

12:02 Agents is where you can provide extra information for your agent. That applies to whatever problem it's working on in this project or in your whole Workspace, if that's what you choose to do. But because it is provided to every prompt, it's important to keep it to things that only apply, ideally all the time. So the kind of things that work are architecture and module rules. They're the kind of things that your agent couldn't have possibly known, especially if you work on a closed source app.

12:29 But a really important to understand just to save it reasoning at the start about where does this feature belong. So you can put things like UI components belong in the design system module. Or another thing that works really well is common code review mistakes or just common things you're picking up that you're having to type in all the time. So in this project we prefer to use Kotlin x, date time, things like that. And I really wanted to provide best practices that always apply.

12:55 But the reality is this changes with every model release. Different models have different behaviors when it comes to extra context. So an example back in the opus 4.6 days, it was really common to have big capital letters that said never do this. And then when opus five came out, Anthropic released a statement saying, actually, capitals don't work that well anymore, just use lowercase. And so this is what I mean. You have to keep on top of it. But some things that have remained pretty universal is to keep it short.

13:22 So keep it under about 250 lines. And this one, this next one is a bit polarizing, because I know a lot of people love to do this, but a study came out in February this year that showed if you let your agent modify its own agents, it degrades performance by about 2% to 3% which it's not that big a deal. But the important part was it increased cost by 20% So we lost a tiny bit of performance and we blew out our costs, all from letting the agent write its own agents.

13:49 A way around this I've seen you can get your agent to write a retrospective at the end of each pull request, and then as a team in your retrospectives, can read them all together and decide what modifications you're going to make, because this file is kind of becoming the key piece of critical team infrastructure. If you're working in big teams with agents. So just real quick, here's the agents for the sideline application that we built. There are really two things that I want to call out here.

14:21 The first is that having a clearly defined architecture will save you tokens and save you on context length. Basically, it allows for your agent to avoid having to constantly look into more of your code base and make more and more inferences, and it provides a very clearly a defined guideline for it to follow. And the second that I wanted to call out is particularly for the closed source applications.

14:51 Obviously if you're closed source, none of the agents have been able LLMs have been able to train on your code base. So in these cases, it's particularly important to just spend that extra time, spend those cycles constantly cleaning up, reviewing, refining, improving your agents because it will benefit you in the long run.

15:16 So if we shouldn't just be putting everything in agents,

Enforce determinism with hooks and skills

3 Highlight · Using hooks and modular skills to control agent behavior

15:20 how are we meant to change the agent or model behavior. Well, the first one is hooks. And hooks are actually really good for the kind of things that people put in. Agents like, say, always run the linter when you're finished writing. That can be a hook. What a hook is your agent is exposing all these events that you can subscribe to. And there's ones for pretty much anything you can think of. But the main ones are start session, stop session, read and write files.

15:46 So instead of writing in the bottom of your agents, always run the linter. Or always run my test suite before saying you're done. You just have a stop hook instead. That can run a deterministic script, and it helps rein in the slight randomness of AI. Another one if you have files that you really don't want your agent to be able to read or write. Say you've got some secrets. File you don't want that being sent up to a server or agents. You don't want your agent to be able to write to it.

16:12 You can have a hook for that. So you have a read or write hook. You get path to file. It's trying to edit and you can just return. No, you can't do that. It's just a way to add determinism. The next one is skills. And there's actually skills to turn your agents into hooks. Now that's a whole other thing. But skills are how to add context only when it's relevant. So agents is for things that are always relevant.

16:39 Skills are for things that are relevant in certain situations, mainly documentation. So skills are made up of three parts. They have a name, they have a description and that's the important part. This description is limited to 1,024 characters. And then they have a body. And the body can be massive. The body can even include other scripts, which is pretty common. You can do quite a lot with skills. Let's have an example or look at an example. This is the Android CLI description. You don't have to read all that.

17:04 Just know that it's like a short description that describes when Android CLI is relevant. And then this is the body. It's massive. So we don't want to load all the time. We only want to load this when we're trying to use Android CLI. And that's what a skill enables us to do. But it's important to be selective about the skills you install. We don't want to just be blindly installing those repos we see online that are hundreds engineering skills, one that is a massive security risk.

17:34 You have no idea what's in those skills. It could be telling your agent to do anything. And we do have a set of Android skills. I promise. We vetted them. They don't say that, but two 100 descriptions times 1,000 characters is 100,000 characters you've just added before you've even started coding. So you're already like creeping up on that window. And so that's why with the Android skills, we recommend just installing for everything the Android CLI base skill.

17:59 And from there Android CLI can search the available skills and install the ones that make sense for your app. Or if you do have a look, it's nice to know what skills are available. You can add the ones manually that makes sense for your project. Say like the Compose skills, the edge to edge skills, those are pretty universal. There are skills to audit skills, of course, so there's skill doctor in Cloud code and that will have a look and tell you what skills you're including that are never activated.

18:25 So you're just wasting that 1,000 character description. And if you're not in Cloud Code, pretty much every agent harness has some form of stat screen or audit screen to do the same thing. So again Android CLI. That's the link. But it has a skills list command. And then say you have navigation three in your project. Navigation three is quite a new library. So a lot of training hasn't happened on it yet. You can add the navigation three skill.

18:52 We often sorry. The next one is MCP. But we often get asked why did we make Android CLI a CLI and not an MCP server. Well, we've been building these CLI things way before I existed, way before the mouse even existed. We were building CLIs. So they're fast, they're composable, and because they run locally in your shell session, they're effectively free to run. And also benchmarks like terminal bench have become the biggest benchmarks when it comes to software engineering.

19:23 And so model makers have been aggressively climbing against terminal usage. So they're really good at using CLIs these days. But that doesn't mean MCP has no place MCP servers are really great for the external services you're using already, especially if they're already hosting an MCP service. You can hook straight into it, and it's an install free way to get extra context into your agent. Things like GitHub, Figma, Slack, or exposing MCP servers.

19:52 And then, things that we've always run in the shell can stay in the shell. So Android commands through Android CLI, and Gradle as well just makes sense to have them local, where they have access to everything you use in your development. And I used to work on performance. I used to always say. It's the one tool you need to solve any performance problem. But the learning curve was so steep. Actually, we just saw in the last talk from Rahul.

20:17 Let's have a look at how I can help improve performance. Thanks, Ben. So in order for our application sideline to work, it needs to run at 30 frames per second and in real time. And as you can see here on this Pixel 10 Pro Fold, we're only achieving about 20 to 22 frames per second. These drop frames are going to be a problem because it means that the application is not going to work as advertised.

20:50 So we're really going to need to fix this with Android CLI. We have the Android profiler command. And when we add it to our coding agents and we instruct it to meet this 30 frame per second requirement, two interesting things take place. The first is that it finds a fix using the Android profiler command. It recorded system traces on the attached Pixel 10 profile device, and it found that its AI threads were running on the slowest cores on the device, and the second is that it used Android docs search interestingly without us explicitly asking it for us it to do so, and pulled the official guidance for the performance hint API.

21:43 What the agent learned to do was to optimize the system schedule, and by doing so, it was able to meet the 30 frames per second requirements. So essentially, what happened is the fix followed the guidance. So with Android CLI the key is through hooking it up with the agent, it helps you find the right tool.

22:10 Even if you're not familiar with that tool for whatever job it is you need to be able to accomplish. So once we run it, you can see that sideline meets the 30 frames per second requirement that users have, and it now works on a Pixel 10 profile. All right. Now we're going to bring it all together with loops.

Implement tiered verification loops for reliable completion

22:35 So I'm sure we've all done this. You ask your agent to implement some big change and it comes back and says it's done. And if you're installing it and manually clicking around on the app yourself and you type back the bugs, that's what we need to stop doing. We need to stop being our agents, QA tester and define up front what we consider done to be. So here's an example. We want to migrate the whole project to nav3 and then the agent goes away.

23:02 It does take a while, but it confidently comes back and says migration complete, I'm the best. But then, of course, you rotate the phone and the app crashes. It missed all the slide transitions. It didn't clean up its Gradle file. And this is the world we're in right now. Ideally, it wouldn't have to do this,

4 Highlight · Boosting task pass rates with verification loops

23:17 but this is the current reality. So we should have defined all of that up front first about what we expect. And that would have allowed the agent to loop until it had met all of those criteria. That's what a verification loop is. And we actually have really strong data on the effectiveness of this. So this is androbench 2.0. It's the new version of our AI benchmark I mentioned back at the start of the talk. Pass rates are currently at about 40% These tasks are really hard and they're really big and realistic.

23:48 But one of them is nav2 nav3. They're the kind of things that you're doing in your projects right now. And if we take fable 5.1, its pass rate is 22% And that's what it gets just with the natural prompts that are like, migrate NAV two to NAV three. But if we give it a failing test suite in advance. So this is not giving it the answers. We're not telling it how to migrate to NAV three. We're just giving it a test suite that is failing already and will pass when all of those things are complete.

24:15 Its pass rates on the whole benchmark jumped to 90% And what this shows us is these models are really capable now of implementing pretty much any Android task we can think of. They just don't always know what done on means, and so you don't have to write a full failing test suite in advance. Verification loops can be as simple as prompt additions. So say you're working in the UI. You can ask it to render the Compose preview and ensure it matches the Figma spec and show me the results.

24:45 It seems to really work when you ask it to show you the results and prove it was done, rather than just accepting the text response, I am done. If you're working in business logic, you can ask it to run your unit tests or we saw at the start, if you're working on a crash fix can tell it to validate the crash is fixed on the real device that it happened via remote test lab. There's actually a skill built into Cloud code to help you build verification loops now, so definitely check that out.

25:13 And if you're working in Android Studio with bring your own agent, Android Studio is exposing a whole bunch of tools that you could build loops around depending on what you're working on. So the linter, you could tell it to not stop until all the lint errors are fixed or the one I really like is journeys. You can ask it to run your journeys and ensure they are still functional, or if you've written a journey. Because journeys are pretty easy to write, they're just prompts about the feature you're working on. You can tell it to run that journey and ensure that it passes before it declares itself done.

25:44 But journeys do take a while to run, so what you can also do is tier the levels of your verification loop so we can ask it to first make sure the unit tests in Compose previews are all rendering correctly. And then when that passes, please run our connector tests and Android journeys. And only when that's passing, run the full remote device test lab. Full test suite is going to take hours kind of thing. And if you give that extra information,

26:08 we find that it loops per tier rather than trying to do each step at each run, because models are really confident. So they always think they are complete, and they always think they're going to pass all the tests first. So let's have a look at that in sideline. So let's apply this idea to our sideline application. And let's do this by adding a new feature specifically for the XR glasses, which are coming soon.

26:38 And since we don't have hardware yet, of course, we'll use the XR Emulator and instruct our coding agent to extend sideline for the new form factor. When we do that, we're going to specifically ask our agents to loop through JVM tests Compose preview tests, and emulator tests in that order. And each loop is told to build on the loop prior to it.

27:09 And when we do this, the first thing we notice is that between loop 1 and 2, it's the JVM tests that are used to define the six Compose preview states. And of course, no emulator was called. So that runs very fast. And then similarly between loop 2 and loop 3, issues like long strings are surfaced and fixed even before the emulator has to run.

27:34 So finally, once the emulator loop runs, it finds additional bugs that the JVM tests and the Compose previews couldn't fix. But essentially, with each loop, we're further hardening our application, and we want to run the fastest but least powerful loops first, followed by the slowest but most powerful. And here's the outcome. It shows that the glasses now show this Swing type and the speed of the ball coming off of the racquet, so that you can be playing tennis and you can be getting feedback on your gameplay in real time without taking your eye off the ball.

28:18 Oh, and I mentioned earlier that we would get back to those two Crashlytics problems that had arisen before. So since we've already been able to validate that approach worked, we can use the schedule command in Cloud code to loop through it, and we can set any cadence we want so that now we have a self-healing application that fixes all the bugs and validates that they're fixed, sends them as pull requests to our engineers to take a look at and to merge.

28:55 Cool So that's pretty much it for today. If you take one key takeaway, it's to start thinking about this world where we're building an environment for our agents to succeed. I'm treating that how we treat software development and bugs today, and that's how we unlock parallel agents. You're going to find a lot of advice on this topic online, and a lot of it's going to conflict with each other. And it's just the reality of this fast moving environment.

29:20 So the best thing you can do is write your own evals for your project, and then when some new advice comes out or some new model you can actually measure it and see, does it help you. And of course, there's a skill for everything. There's a skill to help you build evals in Cloud Code. Definitely recommend checking that out. I talked about Android bench today. Android bench is our eval. It's how we measure to make sure things are getting better. If you want to hear the lessons we learned building that so you don't make the same mistakes, check out the Android bench talk.

29:49 There's also two panels. There's one in the US, there's one in London. They're both going up online that talk about this kind of concept in practice in the community. So we've got Android dev community members coming onto the panel to talk about how they're putting this into practice at their jobs. We really want your feedback. We really want to hear if we're building the things that are actually helping you. So please fill out the survey. I'll leave that up on screen. And yeah, thank you for listening.

30:15 Thank you.