Agentic coding workflows with small models

Small language models complement frontier models in developer workflows, offering cost-efficient local execution when properly configured with quantization, expanded context, and task orchestration.

Android Developers · 31:32 · YouTube ↗

Talk · 3 speakers Intermediate · concepts and setup strategy

Local models Coding agents Agent harnesses Gemini

Highlights

5 clips · 6 min of 32 min

- Post-training quantization shrinks model weights but harms output accuracy.
- Quantization-aware training minimizes accuracy loss during model compression.
- Mixed mobile quantization fits large models onto consumer mobile hardware.
- A smaller drafter model predicts tokens verified by a larger model using a shared KV cache.
- Delivers up to a 3x speedup with zero quality loss.
- Highly effective for predictable structured outputs like JSON and code.
- Small context windows cause agent harnesses to evict critical files during compaction.
- Increasing context from 32k to 256k doubled task success rates in Android bench tests.
- Maximize the model's supported context window rather than relying on default limits.
- Add explicit verification hints for unit tests, emulators, and build commands.
- Avoid bloated system prompts and massive tool descriptions that degrade small model performance.
- Set up fallback models to avoid downtime, rate limits, or offline blocks.
- Use small models for execution with advisor or peer-review models for validation.
- Cascade tasks from frontier planning models to small execution models for cost efficiency.

Questions this video answers

9 · every answer cites the moment it was said

01 How do small language models compare to frontier models in cost and capability? 2 sources · 1:28

Small language models are significantly more cost-effective, running datasets at a fraction of the cost of frontier models. While frontier models handle complex tasks easily, small models can solve a substantial percentage of common tasks when properly guided.

02 What are the primary reasons to choose a local small model over a cloud API? 2 sources · 6:18

Developers choose local models for data privacy, zero latency without round trips to servers, zero marginal cost during error loops, and offline access. These benefits make local execution ideal for sensitive code and disconnected environments.

03 How does quantization reduce memory requirements for running models locally? 2 sources · 7:31

Quantization shrinks the precision of model weights, such as compressing a 16-bit floating-point model down to a 4-bit integer format. This dramatically reduces video RAM requirements so the model can fit on standard consumer hardware.

04 What is speculative decoding and how does it speed up generation? 2 sources · 8:58

Speculative decoding uses a smaller drafter model to predict a few upcoming tokens ahead of time, which are then validated by the larger model. Because they share the same context cache, this technique achieves significant speedups without degrading output quality.

05 Why is expanding the context window important for small model performance? 2 sources · 10:43

Small models can fail tasks if important files or logs are pushed out of memory by agent compaction. Increasing the context window ensures the model retains critical file information and significantly improves its success rate.

06 How can developers evaluate whether a model configuration is ready for their tasks? 2 sources · 12:10

Developers can use a readiness framework that scores tasks based on reliability, latency thresholds, and RAM usage alongside the model and hardware configuration. Green scores indicate the setup is ready, while other colors suggest adjustments are needed.

07 What types of coding tasks are small models best suited for? 2 sources · 20:22

Small models excel at small tasks like basic Compose layouts and medium tasks that require reading and writing just a few files. However, they struggle with long horizon tasks like massive migrations that span dozens of files.

08 What adjustments can developers make to help small models succeed on tasks? 2 sources · 24:31

Developers can improve small model success by ensuring full context windows are enabled, using CLI tools for niche documentation, and adding verification hints for tests. Avoiding bloated system prompts also prevents performance degradation.

09 How does model orchestration provide resilience and efficiency in workflows? 2 sources · 27:54

Orchestration allows developers to set up fallback models to prevent workflow blocks during provider outages or offline scenarios. It also enables cascading tasks, routing simpler jobs to smaller models and complex planning to frontier models for cost efficiency.

Section-by-section summary

6 parts

Small language models complement frontier models efficiently

Small language models offer a highly cost-efficient alternative to frontier models for many standard coding tasks. While they solve roughly 50 percent of medium-to-hard tasks out of the box, they consume a fraction of the tokens and cost significantly less.

Understanding small model types and architectures

The small model ecosystem includes autoregressive models for token prediction, diffusion models for block generation, and embedding models for semantic mapping. Specialized models also exist for specific jobs like function calling, classification, and vision.

Evaluating local model hardware and quantization trade-offs

Running models locally provides privacy, zero marginal cost, offline access, and zero network latency. Hardware deployment relies on quantization and quantization-aware training to drastically reduce RAM footprints without sacrificing output accuracy.

Trying local models directly in Android Studio

Developers can easily integrate and test models like Gemma locally using Android Studio's model provider settings. Switching models mid-conversation resets context, so managing conversation state is critical when swapping backends.

Task categorization using Android bench evaluations

Coding tasks are categorized into small, medium, big, and long-horizon based on scope and complexity. While small models excel at localized edits and basic Compose tasks, they often skip verification steps and require targeted assistance for broader architectural changes.

Orchestrating models for resilience and cost savings

Advanced orchestration prevents downtime via fallbacks and improves answers through peer review or consensus workflows. Routing initial planning through frontier models while cascading execution to small models maximizes token and cost efficiency.

0:00 / 31:32

k play/pause · f fullscreen · m mute
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said

Small language models15× · first 0:28

Language models with smaller parameter sizes used to complement frontier models in developer workflows.
Quantization4× · first 7:25

A process that shrinks model precision to reduce memory footprints and fit on local hardware.
Speculative decoding1× · first 9:00

A technique using a smaller drafter model to predict tokens and a larger model to validate them for speedups.
Android Studio5× · first 2:27

An integrated development environment where local models like Gemma can be run directly.
Android bench4× · first 10:35

A benchmark capturing the Android ecosystem and developer tasks for agent evaluation.
Context window7× · first 5:31

The space available for processing data and maintaining model memory during tasks.
Embedding models1× · first 4:14

Specialized models that turn concepts into high-dimensional vectors for similarity comparisons.
Quantization aware training1× · first 8:00

A technique where models are trained to handle smaller sizes to prevent accuracy loss.

5,391 words · about 22 min to read

Small language models complement frontier models efficiently

All right. Hi everyone. My name is Josie and I'm here with Ian and Zoe. We all know about LLMs. We use them for everything. Finding recipes, summarizing YouTube videos and coding. There are the really big LLMs, such as the Gemini models that are capable of truly incredible things. But in this talk, we are going to talk about small language models.

They are also LLMs because something that has four billion of something cannot be small. But we'll make this distinction from now on. Frontier models and small language models. The rocket and the bike. We'll discuss small models, not necessarily as an alternative to frontier models, but mostly to complement them. We'll explore all the benefits of small language models, especially when they are open and open weights and downloadable.

But let's look at cost for a second. For this talk, I've been running lots of benchmarks, and one of them was a data set of 200 medium, and hard tasks, using almost 131 million tokens for less than $9. I used semaphore for this, so I could have just downloaded it and run it on my own hardware for free, but I run it on a commercial gateway just for a fair comparison.

So how much would a frontier model cost. Around 20 to 200 times more, depending on token caching and other factors. So the bicycle might not be the fastest means of transportation, but it's definitely the most efficient. It's true. So now the obvious question is, what's the difference in quality and accuracy in this particular data set. Semaphore can solve 53% of the tasks.

This means two things. If you're using a frontier model to run these tasks on the left, it might be overkill. And if you're using a small model to run these tasks on the right, you'll have to guide the model towards a solution or use a larger one. In this talk, we want to help you get more out of every token. So we will walk you through how to choose a small model.

And then we'll talk about the models so that you understand how to frame your thinking about what they can enable. Then I'll quickly show you how to try Gemma for in Android Studio. And finally, we'll show you how to fit small models into your daily workflows and also real Android development benchmarks to help you build an intuition for what they can do.

Understanding small model types and architectures

With that, over to showy. Thank you so much, Josie. So using one model for everything is a bit like cracking a nut with a sledgehammer. It is efficient, but it's a little bit disproportionate. So when you think about small models, it's almost like you're expanding your toolset to know what you have to bring into your workflows, you need to open the toolset. See what's available. What are their specs. What are they good for. So to choose small models, you need to know small models and the type of model that most people think of whenever they're thinking of a language model is an autoregressive language model.

This is autoregressive because based on an input chunked in tokens, it chooses the most likely next token out of a list of candidates. And then that most likely next token that has been chosen goes back into the input, and on and on we go. Which is capable of producing coherent and intelligent sounding responses. But it does have some caveats, including but not limited to working linearly so we can work non-linearly a diffusion model has the same approach, but it works a block of tokens at a time.

It's almost like a sketch that's coming into focus, and in this case, diffusion. Gemma works on chunks up to 156 tokens at a time. So basically get a pass that works on the whole block, being able to generate results with less latency. OK, our tool set has models that can write right. So basically a embedding models essentially turns objects like an image or text or any other concept into a vector.

So that provides us a mapping of let's just say coordinates into a high dimensional space. That allows us to compare two things conceptually. So for example, if a hot dog is a sandwich, they will likely lead close or end up in a close space because of this. Being able to check whether two concepts are related becomes let's say, a little bit of math instead of a model pass, which arguably is also a bit of math.

So this paradigm goes even further. And there are smaller and specialized models that are capable of doing one job. And they do it really well. Some examples here is, for example, you have function models that turn natural language into API calls, vision models that I'm sure you know what they are capable of doing by the name. Or for example, classifiers that can give scores to an input based on certain parameters.

OK, so there is a lot to choose from right. And there's a lot of different types of models. And they have different dimensions different sizes, different context Windows, different capabilities, different memory footprints. And once you open the toolbox and you account for every possibility, well, the options grow exponentially to the point that any DIY fan would be absolutely ecstatic or frozen by decision paralysis.

So how do you actually choose a small model.

Evaluating local model hardware and quantization trade-offs

Let's start by opening a toolbox of open, small models that can run locally. The demo models. Over to you Ian. Thank you Zoe. So as Jose mentioned at the start, a small language model can do maybe about 50% of tasks out of the box. But why specifically may you choose a local model. So it comes down to about four things. So the first one is privacy. So for instance, you might have for instance API keys, proprietary code or things that just literally cannot leave your laptop.

The second one is latency. So when the model is actually loaded into memory, either graphics memory or unified memory, then you get a response as soon as it's finished computing. So no round trip times the servers, no additional latency costs there. Next is 0 marginal cost. So models are cost effective when you get to control where they run. So in the case of running something on a local machine for instance, it might be doing additional retries in an error loop.

It doesn't cost you anything and there's no kind of surprise API cost part of that. And then finally, the fourth one is offline access. So imagine you can use this on a plane or a train or a bus or in the tube, basically anywhere where you have the model running on your hardware. So that sounds great. But there are two major factors that we need to consider when making these decisions. So one is memory. And then the second one is speed. So if you look at this diagram here, if you look at the two bottom models, the 26b and the ME31B,

1 Highlight · Quantization-aware training reduces memory overhead

they both take over gigabytes of video RAM to run in bf16, which is a 16-bit quantization. So the challenge here is that it wouldn't fit on a lot of hardware. So what we do is we do a process called quantization that shrinks that precision down to say a 4-bit into a float. Now it's a bit more reasonable. We're talking about 14gb and 17gb, right. That would now fit on something like a unified memory map.

Now the challenge with this is in normal post-training quantization. What will happen is as you compress those weights, you lose a bit of the accuracy in the model from the base model. And we have a technique to combat that. It's called quantization aware training. And what happens when we do that is as the model is trained, it is trained to know that it's going to be on these smaller sizes, and therefore there's actually less loss compared to that. So we get a 75% reduction in RAM requirements.

And now it can fit on those devices. And we go one step further. So the mobile quantization approaches that we have that work on a lot of mobile phones can shrink those models down even further by understanding how all the different layers operate and what sizes they can operate at. So we have a mixed quantization bag. So we go from an 11.4 gem to be down to a 1.1gb. So now it's actually much more reasonable on mid to high end phones.

The next is speed. So when you use an autoregressive model, as I mentioned, you're predicting

2 Highlight · Speculative decoding accelerates autoregressive generation

the next set of tokens. So in a coding task you might find a lot of these tokens are actually quite predictable, it's the end of a method signature, or it's an opening or closing brace. So what you can do is you can use a technique called speculative decoding. And what that will do is it will use a smaller drafter model to predict a number of tokens, which then get validated by a bigger model. And the reason this works in the Gemma 4 architecture is that they share the same CVE cache.

So think of this as the model's kind of short term memory. So what will happen is the small drafter model will go predict three or four tokens ahead of time. And the bigger model will then look at that same CVE cache and will basically validate the output. And what this can result in is about 3x speedup in terms of performance, with no actual loss of quality of the output of the model, because it's just predicting things and then deciding whether it's pass or fail, whether this is correct or not. So again, it depends a little bit on the task you're doing.

So this doesn't work so well on things like creative writing because there's more variability. But it does work very well on predictable things like JSON structures and coding. So how would you actually evaluate one of these models.

So the most obvious thing you might do is go look at a benchmark like live Code Bench or MMLU, and you look at a score, but that will give you only a high level indicator about the model's capability. What you really need to do is evaluate the whole stack in which it's running. So this is the model that you're using the weights. And the quantization approach is the inference of the runtime. So you might use something like Llama CPP or Ollama or Lite ht.

3 Highlight · Expanding context windows prevents agent memory loss

The agent harness that you're using, whether that's Pi or open code or anti-gravity. And then the context window, which is the space for all the data that you want your model to be able to process. And a combination of all these things results in with this kind of viable. So to give you a quick example, we were doing some testing with Gemma for on Android bench, and we were running an older version with a default 32k context. And what we were finding is in one of the tests where it was looking for a bug, the compaction of the agent harness would move that file out of memory, and then the model wouldn't be able to complete the task and it would fail.

So just by increasing the context from 32k to 256k, now it's able to keep that in memory. And the model is able to pass that particular test.

And we saw a double. We saw a doubling of the success rate on that particular job. So just goes to show that even one of these small factors can have a big impact as to whether this model is suitable or not. So how do you actually do your own evaluations. So here's three quick suggestions. So one is you could take a Git history. So a bunch of PRS that you've already got that are successful. And you could use them as prompt inputs to the model and see how well it performs on those.

The next is you need something like a deterministic grader that will help you evaluate whether the output of the model is correct or not. We've got two really good examples of that. We've got a compiler and we've got unit tests. Whatever gets generated, you can run it through that. And it will tell you pass/fail as to whether you've got a success out of that particular task. And then the third one is log trajectories. So when you're running an agent, you want to be able to trace the different behaviors that the agent has and see where it succeeds and where it fails. And you can use that to evaluate how well the model is able to interact and use tools.

And I want you to try and change your framing of the question from is this model as smart as this big frontier model to is this task possible on this hardware with this particular configuration. And to figure that out, we invented a framework called are there yet or and the whole principle of this framework is to score on that axis all these different tasks in terms of readiness.

So the model itself, the inference engine that you're using the harness and then the hardware configuration, and that will tell you whether you're ready yet. So for instance, you might be looking at how reliable that particular task is if you run it 10 times in a row, does it do the same thing each time. You might be looking at the latency. Maybe you need it to be a real time response. So what are your thresholds for the speed of doing the task. And then you might be actually looking at the RAM requirements.

So it's not just the RAM that the device has, it's the RAM that the device, the carve out that you can take, and the other applications and other tooling that you'll also need to run on that hardware. So can you take up 50% or can you only take up 10% of the RAM and you get this final kind of readiness score. So if it's green, that means that configuration is ready to go. If it's orange or yellow and/or red, that means maybe you need to change something about the configuration.

Maybe you need a faster accelerator for lower latency. Maybe you need a bigger model that's more capable. So yeah, I suggest having a look at this. We've got a preview at the moment, and we'd love to get your feedback on how useful this tool is.

Trying local models directly in Android Studio

So Zoe's going to cover a bit more about how we set up some of the evaluations later, but I'm going to now hand over to Josie, who's going to show you a very quick way to get started with gemmifer. Thank you, So yeah, the fastest way to try Gemma for today is probably through Android Studio. So let me show you how. Let's switch to the laptop, please. There you go. So if I go to the model providers page in the settings and I select Gemma, this gives you a pretty clear idea of how big each model is.

I just have to download the models onto my hard drive. So this opens or starts a server. And it's correctly configured for you from the get go. But you can also feel free to use your own server or llama or LLM Studio, whatever you want. For that just come here and add a local provider.

And also it makes a lot of sense to use a small models through a remote provider. So you can add one here. As long as it's OpenAI or Anthropic compatible, you can connect to it. All right. So let's see. So now. Oh, by the way, so the way you select a model in Android Studio is through this dropdown obviously. So I have a M426 be selected.

And I can ask what is a view model. So one very important thing you need to remember is that whenever you change model, the whole conversation is sent again to the new model from the beginning. So if you generate a really long conversation with a model and you want to switch to a new one, you should probably start a new conversation or compact it or whatever. So now back to the slides and to showy to learn about the different workflows you can run with small models.

Thank you so much again. And thank you, Ian, for the overview of the models and whether we're there yet or not. So we've seen what types of models

Task categorization using Android bench evaluations

are there, what their specs are. And now it's a matter of understanding what are some types of jobs that you can put them to do. The first type of jobs that likely you're already doing are coding workflows like debugging or troubleshooting or something that I'm sure everybody or at least I will very deeply, sarcastically miss doing by hand, which is increasing test coverage. So these are some of the things that I'm pretty sure everybody's already doing, even likely without realizing.

But I would like to encourage you or I would I'd want to encourage you to think beyond that. So considering that small models can be open weight and run locally, what is something that could be giving you back time. We're talking about not using the rocket. So why are we using a brain for that. So if we could go beyond that, what are some things that small models could do.

What are some workflows that you could bring them into that could make your life easier and give you back more capacity to do things that maybe you want to do more of. So, for example, something like making sure that your documentation is always up to date with the changes that you made, or something like ensuring that you have automatic regression tests that are part of a pre-commit hook, or even making sure that the context that your agent is receiving as part of your workflows is optimized for it, and it works even faster.

So these are some examples. And at the end of the day, the sky is the limit to some extent or architecture is the limit. And there are a lot of things that you could think of how to implement it. So having the specs, having the tools, having the workflows, are you still not sure where to begin. Well, that's what benchmarks are for.

And as Ian said, a generic leaderboard, especially one for Python, is going to tell you very little about whether a small model is going to be helpful for your Android workflows. Just very much like a drywall drill might not tell you much about a concrete drill, but you should invest in your own evals. You should invest in testing your own architecture. But before you do that might not want to put all the resources.

It takes time, it takes money, it takes effort. So looking into publicly available benchmarks can give you an idea and a short list before you decide to own evals. And there's three things that you should look at whenever you evaluate the benchmark. It's a bit meta, but basically the first thing is what is the type of task that this model is doing. How specified it is. Does it look anything the type of task that you will give it.

The second one would be the scope of the task. So is it writing documentation. Is it resolving a PR. Is it something that looks the type of job that you have. And finally, is the complexity of the system akin to the complexity that you're going to give. And here is where we can see why a Python benchmark is not going to be very representative of Android.

That's why we built Android bench, which is a benchmark that captures the Android ecosystem, and Android work. The first version has hundreds PR sized Android tasks, and you can run it yourselves today through Harper to check any agent, model and skill combination. We launched 2.0 with more complex long running tasks to understand what is the capabilities of frontier models to work in jobs like the ones that you give it today.

So let me hand it over to Josie now to show you the results that we got through this and some ways that you can optimize working with small models. OK thank you. Zoe all right. So let's look at the tasks that a small model can solve and where they start to struggle. And let's arbitrarily define four categories of tasks small medium big and long horizon long horizon tasks as are tasks that might take a human days or weeks to complete.

For example, a migration, a big refactoring, and spoiler alert you cannot do those with a small model, at least in one go. We'll talk about that later. OK small tasks. They are jobs that require reading and writing, maybe one or two files and they don't require much knowledge. For example, small models are pretty good at Compose at least the basics of Compose. So things like there must be apps between elements or use a theme instead of hard coded colors or abstract strings.

Those are very easy for a small model. This one, however, covers a more advanced topic, but the task is still easy. The difficulty in this case is about scope, not about knowledge. So the prompt actually tells us which composable to optimize and what's wrong with it. It's recomposing on every frame, so this makes the job pretty easy, even for a small model such as jm4.

It took in this case three minutes and it is a one line change. All right. So I think we have a pretty good idea of what a small model is. Let's look at sorry a small task is let's look at medium tasks. Small models should be able to do most of them in this task from the app pocketcasts, the user agent is missing from in a request. And also there's an error in the retry logic.

So this one is a little bit harder because first we're requesting two things in this at the same time. And there are no hints about where these things live in the code base. But I passed. It took 10 minutes. It modified three files. And by the way, it costs $0.06 through open router. Now let's look at a task that failed from the same data set. This is an example of now in Android with a pretty comprehensive report or prompt.

But it doesn't mention a class name or a file. So the agent needs to spend more effort into understanding the architecture of the app, the data layer, the sync mechanism in this case, et cetera. So in this particular run it failed. Everything was implemented almost correctly, but it just assumed that the value of the initial sync would be minus 1. But it's actually 0. That's it.

This bug would have been caught if the agent had just run the unit test the existing unit tests. So this shows how smaller models in general tend to stop a little bit earlier than frontier ones. We just ask the agent to fix the issue, not to verify it. So maybe we should modify the task so that it contains a verification step. We'll talk about that in a minute as well. So all right let's talk about big tasks.

About half of the Big tasks in this definition pass every time. Some of the ones that fail are actually too long and complex for a small model. Not much to do there. But there are some that when you look at what actually happened realize that the model was 90 or 90% there. So let's pick one of them. I picked this one because it's a flaky one. It passes around half of the time that it's run. So I consider this to be at the frontier of what a small model can do, or at least a model from April.

It's about parsing malformed URLs in the coil library. As you can see, the PR description is pretty comprehensive, which is great, but it missed a requirement. So let's quickly look at it all runs checked for typical URL encodings and errors, but only some half of them add added tests for badly encoded characters so that test case they're. The ones that didn't failed the evaluation. So all runs are almost perfect.

But this goes to show, again that small models don't always go the extra step. They follow instructions, but they don't try to impress you like frontier models. Do you know how you can set the effort in an agent nowadays. Just consider that it's set to low when you use a small model. So let's define big tasks as those that might be completed by small models around 50% of the time, which is what gma500 b-1b can reach against the Android bench V1 data set, which is what showy talked about before.

So before talking about long horizon tasks, can we do anything to help small models with those tasks. Absolutely and these things don't require any effort at all. First, if you use a model such as jm4 locally, as Ian mentioned, make sure that it's configured correctly. For example, if you use a llama to serve. Gemma for it defaults to a context window of 32,000 tokens when it actually supports eight times more.

4 Highlight · Optimizing small model prompts and tool descriptions

Of course, if you use Gemma for through Android Studio, you don't have to configure it. It comes correctly configured by default in our set of evals. That made the enabling the full context. Made the results go from 26 to 53% Next, use Android CLI. The Android CLI tool, especially for searching documentation. This doesn't always apply as models are pretty good at the basics.

But whenever you use a new or niche API, having the documentation is key. Much better than using web search tools in our experience. Next, add verification hints. We talked about this. Tell the model how to run tests, what types types of tests are available, how to manage emulators, et cetera. You can do this in an agent's file, which would make the instructions available to all agents. But I mean, that's a good thing anyway, you should have it.

But also watch out for Super long agents files, because if you try different agents, you'll see that some of them have massive system prompts and tool descriptions, and they seem to degrade the performance of small models. We actually tried running these benchmarks with open code, and it performed a lot lower than Pi, probably because it uses bigger tool descriptions and system prompts.

Next, add scales, but also remove scales. Small models might need might need a nudge when

using new and niche APIs, but it's better not to fill their context with general Compose scales, for example. So what about long horizon tasks. They are tasks that span dozens of files, require serious architectural decisions, and are normally handled by a thinking model that creates a plan and starts a swarm of subagents that execute the different tasks.

Examples of long horizon tasks would be migrating a whole app from XML to composed, creating a complex app from scratch, or converting cross-platform apps to Native Android. If we look at the results from Android V2, which focuses exclusively on Long horizon, we see that frontier models are scoring 30% to 40% So for small models, I think it's safe to say it's out of their league.

However, routing the initial analysis through a frontier model and then cascading simpler tasks to smaller models would be far more token and credit efficient.

Orchestrating models for resilience and cost savings

Let's see how we can do this. Back to showy. Thank you so much for saying we'll get to the cascade. But before I show you some ways or before we deep dive into this, I want to take a step back and show you other ways in which orchestration of different models can help you. So orchestration can, for example, help you never be blocked. So whether you're seeing a provider go down, or whether you've hit an API rate limit or whether you are on a plane or a train, like Ian was saying can set up a fallback model and have make it so that whether that is online or locally, you can keep going if anything happens.

5 Highlight · Model orchestration for fallback and cost efficiency

The other thing that you could do is orchestration is get better answers. For example, you could have a small model work on its own and call an advisor whenever it needs a little bit of support to get that help. You can also set up a Console in which different types of models or different models are all executing the same task at once. And then you get to evaluate the different responses and how the weights and biases of each model have affected the outcome.

You could also use a separate model as the let's just say as the tests. So it's like a peer review and you could set it through this. And of course, finally orchestration can help you be cost efficient through something like that cascade. Basically, you can set it up manually. So you could set one model for planning. You could set a different model for execution, another one for tests.

There's multiple ways to do it, either on the Cloud or you could set up a proxy to do it locally, or you could just use a classifier

to route it automatically to the right type of model based on the request, which is the proxy idea that Ian mentioned, but automatic. So let's wrap up by going back to the question that Josie asked at the beginning of the talk, which is what's the difference in quality and accuracy between small models and frontier models. And the answer with, as with many things in computer science, is it depends.

However, this is something that I would like to bring back with you is that this is something you can engineer. This is something that you can make a difference in depending on how you set it up and depending how you play with the pieces. So the first piece here is pick the right tool. There's multiple types of tools. Get to know them, explore them right and then find out what jobs are they good for or even which jobs they could be doing.

The second one is make sure to adjust the scope that the model is getting in order to do the job. So, for example, things like the instruction for the task, things like providing the right context for the task and only the right context for the task, and ways to check its own work. Then, of course, you need to test the whole setup and not just the model in isolation, because this is going to make a difference.

And maybe the results that you're getting could be tweaked if you set it up with, let's just say, a different configuration of elements. And at the end of the day, you need to work on setting up or creating your own evals. And finally, you can orchestrate the bicycle and the rocket so that you never have to choose between one or the other and spend effectively with this.

But most importantly, you can continue to explore, continue to build, continue to have fun, right. And you can begin that today by setting up a short list of models through Android bench and the findings you have there, or starting with Gemma for on Android Studio today. So thank you very much.

Thank you.