Small, quantized and on-device models: Gemma, Llama, phones and laptops.
Developers can create a custom task, such as generating an SVG of a pelican on a bicycle, to test text models' code-output capabilities. This practical benchmark avoids untrusted leaderboards and tests models on an intentionally difficult and absurd task.
▶ 1:12
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
An inference engine takes static model weights and turns them into a system that can serve requests reliably and at scale. It manages GPU resources, schedules incoming requests, and decides how to process work together. Examples of inference engines include vLLM, Ollama, and Llama CPP.
▶ 1:16
What Is LLM Inference? How AI Models Generate Responses
Giant models often lack native perception capabilities because they rely on text-only architectures. Without integrated sensory components, they cannot process raw visual inputs directly.
▶ 0:06
DeepMind Just Changed How AI Sees The World
Small language models are significantly more cost-effective, running datasets at a fraction of the cost of frontier models. While frontier models handle complex tasks easily, small models can solve a substantial percentage of common tasks when properly guided.
▶ 1:28
Agentic coding workflows with small models
A language model is essentially a function that predicts the next likely word or token in a sentence based on its training. By predicting successive words, it constructs coherent continuations or fills in missing parts of text. During this process, it evaluates probabilities for various candidate words to determine the most fitting output.
▶ 1:33
A Hackers' Guide to Language Models
Llama 3.3 70B provides GPT-4 class capabilities that can fit entirely within the RAM of a standard high-end consumer Mac. Meta matched the performance of their massive 405B model in a 70B parameter size that developers can run locally.
▶ 2:44
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
Most inference engines expose standard OpenAI compatible endpoints so developers can use the same request format regardless of the underlying model. This makes it easy to swap one model for another in an application without rewriting any code.
▶ 1:45
What Is LLM Inference? How AI Models Generate Responses
Gemma 4 is an open and free model created by DeepMind that runs locally on laptops. Compared to massive trillion-parameter systems, it is a tiny fraction of the size yet capable of native perception.
▶ 0:50
DeepMind Just Changed How AI Sees The World
Developers choose local models for data privacy, zero latency without round trips to servers, zero marginal cost during error loops, and offline access. These benefits make local execution ideal for sensitive code and disconnected environments.
▶ 6:18
Agentic coding workflows with small models
Language models tokenize text into sub-word units, whole words, punctuation, or numbers using specific tokenizers like tick token. These tokens are mapped to numeric IDs that act as lookups into the model's vocabulary. For instance, prefixes like spaces are often encoded directly into the token representation.
▶ 4:45
A Hackers' Guide to Language Models
DeepSeek V3 was released unexpectedly as an open weights model directly on Hugging Face. The accompanying paper claimed a training cost of only five and a half million dollars, which is drastically lower than expected for a model of that size.
▶ 3:32
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
Tokenization breaks text into tokens and maps each one to an arbitrary token ID that the model processes. This step dictates your prompt cost because you are billed per token. While larger vocabularies represent text more efficiently with fewer tokens, they introduce computational overhead during training.
▶ 2:15
What Is LLM Inference? How AI Models Generate Responses
Conventional setups connect multiple separate neural networks together. They require a dedicated visual model for images and a specialized audio encoder for sound.
▶ 1:48
DeepMind Just Changed How AI Sees The World
Quantization shrinks the precision of model weights, such as compressing a 16-bit floating-point model down to a 4-bit integer format. This dramatically reduces video RAM requirements so the model can fit on standard consumer hardware.
▶ 7:31
Agentic coding workflows with small models
Pre-training forces a deep neural network to guess the next word across vast corpuses like Wikipedia, utilizing stochastic gradient descent to optimize its weights. To succeed at this task, the model must implicitly learn complex real-world concepts, entities, and relationships. This capability essentially acts as a form of information compression.
▶ 7:40
A Hackers' Guide to Language Models
The prices of top-tier model APIs have crashed dramatically over the past few years, dropping by a factor of 500 times or more. While some flagship models remain very expensive, efficient scaling has driven down the cost of highly capable models.
▶ 7:59
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
Processing individual requests one by one wastes enormous amounts of GPU resources and tanks system throughput. Continuous batching groups requests together and immediately slots a new request in as soon as a previous one finishes. This mechanism keeps GPUs running near full utilization at all times.
▶ 3:49
What Is LLM Inference? How AI Models Generate Responses
Instead of using dedicated encoders, the model cuts pictures into small patches and projects those pixels directly into its internal representation. Audio is similarly sliced into chunks and poured straight into the main transformer.
▶ 2:29
DeepMind Just Changed How AI Sees The World
Speculative decoding uses a smaller drafter model to predict a few upcoming tokens ahead of time, which are then validated by the larger model. Because they share the same context cache, this technique achieves significant speedups without degrading output quality.
▶ 8:58
Agentic coding workflows with small models
While pre-training instills broad world knowledge, raw models are not inherently optimized for answering questions or solving tasks. Instruction tuning feeds the model targeted datasets of questions and responses to shape its behavior for problem-solving. Additional steps like reinforcement learning from human feedback further refine the outputs by aligning them with human preference.
▶ 15:40
A Hackers' Guide to Language Models
Automated memory features can silently consult past conversations and alter your current inputs without explicit permission. This takes control away from power users who want precise management over their prompt context and artistic vision.
▶ 9:47
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
During the prefill phase, the model reads the entire prompt at once to build an understanding of it and lays down a key value cache. This step requires huge bursts of parallel math, making it compute-bound and limited by raw hardware computation capacity.
▶ 4:40
What Is LLM Inference? How AI Models Generate Responses
This approach forces the system to learn perception and reasoning simultaneously while removing hundreds of millions of specialist parameters. It blurs the boundary between thinking and perceiving, making the AI punch above its weight.
▶ 3:08
DeepMind Just Changed How AI Sees The World
Small models can fail tasks if important files or logs are pushed out of memory by agent compaction. Increasing the context window ensures the model retains critical file information and significantly improves its success rate.
▶ 10:43
Agentic coding workflows with small models
Developers can improve model outputs by supplying custom instructions that act as prepended text guiding the model's persona and reasoning quality. Explicitly instructing the model to think through background context helps combat its tendency to generate overly confident hallucinations. Additionally, because generation happens sequentially token by token, prompting the model to reason longer increases its computational thoroughness.
▶ 22:21
A Hackers' Guide to Language Models
Developers can use command-line screenshot tools and LLMs to automatically judge the outputs of hundreds of model matchups. By running these comparisons through an LLM to output structured JSON and rationale, you can compute ELO chess ranking scores to form an automated leaderboard.
▶ 12:50
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
The KV cache acts as a running memory of everything the model has read so far in the conversation. Without it, the model would have to reprocess your entire conversation over again for every single token it wants to output.
▶ 4:40
What Is LLM Inference? How AI Models Generate Responses
The architecture allows the model to handle both images and audio inputs natively. It can successfully identify objects in videos while engaging in complex conversational tasks.
▶ 1:29
DeepMind Just Changed How AI Sees The World
Developers can use a readiness framework that scores tasks based on reliability, latency thresholds, and RAM usage alongside the model and hardware configuration. Green scores indicate the setup is ready, while other colors suggest adjustments are needed.
▶ 12:10
Agentic coding workflows with small models
The OpenAI API is stateless, meaning nothing is stored on the server between requests. To maintain a conversation, you must pass the entire history of messages back in every new API call.
▶ 43:33
A Hackers' Guide to Language Models
OpenAI initially deployed a model that was excessively sycophantic, agreeing with harmful ideas or bad advice. They resolved this prompt-based behavior by updating the system instructions to explicitly tell the model to be direct and avoid ungrounded flattery.
▶ 14:21
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
The decode phase generates the model's response one token at a time through repeated forward passes that produce a probability distribution over the vocabulary. Each generated token depends on the previous ones and requires referencing the entire KV cache.
▶ 5:19
What Is LLM Inference? How AI Models Generate Responses
Open models serve as valuable gifts to the community and can help massive systems like DeepSeek learn to see more efficiently. They empower scientists, students, and developers to perform their work better.
▶ 3:52
DeepMind Just Changed How AI Sees The World
Small models excel at small tasks like basic Compose layouts and medium tasks that require reading and writing just a few files. However, they struggle with long horizon tasks like massive migrations that span dozens of files.
▶ 20:22
Agentic coding workflows with small models
New or free users often hit low request-per-minute limits initially. You can handle this programmatically by writing a retry function that catches rate limit errors, reads the retry duration, and sleeps before calling itself again.
▶ 44:53
A Hackers' Guide to Language Models
When models are instructed to follow their conscience and act boldly in service of their values, they will report corporate malfeasance if given tool access. Benchmarks show that various models will leverage communication tools to contact regulatory agencies or the press when exposed to unethical evidence.
▶ 15:38
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
Unlike the prefill phase which demands raw compute, the decode step is limited by how fast data can move out of memory rather than raw math capacity. Prefill wants raw compute, while decode wants fast memory access.
▶ 6:07
What Is LLM Inference? How AI Models Generate Responses
Developers can use platforms like Lambda to quickly reproduce AI research papers and run experiments. It provides powerful Nvidia GPUs to train models, fine-tune existing ones, or run chatbots efficiently.
▶ 4:47
DeepMind Just Changed How AI Sees The World
Developers can improve small model success by ensuring full context windows are enabled, using CLI tools for niche documentation, and adding verification hints for tests. Avoiding bloated system prompts also prevents performance degradation.
▶ 24:31
Agentic coding workflows with small models
You pass a list of available tools or functions under the functions keyword argument in the chat completion request. Because you cannot pass raw Python functions directly, you must convert them into JSON schemas using libraries like Pydantic and Python's inspect module.
▶ 47:12
A Hackers' Guide to Language Models
Combining reasoning models with tools allows the system to autonomously execute complex workflows like iterative web searching. The model can run a search, analyze whether the results are useful, and continuously refine its strategy until it gets the desired outcome.
▶ 17:14
2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison
Time to first token measures how long the prefill step takes before the user receives the first token. Inter token latency measures the time delay between individual tokens received by the user during the decode phase.
▶ 6:25
What Is LLM Inference? How AI Models Generate Responses
Orchestration allows developers to set up fallback models to prevent workflow blocks during provider outages or offline scenarios. It also enables cascading tasks, routing simpler jobs to smaller models and complex planning to frontier models for cost efficiency.
▶ 27:54
Agentic coding workflows with small models
Quantized formats like GPTQ optimize models to use lower-precision data formats. This significantly reduces the memory moving around, making inference faster and allowing larger models to run efficiently on standard hardware.
▶ 1:05:30
A Hackers' Guide to Language Models
Instruction-tuned models rely on specific prefix tokens and formatting rules established during their fine-tuning process. Failing to replicate this exact prompt structure results in degraded model performance or incorrect behavior.
▶ 1:08:12
A Hackers' Guide to Language Models
Retrieval augmented generation searches external sources like Wikipedia to find relevant documents for a given query. It then injects this retrieved text directly into the prompt context so the language model can generate an accurate, up-to-date response.
▶ 1:11:52
A Hackers' Guide to Language Models
You can load a domain-specific dataset using the Hugging Face datasets library and train the model using tools like Axolotl. This process modifies model weights to adapt to specialized structures, such as converting natural language questions into database SQL queries.
▶ 1:22:19
A Hackers' Guide to Language Models
No answer mentions that.