What Is LLM Inference? How AI Models Generate Responses

1,128 words · about 5 min to read

Inference costs and engine basics

0:00 Training a Frontier model can cost hundreds of millions of dollars and take months. But once that model is produced, it's just a huge set of numbers sitting idly in memory. A model doesn't know how to do anything on its own. So what's going on behind the scenes to bring that model to life? That process is called inference. And LLM inference is behind most modern chatbots, code assistants, and agentic applications today.

0:26 Inference happens every single time someone sends a message to the model as opposed to training which only happens once. A trained model is a static pile of weights or specialized numbers that are tuned during the training process to represent information. Inference is the process of taking that train model and running it against new input to produce some sort of output.

0:53 Inference surprisingly usually costs more over time than training as it reoccurs with every single request. As time goes on, it becomes more and more crucial to serve models as efficiently as possible to reduce this cost. Today, we'll look at what really happens between prompt and response and how you can make that serving process as fast and cheap as possible. The software that manages inference is called an inference engine.

1:20 And its job is to take a pile of weights and turn it into something that can serve requests reliably, quickly, and at scale. That means scheduling incoming requests, managing GPU resources, and deciding how to process work together. If you've ever heard of vLLM, Ollama, or Llama CPP, these are all examples of inference engines. Let's take a look at what happens when you send a message to an LLM.

Tokenization and hardware routing

1:45 Most inference engines expose a standard OpenAI compatible endpoint, which means you can use the same request format to send your query regardless of which model is running under the hood. This is super helpful when you have an application built to call one model and want to easily swap in another without rewriting any code. But before that request even enters the data center, we need to translate it into a format that the model can understand.

2:15 This process is called tokenization. Breaking your text into tokens which might be entire words, just a piece of a word or some punctuation and mapping each one to an arbitrary token ID. The model never really sees these words, only the stream of IDs. Though it takes next to no processing power. This step defines how much your prompt will cost as its build per token.

2:47 Models with larger vocabularies can represent text more efficiently using fewer tokens per prompt, but a larger vocabulary adds computational overhead in the training process. Your request needs to be processed on physical hardware. In most cases, that hardware lives in a data center. Once your request arrives at the data center, it will be authenticated via an API key or token, something that proves you're allowed to use those GPU resources.

Continuous batching maximizes GPU utilization

3:16 Next, you'll be routed to specific hardware. The inference router looks at your prompt, how long it is, what content it contains, and routes it to available hardware that's most optimized for your request at that time. However, your request isn't the only one coming in, and sending each request through the model individually would be like a bakery using an industrial oven on a single pastry at a time.

3:41 It would waste enormous amounts of GPU resources and would tank the systems throughput or how many requests a system can carry at once.

1 Highlight · Continuous batching for high GPU throughput

3:49 Batching groups requests together so the GPU processes them as one larger more efficient unit of work Not all requests take the same amount of time to process. So modern engines use continuous batching. As soon as one request finishes, another one will be slotted in to take its place. That way GPUs sit near full utilization at all times. A scheduler will handle this batching process and get your requests routed to the hardware where the real computation is performed and your response is generated. Let's double click into this last step.

Prefill and decode computational differences

2 Highlight · Prefill versus decode phase bottlenecks

4:25 LLM inference contains two distinct phases, prefill and decode. Once your request reaches the model, it enters the prefill phase. Prefill is where the model reads the entire prompt at once and builds an understanding of it, laying down what's called the key value or KV cache.

4:51 A running memory of everything it's read so far. Without it, the model would have to reprocess your entire conversation over again for each token it wants to output. It's the reason your conversations don't get exponentially slower as they get longer. This step is computebound, meaning it takes huge bursts of parallel math. So its speed is limited by how much raw computation the hardware can support at once.

5:19 Next, that KV cache is passed over to the decode phase. Decode is where the model's response is generated one token at a time. On each step of decode, the model runs a forward pass and produces a probability distribution over every token in its vocabulary. Essentially, this is a ranked list of what token could potentially come next.

5:47 Once a token gets selected from the distribution, it gets appended to the sequence and the entire process starts over again with each token generated. Each token depends on the ones before it, and for each step, the model also has to refer back to this entire KV cache to remember the information it learned in prefill. This step is memory bandwidth bound, meaning its speed isn't limited by how much math it can do, but by how fast data can move out of memory.

6:17 Prefill wants raw compute, while decode wants fast memory access. This is also where two common benchmarking numbers come from. Time to first token or TTFT

Benchmarking latency metrics

3 Highlight · Time to first token and inter-token latency

6:31 is how long it takes for the first token to be received by the user. This effectively shows you how long your prefill step is taking. And inter token latency or ITL, this is the time it takes between tokens to be received by the user. This essentially shows how long each step of your decode phase is taking.

6:57 Together, these make up most of what people mean by an LLM's latency. Inference is how a static trained model actually generates a response. Every request is scheduled into batches to keep hardware busy across uneven requests. Requests are routed to hardware, moved through a computebound prefill phase, and a memory bandwidthbound decode phase. Put together, this is the machinery that sits between a trained model and a product people can actually use in real time.

7:27 Hit me with any follow-up questions in the comments or let me know which part of your inference stack you're already optimizing.