Every time you message a chatbot, that is inference, not training. Here is what LLM inference actually means, how it works, and why it costs what it does.
LLM inference is the process of using a trained language model to generate a response. Every time you send a message to ChatGPT, Claude, or another AI application, inference is happening in the background. It's the model actually being used, as opposed to training, the separate, earlier process of teaching the model in the first place. This guide covers what LLM inference actually is, how ai model inference works step by step under the hood, and why it's become one of the biggest cost and infrastructure questions in AI today.
Key takeaways
"Inference" is a general term in statistics and logic that means drawing a conclusion from evidence. In AI, the same basic idea applies to any trained model, not just language models: inference is the model using what it learned to produce an output on new input. This post focuses specifically on LLM inference, the version of that idea that applies to large language models like GPT, Claude, or Llama.
LLM inference is the process of running a trained large language model to generate output from a given input. The llm inference meaning comes down to that one distinction: it's the "using" phase, not the "building" phase. When you type a question into a chatbot, the text you sent is the input, and the process the model goes through to produce its reply is inference. Nothing about the model itself changes during this process. It's using knowledge it already has, not acquiring new knowledge.
This is worth being precise about, because it's one of the most common points of confusion for people new to how these systems work: an LLM does not learn from your conversation. If you correct it, explain something new, or have a long back-and-forth, none of that gets written into the model's underlying parameters. The model reads your entire conversation history again each time (this is what a "context window" is), and generates its response fresh from that, but the model itself is frozen. Whatever it learned during training is the entirety of what it knows going into every single inference request.
Understanding llm inference basics starts with separating it clearly from training, since the two are easy to conflate but work completely differently in almost every practical way.
A common analogy: training is like a student spending years in school studying and building knowledge. Inference is like that same student taking an exam afterward, drawing only on what they already studied. They're not learning new material during the exam, no matter how the questions are phrased. If you want the deeper mechanics of what training itself actually involves, treat this post as the inference half of that larger picture.
Inference happens one token at a time. A token is roughly a word or a piece of a word (the word "inference" might be one token, while a longer or less common word might split into two or three). The model doesn't write out its entire response in one step; it predicts the single most likely next token, adds that to the sequence, then predicts the next one based on everything so far, and repeats until the response is complete.
This breaks down into two distinct stages:
To avoid redoing the same work for every new token, the model keeps a running memory of what it's already processed, called a KV cache. This is what makes generating the second token of a response much faster than the first; the model reuses its earlier work instead of reprocessing the entire conversation from scratch every time. For a full explanation of how that mechanism works and why it matters at scale, see the packet.ai What Is KV Cache guide.
Here's llm inference explained with an actual sequence, rather than just describing it abstractly. Say you ask a model to complete the sentence "The capital of France is." During prefill, the model processes all five words of that prompt in one pass and builds an internal representation of the full sentence. Then decode begins: the model looks at everything so far and predicts the single most likely next token, which might be "Paris." That token gets appended to the sequence. The model then looks at the now-longer sequence ("The capital of France is Paris") and predicts the next token, perhaps a period to end the sentence. Once it predicts a special "end of response" signal, generation stops and the complete answer is returned.
For a one-word answer like this, the difference between prefill and decode is barely noticeable. For a long response, an essay, a piece of code, a detailed explanation, decode's one-token-at-a-time nature is exactly why longer outputs take longer to generate, token by token, even though the model "knew" the general shape of the answer as soon as it read the prompt.
Every inference request needs a GPU to actually run the model's calculations, and that GPU time costs money whether you're paying a provider per token through an API or renting the hardware yourself. Training is a one-time cost paid by whoever built the model. Inference is a recurring cost paid every time the model is used, by whoever is running it.
This is why, for most AI products actually in use by real customers, inference ends up being the larger ongoing expense over the model's lifetime, even though training gets more attention as a single eye-catching number. A model trained once for tens of millions of dollars can rack up inference costs that exceed that figure within its first year if it's serving meaningful traffic. For the full breakdown of what that actually costs across different providers and hardware choices, see the packet.ai LLM inference cost guide.
Two factors drive inference cost more than anything else: how many tokens a request involves, and how big the model is. A longer prompt and a longer response both mean more tokens to process, and more tokens means more GPU work. A larger model, one with more parameters, requires more memory and more computation for every single token it processes, whether that token is part of a one-word answer or a ten-page report. This is why providers typically charge more per token for their largest, most capable models, and why choosing a smaller model for a task that doesn't need frontier-level capability is one of the most direct ways to reduce inference cost.
Not all inference happens the same way. Real-time inference is what powers a chatbot: a person is waiting for a response right now, so speed matters more than efficiency, and the system optimizes for getting an answer back as quickly as possible. Batch inference is different: a large number of requests are processed together, often with no one waiting live for any individual result, which lets the system prioritize maximizing how many requests it handles per dollar rather than minimizing the wait time for any single one.
A customer support chatbot needs real-time inference, since a slow response directly hurts the experience. A pipeline that summarizes ten thousand documents overnight is better served by batch inference, since nobody is watching a spinner waiting for document number 4,732. Most production AI systems that scale end up using a mix of both, real-time for anything user-facing and batch for background processing, since the two workloads have genuinely different priorities.
Anyone running LLM inference in production is choosing between two basic approaches, and the right one depends almost entirely on volume and control requirements.
Managed API
Pay a provider per token to use a model they host and run for you. No infrastructure to manage, works immediately, scales automatically. The straightforward choice for lower or unpredictable volume, or for anyone who doesn't want to think about GPUs at all.
Self-hosted
Rent your own GPU and run an open-weight model yourself, using serving software like vLLM. More setup and ongoing operational work, but meaningfully cheaper per token at real scale, and gives full control over the model and your data.
Neither option is universally correct. A side project or an early-stage product with unpredictable traffic is usually better served by a managed API, since the setup cost of self-hosting doesn't pay for itself yet. A production system running millions of tokens a day is usually better served by self-hosting, since per-token API costs compound quickly at that volume. There's also a middle path worth knowing about: a managed inference API built specifically around open-weight models, like packet.ai's Token Factory, which offers per-token pricing without requiring you to manage your own GPU infrastructure.
Last reviewed: August 17, 2026. For how inference cost breaks down across providers and hardware, see the LLM inference cost guide. For the memory mechanism behind fast token generation, see the What Is KV Cache guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
