KV cache is why your second token is fast. Here is how it actually works, why it eats GPU memory at scale, and new research on sharing it across models.
KV cache is the reason an LLM's second token generates faster than a fresh forward pass through the entire model would take. Every attention layer stores the key and value tensors it computes for each token, and reuses them instead of recalculating the full sequence on every step. It's the single most important mechanism behind LLM inference optimization, and getting the KV cache LLM basics right makes everything else in this post make sense. This post covers what KV cache actually is, why it becomes the dominant memory cost at serving scale, and a genuinely new research direction: reusing one model's KV cache in a different model from the same family, which could change how multi-model serving stacks like Token Factory handle cost and latency.
Key takeaways
LLMs generate text one token at a time, and each new token depends on the model's understanding of everything generated before it. The naive way to do this would be to feed the entire sequence back through the model at every step, recomputing attention over every prior token from scratch each time. That's wasteful: the model already did that work one step ago, and the only thing new is a single token.
KV cache eliminates the waste. Every attention layer computes a key (K) and value (V) tensor for each token, alongside the query (Q) used to compute that step's output. The key and value tensors depend only on the token itself and everything before it, not on what comes after, so once they're computed, they stay valid for the rest of the generation. KV cache stores those key and value tensors as they're produced, so the next token's forward pass only computes a new query, key, and value for the one new token, and attends to everything already cached rather than recomputing it.
⚡ Prefill vs. decode
Inference splits into two phases. Prefill processes the entire input prompt in one pass and populates the KV cache for every prompt token at once. Decode then generates one token at a time, each step adding just one new key-value pair to the cache and attending over everything stored so far. Prefill is compute-heavy and happens once per request; decode is memory-bandwidth-heavy and repeats for every output token.
A concrete way to see it: if a model has already processed "Time flies" and predicts "fast" next, the KV cache already holds the key and value tensors for "Time" and "flies" at every layer. Generating the token after "fast" doesn't require reprocessing "Time flies fast" from scratch; it computes one new key-value pair for "fast," appends it to the cache, and attends over the full cache to produce the next prediction. Without caching, that step would mean running the entire growing sequence through the entire model again, every single time.
KV cache size follows a straightforward relationship: roughly 2 × attention heads × head dimension × layers × bytes per element, per token. That "per token" is the part that matters operationally, because it means cache size grows linearly with context length, and multiplies by however many requests are being served concurrently. A single long conversation, a large retrieved document, or a big batch of simultaneous users can turn kv cache memory from a minor detail into the dominant consumer of GPU memory, ahead of the model's own weights in some long-context or high-concurrency deployments.
This is the reason a whole category of kv cache optimization techniques exists specifically to manage cache rather than model weights. PagedAttention manages cache memory the way an operating system manages virtual memory, in fixed-size blocks rather than one large contiguous allocation, which is the mechanism underneath vLLM's memory efficiency. Prefix caching, covered in more depth in the packet.ai vLLM prefix caching guide, reuses KV cache across different requests that share a prefix, like a common system prompt, rather than recomputing it for every new conversation. And some newer model architectures, like the hybrid Mamba layers in Nemotron 3 Super or the sparse attention in MiniMax M3, are specifically designed to avoid the linear KV cache growth of standard attention altogether.
Worth distinguishing precisely: KV cache itself is the per-token mechanism inside a single generation. Prefix caching is a separate, later technique that reuses KV cache across different requests to the same model. Both matter, but they're not the same thing, and confusing the two is a common source of imprecision in casual explanations.
Until recently, the honest answer was effectively no. KV cache is a direct function of a specific model's learned key and value projection weights; a 14B model and a 32B model in the same family process the same input token completely differently internally, even if they were trained on similar data with a similar architecture. Reusing one model's cache in another wasn't just unsupported, it produced meaningless output, because the cached tensors encode a different model's internal representation entirely.
Two research directions changed that picture. DroidSpeak, published in late 2025, showed that KV cache sharing across different fine-tunes of the same base model is viable if a few layers are selectively recomputed rather than reused wholesale, achieving up to 4x throughput improvement with negligible quality loss, but strictly required identical architecture between sender and receiver. SwiftCache took a different angle, sharing KV cache across heterogeneous models over NVLink within a single server to reduce time-to-first-token, rather than transferring cache between genuinely different model weights.
⚡ A very recent finding, worth treating as early research
A 2026 paper found something more direct: across matched key-value pairs from different-sized models in the same family that share KV head count and per-head dimension, the relationship between the two models' KV tensors has substantial linear structure. On Qwen3 14B to 32B, one source layer explained 56% of the variance in the target model's keys and 32% in its values; using multiple source layers pushed that to 79% and 65%. The paper fits a closed-form ridge regression mapper, no gradient training required, to translate one model's cache into a usable approximation of the other's, explicitly to skip prefill on model switches. This is a one-week-old result at time of writing, not an established production technique. Treat it as a promising research direction, not something to deploy today.
The practical framing the paper itself uses is directly relevant to anyone running a multi-model serving stack: production deployments frequently cascade between model sizes for cost and quality tradeoffs, switch models mid-conversation, or route different requests to different models. Every one of those transitions currently means the receiving model repays the full prefill cost from zero, even though the previous model already did equivalent work processing the same context. A working cross-model cache transfer would let a system switch from a cheap, fast model to a larger, higher-quality one (or the reverse) without discarding the computation already done.
Token Factory serves a curated catalog of different open models, Llama 3.3, Qwen2.5, DeepSeek-V3, Mistral, and others, behind one OpenAI-compatible endpoint, which is exactly the kind of setup where model switching, cascading, and model routing llm decisions between different model sizes happen constantly in production. That's precisely the scenario this line of research is aimed at: reducing the tax paid every time a request moves from one model to another.
To be precise about where things stand: cross-model KV cache transfer isn't a shipped feature of Token Factory or any production serving system today. It's an active, fast-moving research area, with real early results but not yet a settled, deployable technique. What's worth understanding now is the shape of the problem, since it directly explains why switching models mid-request is more expensive than it looks on paper, and why a technique that closes that gap would matter for anyone running requests across a model catalog rather than a single fixed model.
Last reviewed: August 14, 2026. For prefix caching specifically, see the vLLM prefix caching guide. For architectures that reduce KV cache growth directly, see the Nemotron 3 Super and MiniMax M3 guides.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
