vLLM Prefix Caching Explained: Reduce Latency for Repeated Prompts

vLLM's automatic prefix caching reuses the KV cache from a previous request's prompt when a new request shares the same starting tokens, skipping recomputation for anything already cached. It's enabled by default on vLLM's V1 engine as of recent releases, so most teams running vLLM already have it on without realizing it. This is a form of KV cache reuse and, more broadly, LLM prompt caching at the infrastructure level, distinct from the provider-side prompt caching offered by hosted APIs. This post covers what the mechanism actually does, how to confirm it's active and tune it, and how to read your own hit rate.

Prefix caching is one of the more consequential vLLM defaults most teams never look at directly. It works quietly in the background on any workload with repeated prompt structure, which describes most production chat, RAG, and agent traffic. This post belongs to the packet.ai LLM serving frameworks cluster; for the broader deployment picture, see the vLLM deployment tutorial and the vLLM Docker guide, which covers PagedAttention's memory management in depth. This post is about the mechanism itself and how to work with it directly: what the hash-based cache actually does, how to enable, disable, or tune it, and how to read your own hit rate. It isn't about whether caching is worth it in dollar terms for your specific traffic; that's a cost-modeling question with its own dedicated coverage elsewhere on this site. If you're trying to size the financial upside of caching before you touch any config, that's the post to read first.

What Automatic Prefix Caching Actually Does

Every request to an LLM starts with prefill: processing the full input prompt to build the initial KV cache before generation begins. If two requests share the same opening tokens, a system prompt, a set of few-shot examples, an accumulated chat history, the first one already did the compute work the second one is about to repeat. System prompt caching is the most common real-world case: nearly every production deployment sends the same system prompt on every call, and it's exactly the kind of static prefix this mechanism is built to catch. Prefix caching is vLLM keeping that work around so the second request doesn't have to redo it.

vLLM manages its KV cache in fixed-size blocks (16 tokens each by default) through the vLLM block manager, and prefix caching identifies each block with a hash that chains it to the block before it: the hash covers the block's own token IDs plus the hash of its parent block. That chaining means a cache hit isn't "roughly similar text," it's an exact token-by-token match of the entire prefix up to and including that block. When a new request comes in, vLLM checks its block hashes against what's already cached; anywhere the chain matches, it reuses the existing physical block instead of recomputing it. The first point where the chain breaks is where fresh computation starts.

This is the same block-based, hash-table approach used for PagedAttention's memory management generally, extended to do double duty as a cache key. There's no separate tree structure to maintain; every block is independent, addressable by its own hash, and can be allocated or freed on its own.

Enabling, Disabling, and Tuning It

Here's the part that surprises people coming to this fresh: on vLLM's V1 engine, prefix caching is already on. The flag most documentation still shows as --enable-prefix-caching exists, but if you're running a reasonably current vLLM version, you're more likely to need its opposite:

# Confirm it's on (this is the default, so this line does nothing new)
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-prefix-caching

# Explicitly turn it off
vllm serve meta-llama/Llama-3.1-8B-Instruct --no-enable-prefix-caching

There are legitimate reasons to disable it: multi-tenant setups where you don't want any KV state persisting across requests as a hard security boundary, or workloads where every prompt is genuinely unique and the hash lookup itself adds overhead with zero hit rate to show for it. For the more common case, shared system prompts, RAG contexts, multi-turn chat, leaving it on is the right default.

Two configuration details worth knowing about beyond the on/off switch:

Measuring Your Actual Hit Rate

Whether prefix caching is doing anything for your workload is an empirical question, not something to assume from the fact that it's enabled. vLLM's V1 engine exposes this through two Prometheus counters: vllm:prefix_cache_queries and vllm:prefix_cache_hits. An older single gauge, vllm:gpu_prefix_cache_hit_rate_perc, is deprecated and being phased out; the current approach is deliberately counter-based so you calculate the rate yourself over whatever time window you choose, rather than trusting a built-in average that can dilute a sudden change in traffic pattern. A simple PromQL rate query, hits divided by queries over your chosen interval, gives you the number.

A useful way to think about the resulting number: it's a direct readout of how much prefill computation your traffic is genuinely repeating. A workload where every request opens with the same system prompt and tool definitions should show a high, stable hit rate. A workload of one-off, structurally distinct prompts will show a low one regardless of how well-configured caching is, because there's nothing to reuse. If your hit rate is lower than the shared-prefix structure of your traffic would suggest, that's a signal to check whether your prompts are actually byte-for-byte identical where you think they are, since even a single-token difference early in the prompt breaks the hash chain from that point forward.

Calculate the ratio over a representative traffic window, not a single request or a short burst. Hit rate is a property of your traffic pattern over time, and a handful of early cache-cold requests will understate it if you're not aggregating over a long enough interval.

When This Actually Helps, and When It Doesn't

Prefix caching earns its keep specifically on workloads with real, repeated prompt structure:

One structural mistake worth calling out directly: putting user-specific or highly variable content at the front of a prompt, before the static system instructions or shared context, breaks the prefix match immediately, since the hash chain diverges from the very first block. If your prompt template puts a user ID, timestamp, or per-request variable ahead of your static system prompt, prefix caching has effectively nothing to work with no matter how much of the rest of the prompt repeats. Structuring prompts with static, shared content first and variable content last is a small change that determines whether caching applies at all.

How This Differs From SGLang's RadixAttention

vLLM's approach and SGLang's RadixAttention solve the same underlying problem, reusing computed KV cache across requests that share a prefix, with genuinely different data structures. vLLM's hash table treats every block independently: no shared parent-child relationship needs to be maintained beyond what's encoded in the chain hash itself. RadixAttention instead builds and maintains an actual radix tree (trie) across all cached sequences, which lets it find the longest common prefix against any prior request more directly, at the cost of maintaining that tree structure.

In practice, vLLM's own eviction policy is deliberately designed to approximate RadixAttention's behavior for standard attention models even without a tree, evicting the least-recently-used block, and preferring to evict blocks at the end of longer prefixes first, which keeps more broadly-shared, shorter prefixes alive longer. The two approaches converge on similar practical outcomes for many workloads; SGLang's tree structure tends to show its advantage more clearly at very high concurrency with heavily overlapping prefixes. A related but separate concept worth knowing exists is cache aware routing, directing a request to the specific replica that already holds its prefix warm in a multi-instance deployment; that's a load-balancer-level decision sitting above prefix caching itself, not a configuration option within it. For a full breakdown of when the RadixAttention gap actually shows up in benchmarks and which engine to pick for prefix-heavy traffic specifically, the packet.ai SGLang vs vLLM vs TensorRT-LLM guide covers that comparison directly; this post stays focused on vLLM's own implementation.

Running This on packet.ai

Prefix caching is fundamentally about reusing something expensive to recompute instead of paying for it twice, and packet.ai's Dynamic tier applies the same principle one layer up: instead of one engine reusing KV cache across requests, the packet.ai scheduler shares physical GPU capacity across tenants, keeping latency close to a dedicated card's while billing hourly rather than requiring a fixed allocation. That makes Dynamic a reasonable place to actually test prefix-caching configuration against your own traffic, different block sizes, cache-salt setups, prompt template restructuring, without committing to a Dedicated instance before you've confirmed what your real hit rate looks like. Once you've validated a configuration and settled on steady-state traffic, moving to Dedicated removes scheduler variance from the picture entirely.

Frequently asked questions

Last reviewed: August 5, 2026. For the cost and ROI side of prompt caching decisions, see packet.ai's dedicated cost guide. For engine-level comparisons including RadixAttention benchmarks, see the SGLang vs vLLM vs TensorRT-LLM guide.