Prefill is compute-bound, decode is memory-bound. Here is how disaggregated serving splits them across GPU pools, and when it actually pays off.
Every LLM inference request has two phases with genuinely different resource needs: prefill, which processes the prompt, and decode, which generates the response token by token. This architecture is commonly called prefill-decode disaggregation, or PD disaggregation. Most serving setups run both phases on the same GPU. Disaggregated inference, the prefill vs decode split done at the infrastructure level, runs them on separate GPU pools instead. This guide covers why that split exists, how it actually works, and when it's worth the added complexity.
Key takeaways
Prefill processes the entire input prompt in one pass, computing attention across all prompt tokens simultaneously and storing the results in the KV cache. Because it processes a large batch of tokens at once, prefill is compute-bound: the GPU spends most of its time doing dense matrix math, and reaching a high fraction of peak throughput is achievable.
Decode generates the response one token at a time, and each step requires reading the model's entire set of parameters, plus the growing KV cache, from memory. The actual arithmetic per step is comparatively small. Decode is memory-bandwidth-bound rather than compute-bound, which is why it typically uses a much smaller fraction of a GPU's peak compute capability than prefill does.
The problem isn't that prefill and decode cannot share a GPU. It's that they compete for different resources while serving different latency objectives. A long incoming prompt's prefill work can stall decode for requests already streaming a response, since the GPU can't fully serve both a compute-heavy burst and a steady memory-bound stream at the same time. This contention, the practical cost of forcing one chip to run both halves of the prefill vs decode split, is the core problem disaggregated inference exists to solve; the packet.ai guide to reducing LLM inference latency covers the prefill/decode distinction at the single-request level in more depth.
In a disaggregated serving setup, prefill and decode run on separate, independently sized pools of GPUs rather than sharing the same hardware. A request's prompt is processed on a prefill worker, which produces the KV cache; that cache is then transferred to a decode worker, which handles the actual token-by-token generation and streams the response back. A router sits in front of both pools, directing requests and, ideally, routing follow-up turns to the decode worker that already holds relevant cached history.
The KV cache transfer between pools is the part that makes or breaks this architecture in practice. Within a single rack, this typically happens over NVLink; between nodes, it relies on RDMA-class networking (InfiniBand or RoCE) for the transfer to stay fast enough not to erase the benefit of splitting the phases in the first place. Several connector implementations exist to handle this kv cache transfer, including NIXL and Mooncake-style approaches, and current serving engines expose disaggregated prefilling as a configuration option rather than something you build from scratch.
⚡ Disaggregation is a latency and allocation strategy, not a throughput guarantee
It's worth being precise about what disaggregated inference actually buys you. vLLM's own documentation states plainly that disaggregated prefilling does not improve throughput by itself; the actual gains it targets are controlling tail inter-token latency and letting time-to-first-token be tuned independently from decode latency. The real benefits are latency isolation between the two phases, the ability to scale prefill and decode pools independently based on which one your traffic actually stresses, and better GPU utilization once you're operating at a fleet size where that independent scaling matters. Treating disaggregation as a guaranteed speedup for any deployment size overstates what the architecture is actually for.
Disaggregation adds real operational overhead: two pools to manage instead of one, a router to configure, and a KV cache transfer path that has to stay fast or the whole architecture backfires. That overhead is worth paying under specific conditions, not universally, and the decision is genuinely multidimensional rather than a single line you cross.
Disaggregation becomes worth considering when several of these start pulling in different directions: on the prefill side, long inputs, retrieval-heavy requests, large prompt bursts, or strict time-to-first-token targets; on the decode side, high concurrency, long generations, strict inter-token latency requirements, or a large active KV cache; and on the infrastructure side, enough GPUs to justify separate pools, fast KV-transfer networking already in place, and a workload stable enough to justify the added operational complexity. NVIDIA's own guidance on this is explicit that the optimal operating point depends on the model, GPU, backend, and request-length distribution, not a fixed threshold. For most teams running a single, moderately loaded model replica, a well-tuned co-located server with continuous batching already covers most of what disaggregated inference would add, without the operational cost of the split.
Agentic and RAG-heavy workloads are a genuinely good fit for this llm serving architecture specifically, since they tend to produce large, variable prompts (long tool definitions, retrieved documents, growing agent context) alongside output that streams over many decode steps, exactly the mismatch prefill decode disaggregation is built to resolve.
As of 2026, prefill decode disaggregation has moved well beyond the research-paper stage, with support appearing across major serving stacks, although implementation and maturity vary meaningfully by engine. vLLM documents disaggregated prefilling as an experimental feature, with multiple KV-transfer connectors including NIXL and Mooncake. SGLang has dedicated PD disaggregation with its own router and transfer backends, including Mooncake and NIXL, and its RadixAttention design gives it a particular edge on prefix-heavy workloads. TensorRT-LLM has dedicated disaggregated serving support and deployment templates. NVIDIA's Dynamo orchestration layer provides coordination around these backends and explicitly supports disaggregated deployments on vLLM, SGLang, and TensorRT-LLM.
This is a meaningfully different level of infrastructure commitment than a standard single-pool deployment, and it's worth evaluating against the conditions above before adopting it rather than reaching for it as a default.
Serving architecture decisions like this one are exactly the kind of infrastructure question a managed inference layer is meant to absorb. packet.ai's Token Factory is being built to apply serving optimizations like these at the infrastructure layer, so the underlying architecture choice isn't something you need to evaluate and operate yourself. It's currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: September 1, 2026. For the underlying prefill/decode distinction at the single-request level, see the packet.ai guide to reducing LLM inference latency. For the serving engine most commonly used for this architecture, see the What Is vLLM guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
