Static batching makes every request wait for the slowest one. Here is the scheduling trick that fixed it, where it came from, and the tradeoff nobody mentions.
Continuous batching is the scheduling technique that lets an LLM inference server swap requests in and out of a running batch at every generation step, instead of waiting for the slowest request to finish before starting the next batch. It is the single reason modern inference engines can keep a GPU near full utilization under real, uneven traffic.
Key takeaways
Every inference engine that matters in 2026, vLLM, SGLang, TensorRT-LLM, all lean on some form of continuous batching to keep GPUs busy. vLLM batching specifically, meaning continuous batching vLLM implements by default, is the version most engineers encounter first. It is one of the two ideas, alongside PagedAttention, that took LLM serving from research-lab throughput to something that could plausibly serve production traffic. Most explanations of it stop at "new requests fill empty slots," which is true but skips the actual scheduling problem underneath. This post covers how the scheduling works, where the idea came from, and where it still runs into real friction.
This post belongs to the packet.ai LLM serving frameworks cluster. For the broader picture of GPU sizing across model families, see the VRAM requirements guide. For the memory-management half of the story, see the vLLM Docker deployment guide, which covers PagedAttention in depth.
Before continuous batching existed, inference servers batched requests the way most people would naturally design it: collect a group of requests, run them together, wait for every single one to finish, then start the next batch. This is static batching, and it has one structural flaw that gets worse as traffic gets more realistic.
LLMs generate output one token at a time, and nobody knows in advance how many tokens a given request will need. A summarization request might finish in 40 tokens. A long-form generation might run for 2,000. Under static batching, every request in a batch is locked to the same clock: the batch cannot return any results and cannot accept any new requests until the single longest-running sequence finishes. Every other GPU slot in that batch sits idle, fully allocated, doing no useful work, for however long the slowest request takes.
Under high sequence-length variance, this is not a small inefficiency. Some benchmarks show static batching throughput collapsing to as low as 81 tokens per second on workloads with mixed short and long generations, a small fraction of what the same hardware can sustain once idle time is removed. The GPU is not the bottleneck here. The scheduling policy is.
The diagram makes the mechanism concrete. In static batching, once request A or B finishes early, that GPU slot sits idle, grayed out, for the rest of the batch, because the batch as a whole can't restart until C also finishes. In continuous batching, the same slot gets a new request (D, then E, then F) the moment the previous one completes. No slot goes idle. This is the entire throughput gain in one picture.
Continuous batching, also called iteration-level scheduling or in-flight batching depending on which engine you're reading about, fixes this by making a scheduling decision at every single token generation step instead of once per batch. The mechanism traces to Orca, a serving system presented at OSDI 2022, which proposed exactly this: run the model one iteration at a time, and after every iteration, check whether any sequence has finished. If one has, immediately admit a new request into that now-empty slot. The batch's composition changes continuously across iterations, rather than staying fixed for the batch's entire lifetime.
Orca's own evaluation on a GPT-3 175B model reported a 36.9x throughput improvement over NVIDIA FasterTransformer at the same latency, by this mechanism alone. That is a striking number for what is, underneath the terminology, a fairly simple idea: stop treating the batch as a fixed unit and start treating the GPU's available slots as a resource to refill continuously.
"Dynamic batching" sometimes gets used as a synonym for continuous batching, but they are not quite the same thing, and this matters if you're searching for dynamic batching LLM guidance and landing on continuous batching content instead. Dynamic batching, in the stricter sense, still batches at the request level: it waits for a time window or a batch-size threshold, then launches whatever has arrived, without waiting indefinitely for a full batch. Continuous batching goes a step further and makes the decision at every token, not just at batch-launch time. If you see the two terms used interchangeably, check which mechanism the source is actually describing before trusting the distinction.
Iteration-level scheduling gets most of the attention, but Orca's paper actually proposed two techniques together, and the second one solves a problem the first one creates. Once the batch's composition can change every iteration, sequences in that batch are no longer guaranteed to be at the same point in generation, some may be on their 3rd token, others on their 200th. Standard batched matrix operations assume every sequence in the batch is at the same shape and stage. That assumption breaks the moment iteration-level scheduling is introduced.
Orca's answer, selective batching, is to apply batching only to the operations that tolerate sequences being at different stages, mainly the linear and normalization layers, while running attention separately per sequence, since attention is the one operation whose computation actually depends on each sequence's own accumulated context. This is a real engineering compromise, not a free optimization: it trades some batching efficiency on the attention operation specifically in exchange for making iteration-level scheduling possible at all. Later systems, including vLLM's PagedAttention, took this further by fusing per-sequence attention into a single CUDA kernel that reads from non-contiguous memory, which is a different, complementary solution to the same underlying constraint.
⚡ Note
This is the part most explanations skip, and it's the part worth understanding if you're actually tuning a production server rather than just reading about the concept.
LLM inference has two distinct phases with very different computational profiles. Prefill processes an entire incoming prompt in one shot and is compute-heavy: it's a large, parallel matrix operation. Decode generates one token at a time for an already-running sequence and is memory-bandwidth-bound: it's a small operation, repeated many times. Continuous batching has to decide what happens when a new request's prefill arrives while other requests are mid-decode.
Run the new prefill immediately, and you delay every currently-decoding sequence in the batch by the time the prefill takes, which can be substantial on a long prompt. Delay the prefill until decode finishes, and the new request's time-to-first-token gets worse. There is no version of continuous batching that makes this tradeoff disappear; different engines just make different choices about where to sit on it. Orca's original design supported hybrid batches, mixing prefill and decode requests in the same iteration, from the start. vLLM's own scheduler initially took the opposite approach by default, prioritizing prefill-only or decode-only batches to optimize time-to-first-token, at some cost to steady-state throughput.
Chunked prefill is the technique most production engines now use to soften this tradeoff instead of picking a side outright. Rather than running an entire prompt's prefill in one uninterrupted pass, chunked prefill splits it into smaller pieces, typically a few hundred to a couple thousand tokens each, and interleaves those pieces with the decode steps of requests already running. Instead of one long prefill blocking every decoding sequence for its full duration, decode work gets a turn between each chunk.
The technique comes from Sarathi-Serve, and it has since been adopted directly into vLLM's own scheduler rather than staying an external add-on: vLLM's V1 engine enables chunked prefill by default wherever possible. The tuning knob that controls it, max_num_batched_tokens, sets the token budget per scheduling step. A smaller budget means decode requests get interrupted less often, which improves inter-token latency; a larger budget lets more prefill through per step, which improves time-to-first-token for the new request. Neither setting eliminates the tradeoff, it just moves where you feel it.
This is worth being direct about: chunked prefill is not a free upgrade over plain continuous batching, it's a more granular way to manage the same underlying prefill-decode tension described above, and getting real benefit from it means tuning the chunk size against your own traffic's prompt-length distribution rather than trusting a default value.
If you're comparing inference engines and the terminology feels inconsistent, that's because it genuinely is, not because the underlying mechanism differs meaningfully between them.
| Engine | Name used | Notes |
|---|---|---|
| vLLM | Continuous batching | Paired with PagedAttention for memory management |
| Hugging Face TGI | Continuous batching | Same term as vLLM, same underlying mechanism |
| TensorRT-LLM | In-flight batching | NVIDIA's own name for the identical iteration-level idea |
| LMDeploy | Persistent batching | Same mechanism, third naming convention |
None of these are competing techniques. They're the same scheduling idea, implemented independently, named differently by each team. If a benchmark or vendor comparison treats "in-flight batching support" as a differentiator against an engine that has "continuous batching," that's a marketing framing, not a technical one, worth checking closely before trusting the comparison.
These two ideas get bundled together often enough that it's worth separating them cleanly. Continuous batching is a scheduling technique: it decides which requests run in which iteration. PagedAttention is a memory management technique: it decides how the KV cache for those requests is stored in VRAM. They solve different problems and were developed independently, though vLLM popularized running them together, which is likely why they get conflated. You can have continuous batching without PagedAttention, and in fact Orca did, since PagedAttention came a year later. For the memory-management side of this story in full, the vLLM Docker deployment guide covers it in depth; this post stays on the scheduling half.
Getting continuous batching right in practice usually means tuning concurrency settings, flags like vLLM's --max-num-seqs, against your actual traffic shape rather than a default value, and that tuning process involves running the same workload several times at different settings to find where throughput actually plateaus.
packet.ai's Dynamic tier is built around a version of the same underlying idea one layer up: instead of one engine sharing GPU slots across requests, the packet.ai scheduler shares physical GPU capacity across multiple tenants' workloads, keeping latency within roughly 2 to 5 percent of a dedicated card while billing by the hour rather than locking you into a fixed allocation. An RTX 6000 Pro on Dynamic runs at $0.66/hr, live in under 5 minutes from API call to SSH-ready, which makes running five or six concurrency experiments in an afternoon a reasonable thing to do rather than an expensive one. Once you've found your actual concurrency ceiling, moving the same workload to a Dedicated instance removes scheduler interference entirely for production traffic.
Last reviewed: August 3, 2026. For the memory-management complement to this post, see the vLLM Docker deployment guide. Browse packet.ai GPU cluster options for multi-node inference.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
