E2E latency is not one number to optimize. Here is what actually composes it, and how to diagnose which component is the real problem.
End-to-end (E2E) latency is the total time from sending a request to receiving the complete response, and it's the number most people mean when they say an LLM app's inference speed "feels slow." Understanding what actually happens during LLM inference is the foundation for this; E2E latency isn't one thing to optimize on top of that, it's the sum of several genuinely distinct latency metrics, each governed by different bottlenecks and fixed by different techniques. This guide covers the llm performance metrics that actually compose E2E latency, how to tell which one is dragging yours down, and why treating E2E as a single number leads to fixing the wrong problem.
Key takeaways
A full latency breakdown of a typical llm response time, sometimes called time to last token to distinguish it from time to first token, can be usefully broken down into four main pieces. Real serving stacks have additional measurement conventions and overhead beyond this, but this is the framework that matters for diagnosis:
NVIDIA's own current LLM benchmarking documentation defines TTFT as generally including request queuing time, prefill time, and network latency, and defines end-to-end request latency as including queueing, batching, and network latency, essentially the same breakdown used here. Time to first token, the delay before anything appears, is covered in depth in the packet.ai What Is TTFT guide, and the tradeoffs between TTFT and per-token generation speed are covered in the TTFT vs Tokens Per Second guide. This post's job is narrower: showing how those pieces, plus queueing and delivery, add up to the single end to end response time a user actually experiences.
Prefill processes the entire prompt in one compute-heavy pass, which is why it scales primarily with prompt length and available compute throughput. Decode generates output one step at a time, reading the model's full parameter set from memory on every step, which is why it scales with output length and available memory bandwidth rather than raw compute. These are architecturally distinct phases, covered in detail in the packet.ai Prefill vs Decode guide, and some production serving architectures now run them on genuinely separate GPU pools specifically because their resource profiles don't share a bottleneck.
Queueing is a different category of delay entirely: it's primarily driven by concurrency, scheduling policy, admission control, and available capacity, not by how fast the model itself generates tokens. A model with excellent prefill and decode speed can still have terrible E2E latency under high concurrent load if requests are queueing before they even start processing. This is why a latency sla quoted from a single-request benchmark, with no other traffic on the system, can be meaningless for a production deployment running at real concurrency.
The measured E2E number and how fast an app actually feels to use are related but genuinely separate things, and conflating them leads to optimizing the wrong target. Streaming a response, covered in the packet.ai guide to how LLM response streaming works, doesn't change the measured E2E time at all: the full response still takes exactly as long to generate. What streaming changes is when the user starts seeing output, which can make a mediocre E2E time feel fast if TTFT is quick and content streams in steadily afterward.
The reverse is also true: a genuinely fast E2E time can still feel slow if the client buffers the response before displaying it, or if a reasoning model's longer prefill delays the first visible token even though total generation finishes quickly. Optimizing the E2E number without accounting for how it's actually delivered to the user can produce a system that benchmarks well and still feels unresponsive.
A single average E2E latency figure collapses two very different failure modes into one number. A system with a slow, consistent TTFT and fast decode looks the same on average as a system with a fast TTFT and slow, inconsistent decode, but the fix for each is completely different: the first needs prefix caching or better prompt handling, the second needs serving engine tuning or different hardware. Reporting llm performance metrics only as one blended average, rather than breaking out TTFT, TPOT, and queueing separately, makes it impossible to tell which lever to actually pull.
The same problem applies to percentiles, not just component breakdown. A model latency figure reported only as an average can look excellent while its p99, the experience of the slowest 1% of requests, is genuinely bad. For production systems, percentiles such as p95 and p99 can reveal tail behavior that an average obscures, though which percentile actually matters most depends on the product's own SLO and traffic pattern, not a universal rule. Since a meaningful share of users having a bad experience shows up somewhere in that tail rather than at the average, a latency sla built entirely around average E2E time can technically be met while those users still have a noticeably worse experience than the headline number suggests.
When E2E latency is worse than expected, the useful question isn't "why is it slow," it's "which component is slow." Logging TTFT and total generation time separately, rather than only the combined E2E figure, is the single highest-leverage change for actually diagnosing this. Current server-side metrics from serving engines like vLLM and TensorRT-LLM already expose queue time, TTFT, and inter-token latency as separate histograms specifically so this diagnosis doesn't have to be reconstructed by hand.
A slow TTFT with fast, consistent decode points toward prefill-side fixes: prompt length, prefix caching, or queueing under load. A fast TTFT with slow or inconsistent decode points toward serving-engine tuning or hardware. Consistently high E2E with both components individually reasonable often points to queueing, meaning the real fix is capacity, not any single-request optimization at all.
Decomposing and tuning each of these components separately, prefill, decode, queueing, and delivery, is exactly the kind of ongoing serving-layer work a managed inference provider is positioned to absorb rather than something every team needs to instrument and re-tune independently. packet.ai's Token Factory is being built with these considerations handled at the serving layer. It's currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: September 3, 2026. For the individual components that compose E2E latency, see the packet.ai What Is TTFT guide, Prefill vs Decode guide, and streaming guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
