What Is Time to First Token (TTFT)? LLM Latency Explained
TTFT is the wait before the first word appears, not how fast the rest streams. Here is what causes high TTFT and how it differs from tokens per second.
packet.ai Team
August 18, 2026
Time to first token (TTFT) is the delay between sending a request to an LLM and receiving the very first token of its response. It's the metric behind that split-second feeling of "is this thing actually working," and it's become one of the most closely watched numbers for anyone evaluating an inference API or serving stack. The ttft meaning is straightforward once you separate it from the metric it's most often confused with: TTFT measures how long you wait before anything appears, not how fast the response streams once it starts. This guide covers what TTFT actually is, what causes high TTFT, how it differs from tokens per second, and how to benchmark it properly.
Key takeaways
TTFT measures the time from request arrival to the first generated token, including queueing, prefill, and network delay, not just raw GPU compute
TTFT and tokens per second (TPS) measure different things: TTFT is strongly influenced by prefill, but also includes queueing and network delay, while TPS measures decode speed. A model can have a fast TTFT and slow TPS, or the reverse
TTFT scales with prompt length, since the entire input must be processed during prefill before the first output token can be generated
Reasoning models have deliberately high TTFT by design, since they perform additional reasoning or computation before the first visible token appears. That's not a performance bug to fix
Batching improves overall throughput but can increase individual TTFT, since a request may wait for a batch to fill before processing starts
What Is TTFT?
Time to first token is the latency between when a client sends a request and when the first token of the model's response is generated and begins streaming back. What is TTFT actually capturing, precisely: request queueing time, prompt processing (prefill), and any network or routing delay along the way. It stops the moment the first token appears; everything after that belongs to a different metric.
TTFT matters because it determines when a system starts responding, while a separate metric, tokens per second, determines how quickly the response continues once it begins. A chat interface with a 200-millisecond TTFT feels immediate; one with a 1.5-second TTFT feels stalled, even if both generate the rest of the response at the same speed. That's the core reason TTFT gets so much attention: it's the model latency number that determines whether a product feels fast, independent of how fast it is once it gets going.
⚡ TTFT isn't the same as total latency
Total latency (sometimes called time to last token, or TTLT) is the full time from request to complete response. First token latency is just the first slice of that. A short response with a slow TTFT and a long response with a fast TTFT can have wildly different totals despite starting from the same first-token wait. Keep the two separate when reading benchmark numbers or comparing inference speed claims; a headline "latency" figure that doesn't specify which one it means is easy to misread.
TTFT vs Tokens Per Second: What's the Difference?
LLM inference happens in two phases: prefill, where the model processes the entire input prompt in one pass, and decode, where it generates the response one token at a time. TTFT latency is strongly influenced by prefill performance, though queueing and network delay are part of the number too. Tokens per second (TPS), the rate at which output streams once generation begins, is a measure of decode speed. These are genuinely separate numbers, not two views of the same thing, and a system can be strong on one and weak on the other.
Fast TTFT, slow TPS. The first token appears almost instantly, but the response then streams out slowly, word by word, with visible pauses. Feels responsive at first, sluggish once it's actually talking.
Slow TTFT, fast TPS. There's a noticeable pause before anything appears, but once the first token lands, the rest of the response streams out quickly. Feels laggy to start, snappy once it's moving. This is the typical profile of reasoning models, which perform additional computation before the first visible token.
A useful, everyday analogy: TTFT is the wait between placing an order and the first dish arriving at the table. TPS is how quickly the rest of the dishes keep coming after that. Total latency is the whole meal, start to finish. All three tell you something different, and a restaurant, or an inference API, can be excellent at one and mediocre at another.
One token is roughly three-quarters of a word in typical English text, so a tokens per second LLM figure translates fairly directly into a reading-speed comparison: 100 tokens per second is comfortably faster than most people read. Whether that matters more or less than TTFT genuinely depends on the application. A short factual answer lives or dies on TTFT, since there's barely any streaming to notice. A long generated report or piece of code depends more on sustained TPS, since the user is reading along as it streams rather than waiting for the whole thing up front.
What Causes High TTFT
High TTFT usually traces back to one or more of a handful of specific, identifiable causes, not a single generic "the model is slow."
Long prompts. Prefill has to process the entire input before the first output token can be generated, so TTFT scales directly with prompt length. A prompt-template change that quietly adds a few hundred tokens to every request, a longer system prompt, more few-shot examples, a bigger retrieved document, will measurably raise TTFT across the board, often without anyone noticing why.
Queueing and batching. A request doesn't always start processing the instant it arrives. If the serving system batches requests together for efficiency, an individual request may wait for a batch to fill before its prefill even begins. This is a genuine tradeoff, not a bug: batching raises overall system throughput at the cost of some individual TTFT.
Reasoning models, by design. Models that perform additional reasoning or computation before producing visible output have deliberately elevated TTFT, since that work happens before the first token a user sees. For these models, a high TTFT isn't a performance problem to chase down; it's the feature working as intended, and comparing their TTFT directly against a non-reasoning model's is comparing two different things.
Shared infrastructure contention. On shared GPU infrastructure, a co-located tenant's workload can saturate the PCIe bus, CPU, or network path your request depends on, spiking your TTFT even when your own GPU utilization looks completely normal. This is a real, measurable effect, not a theoretical one; the packet.ai noisy neighbor problem guide covers the mechanism and measured impact in depth, including why p99 TTFT specifically is the number that reveals it.
Two of the mechanisms behind TTFT are worth a specific mention, since they're where a lot of the actual engineering happens. Prefix caching lets a serving engine skip recomputing a prompt's prefix if it's already been processed for an earlier request, directly cutting TTFT on repeated system prompts or shared context; the packet.ai vLLM prefix caching guide covers exactly how that works. And the KV cache itself, the mechanism that lets a model reuse prior computation instead of reprocessing a growing sequence from scratch, is the foundational piece both prefix caching and prefill efficiency build on; see the packet.ai What Is KV Cache guide for the full mechanism.
How to Benchmark TTFT Properly
A single TTFT measurement tells you very little about how the system behaves under real traffic. Real benchmarking needs a few things done deliberately:
Measure percentiles, not just the average. A p50 (median) TTFT can look perfectly healthy while p99 TTFT is several times worse, driven by occasional queueing delays, batch timing, or infrastructure contention. Production systems commonly set an SLO on p99 specifically, since that's the tail experience real users actually hit some percentage of the time.
Standardize the prompt and output length. Since TTFT scales with prompt length, comparing one system's TTFT on short prompts against another's on long prompts isn't a fair comparison. Published LLM TTFT benchmarks typically fix a standard input and output length (for example, 500 input tokens and 200 output tokens) so results are actually comparable across providers or configurations.
Test under realistic concurrency. TTFT measured against an idle system with no other traffic tells you little about what users experience under real load, when queueing, batching, and shared infrastructure effects all become relevant. Benchmark at the concurrency levels you actually expect in production, not a single isolated request.
Open-source load-testing tools built specifically for LLM inference, rather than generic HTTP benchmarking tools, are worth using here since they understand streaming responses and can separate TTFT from total latency automatically rather than requiring you to compute it by hand.
Reducing TTFT with Token Factory
Since TTFT is driven by prefill time, queueing behavior, and infrastructure contention, reducing it usually comes down to controlling those three things directly rather than hoping a bigger GPU alone fixes it. packet.ai's Token Factory runs inference on dedicated infrastructure rather than shared, noisy-neighbor-prone hardware, which directly addresses the contention piece of the TTFT equation without requiring you to manage GPU provisioning or serving configuration yourself.
Frequently asked questions
Time to first token (TTFT) is how long you wait between sending a request to an LLM and seeing the first word of its response appear. It includes request queueing, the time spent processing your full prompt, and any network delay, but not the time it takes to finish streaming the rest of the answer.
No. TTFT measures the wait before the first token appears, strongly influenced by prompt processing but also including queueing and network delay. Tokens per second measures how fast the response streams once it starts, which is a function of decode speed. A system can have a fast TTFT and slow tokens per second, or the reverse; they're independent numbers that both matter for perceived speed.
The most common causes are long prompts (since the full input must be processed during prefill before the first token generates), queueing or batching delays, shared infrastructure contention from co-located tenants, and, for reasoning models specifically, additional reasoning or computation that runs before any visible output, which is a deliberate design choice rather than a problem to fix.
Reasoning models perform additional reasoning or computation before producing the first token a user actually sees, which adds real time before anything appears to stream. This is by design, not a performance defect, and comparing a reasoning model's TTFT directly against a standard model's isn't a fair apples-to-apples comparison, since the two are doing fundamentally different amounts of work before the first visible token.
Measure p99 as well as median TTFT, since the tail experience often looks very different from the average. Standardize prompt and output length across comparisons, since TTFT scales with prompt length. Test under realistic concurrency rather than a single isolated request, since queueing, batching, and infrastructure contention only show up under real load.
