A chatbot needs fast TTFT. A batch job needs high throughput. Here is how to tell which metric actually matters for what you are building.
Time to first token and tokens per second measure two different things, and which one actually matters for your app depends entirely on what you're building. The ttft meaning is specifically about the wait before anything appears; inference speed once the response is underway is a separate question entirely. A chatbot cares about how quickly the first token appears. A batch job cares much more about how much work the system can push through. Most real applications need some combination of both. This guide covers what each metric captures, when each one is the one to optimize for, and what to do when your app genuinely needs both.
Key takeaways
One terminology trap is worth clearing up first. TTFT is fairly straightforward: request in, first token out. Tokens per second is less so. Different benchmarking tools calculate it differently, some include the time spent waiting for the first token in the calculation, others report decode throughput separately from that. If you're comparing systems or providers, check the methodology behind a quoted TPS figure before comparing the number directly. For a more precise picture of the token-by-token streaming experience specifically, time per output token (TPOT) and inter-token latency (ITL) are the more rigorously defined measures, and most serving engines, vLLM included, report these as distinct metrics rather than folding everything into one TPS number.
Time to first token is the delay between sending a request and seeing the first token of the response appear, sometimes described as first token latency. It captures request queueing, prompt processing (prefill), and any network delay along the way, and it stops the instant the first token shows up. Tokens per second llm figures, along with the more precisely defined TPOT and ITL metrics, measure something that starts only after that: how quickly the rest of the response streams out, token by token, once generation is underway. For the full mechanics of TTFT specifically, including what drives it up and how to benchmark it properly, see the packet.ai What Is TTFT guide.
These two numbers, sometimes shorthanded as TTFT and TPS LLM metrics, move independently of each other. A system can have a fast TTFT and slow per-token generation: the first word appears almost instantly, then the rest streams out with visible pauses. Or the reverse: a noticeable pause before anything appears, followed by a response that streams out quickly once it starts, which is the typical profile of reasoning models that perform additional computation before the first visible token. Neither pattern is universally better; which one feels better depends entirely on what the user is actually waiting for, and neither one is simply "model latency" as a single undifferentiated number.
TTFT dominates perceived speed in any interactive, conversational context. A chat interface, a coding assistant answering inline, a customer support bot, a voice assistant, all of these involve a person actively waiting and watching for the response to begin. Research on human perception of delay is consistent on this point: a pause before anything happens reads as the system being broken or unresponsive, even if the eventual full response arrives quickly once it starts. A 200-millisecond TTFT feels immediate. A 1.5-second TTFT feels stalled, regardless of how fast the rest streams afterward.
Short-answer use cases are especially TTFT-sensitive, since there's barely any streaming to notice at all. If a response is one sentence or a single factual answer, TTFT is essentially the entire perceived latency, and per-token generation speed barely factors in. Multi-turn conversational agents, live coding autocomplete, and voice interfaces all fall into this category, and it's exactly where the techniques that specifically target TTFT, prefix caching in particular, earn their keep. See the packet.ai guide to reducing LLM inference latency for a full breakdown of which techniques target which metric.
One exception worth keeping in mind: a low TTFT isn't always the right goal. Reasoning models deliberately spend additional time computing before producing any visible output, which shows up as a higher TTFT by design, not as poor infrastructure. Comparing a reasoning model's TTFT directly against a standard model's isn't a fair comparison, since the two are doing genuinely different amounts of work before the first token appears.
For long responses, per-request generation speed matters because the user, or a downstream process, spends real time watching or waiting for the response to stream out. A long report, a large piece of generated code, or a lengthy document draft depends on sustained per-request generation speed far more than on how quickly the first token appeared, since there's enough output for small differences in streaming speed to actually accumulate.
For batch workloads, aggregate throughput becomes the more useful measure, and it's a genuinely different number from per-request speed: it's about how much work the whole system can complete overall, not how fast any single request streams. Summarizing ten thousand documents overnight, generating embeddings for a large dataset, or running an evaluation suite across many prompts all share the same property: nobody is watching any individual request's first token arrive, so optimizing for that would be solving a problem nobody has. What matters there is how many requests the system can get through per hour, which is an aggregate throughput question, not a first-token question or even a per-request generation-speed question. Continuous batching, one of the techniques covered in the inference latency guide above, is specifically an aggregate throughput optimization for exactly this kind of workload.
The underlying question worth asking honestly: is anyone watching this response arrive in real time? If yes, TTFT is probably your priority. If the output gets consumed after the fact, whether by a person reading a finished document or a downstream system processing a batch of results, TPS and total throughput matter more than how quickly any individual response started.
Any llm ttft benchmark worth trusting reports both numbers separately rather than collapsing them into one generic "latency" figure, since a single blended number hides exactly the tradeoff this post is about. When comparing providers or configurations, look for TTFT latency reported at a percentile, not just an average, alongside a separate TPS figure measured under realistic concurrency. A vendor quoting only one number, or only a best-case average, is making it hard to tell which of your actual priorities their system serves well.
Some real products don't fit neatly into either category. A coding assistant might need fast TTFT for inline autocomplete suggestions and high TPS for generating an entire file on request. A customer support product might need fast TTFT for the live chat widget and high throughput for the nightly job that summarizes every conversation from the day.
The practical answer in these cases usually isn't picking one metric to optimize globally, it's routing different request types to configurations tuned for each. A live, low-latency path prioritizing TTFT, often served with prefix caching and dedicated capacity, handles the interactive traffic. A separate, throughput-tuned path with more aggressive batching handles everything else. Trying to make one configuration excellent at both simultaneously usually means compromising on both, since some of the techniques that help one metric, like waiting to batch requests together, actively work against the other.
Since TTFT and TPS respond to different techniques, packet.ai's Token Factory is being built to apply the relevant optimizations, prefix caching and dedicated infrastructure for TTFT, continuous batching and efficient serving for throughput, at the serving layer rather than requiring you to configure each one yourself. Token Factory is currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: August 26, 2026. For the full mechanics of time to first token, see the What Is TTFT guide. For which techniques target which metric, see the 7 techniques to reduce LLM inference latency guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
