Jensen Huang declared 2026 the year of inference at GTC. Token factories are what that actually looks like in hardware, and the economics behind them are changing how teams size GPU infrastructure.
Published September 21, 2026 · 12 min read · AI Infrastructure
A token factory is a data center optimized to produce AI tokens at the lowest possible cost per watt. Jensen Huang named it at GTC 2026 as the defining model for the inference era, and it changes how every team should think about GPU infrastructure.
Key takeaways
For years, AI infrastructure conversations started with training. How many GPUs do you need? How long will the run take? What is the checkpoint strategy?
Reasonable questions, when training was where most of the compute went.
It is not anymore.
At GTC 2026, Jensen Huang told an arena full of engineers and investors that the inference inflection point had arrived. Every chatbot reply, every agent tool call, every RAG lookup, every code completion: all of it produces tokens, continuously, at a scale that makes training look episodic by comparison. By 2026, inference crossed two-thirds of all AI compute globally (Deloitte). The number is heading higher, not lower.
The infrastructure framing had to catch up. The term Huang reached for: token factory.
This is what that actually means: the economics, the hardware, the stack, and what it looks like for teams running inference on GPU cloud instead of building their own data centers.
If you want the basics first, the LLM inference explainer and the inference vs training breakdown are good starting points. The story of why packet.ai built its own inference product around this model is in Why We Built Token Factory.
A token factory is a data center or GPU cluster reconceived as a production facility whose primary output is tokens. Not uptime. Not FLOPS. Not stored data. Tokens: the units of text, reasoning, and action that AI models produce during inference.
Huang framed it with a formula deliberately aimed at board rooms, not just engineers:
Revenue = Tokens per Watt x Available Gigawatts
Jensen Huang, NVIDIA GTC 2026
Two variables. Both matter in different ways.
Tokens per watt is an efficiency metric: how much inference output you extract per unit of energy consumed. Available gigawatts is the scale variable: how much sustained power your facility can draw. Revenue grows when you improve either. Most teams building on GPU cloud can only control the first one, and that is exactly where hardware selection, inference engine choice, and GPU tier decisions have real monetary consequence.
This is not a metaphor. Once power becomes the binding constraint (and it definitively has become one in 2026, with major data center operators competing for grid access in Virginia, Texas, and Frankfurt), GPU count matters less than tokens per megawatt. A facility that runs 10x more tokens from the same power envelope earns 10x more revenue without adding infrastructure cost. That is the Blackwell pitch in one sentence.
NVIDIA now describes itself as building "AI factories" rather than chips. The shift from accelerator vendor to full-stack inference platform is real: from CUDA to TensorRT-LLM, from individual GPUs to NVLink fabrics tuned specifically for inference communication patterns rather than training all-reduce operations.
packet.ai is a $19M seed-backed GPU cloud built around inference economics: Blackwell B200, RTX 6000 Pro, A100, and L40S, at rates that make the tokens-per-dollar math work for teams that are not hyperscalers.
The hardware looks the same from the outside. Same NVIDIA B200 GPUs. Same HBM3e memory. Same NVLink interconnects. The difference is what gets optimized, and that changes everything downstream, from networking topology to cooling design to how you bill customers.
The networking gap is the one most teams underestimate. Training clusters run large, synchronized all-reduce operations, every GPU talking to every other GPU simultaneously, moving giant tensors. Inference is the opposite: one request arrives, a stream of tokens goes out, and the next request arrives before the first one finishes. High-frequency, small-packet, never-synchronized. Running inference on a fabric designed for training all-reduce means paying for InfiniBand bandwidth your workload barely uses, while taking a latency penalty on the patterns it actually runs.
The cooling gap bites operators who do not plan for it. Training runs for hours or days, then the cluster sits idle. Inference never idles. A token factory running 24/7 production traffic holds sustained thermal load around the clock, a fundamentally different cooling problem than handling peak burst during training.
In 2024, training and inference consumed roughly equal shares of AI compute. By 2026, inference had crossed two-thirds. The shift happened faster than most infrastructure teams expected, and three things drove it.
Reasoning models multiplied token counts. When models started generating chains of thought before answering (thinking tokens, extended reasoning traces) the token count per request jumped sharply. A response that previously required 200 output tokens might now require 2,000 thinking tokens before the actual answer. Same user request. 10x the compute.
Agentic workloads turned one request into dozens. An agent that plans, calls tools, observes results, and revises does not make one inference call per user query. It makes 20 or 40. What the user experiences as one interaction runs dozens of inference passes behind the scenes. For a detailed breakdown of the GPU bottleneck this creates, see the CPU bottleneck in agentic workloads.
Product deployment outpaced training. The number of production AI products shipping in 2025 and 2026 grew faster than new model training runs. Each new deployment adds continuous inference traffic. None of it adds training demand. The ratio shifts toward serving every quarter, automatically.
In May 2026, Google disclosed it was processing 3.2 quadrillion tokens per month across its products, a 7x increase year over year. That is the scale inference has reached at the top. For everyone else, it is the direction of travel.
Token factories are what the infrastructure response to that scale looks like.
FLOPS are a fine metric for comparing GPU chips in a lab. They are a poor metric for running a production inference business.
The problem: FLOPS measure peak theoretical compute throughput. They say nothing about memory bandwidth, which is what actually limits large model inference during the decode phase, when the GPU spends most of its time feeding the attention mechanism rather than doing matrix multiplications. And they say nothing about power consumption, which is the real cost ceiling once your data center has a fixed power budget.
A GPU with 2x the FLOPS does not produce 2x the inference tokens if it is memory-bandwidth limited. This is not a theoretical concern. It is why Blackwell beat Hopper by far more than the raw FLOPS ratio would suggest.
Tokens per watt captures both. It measures actual token output against actual power draw, directly mapping to the business equation: revenue from tokens sold, minus cost of electricity consumed.
NVIDIA Blackwell B200 achieves 60,000 tokens per second per GPU on gpt-oss workloads using TensorRT-LLM, at two cents per million tokens at peak efficiency (NVIDIA InferenceMAX v1, 2026). That is what the current ceiling on tokens per watt looks like in numbers.
For teams choosing GPU SKUs: (tokens per watt) x (power you can draw) = total token capacity from your infrastructure budget. Work backward to cost per token, compare against your pricing or managed API alternative, and you have a real ROI model, not a spec sheet comparison.
The Blackwell architecture deep dive covers why these numbers moved so far from Hopper. For latency-side metrics, the TTFT vs tokens per second guide explains which throughput metric matters for your application type.
A token factory is not a product you buy. It is a stack: six layers, each contributing to or subtracting from the tokens-per-watt number at the top. Every layer is a decision, and every decision has a cost.
The fabric between these layers is tuned for inference communication patterns: small, frequent, never-synchronized, the opposite of what training all-reduce operations need. Co-packaged optical networking, announced at GTC 2026, specifically targets this latency on high-frequency token-streaming traffic.
For teams running batch jobs alongside interactive inference, the economics of each layer look different. The batch inference throughput economics post covers the offline tradeoffs.
You can run inference on a training cluster. Teams do it constantly, usually because the training cluster is already sitting there and the inference budget has not been approved yet.
It works. It also leaves a lot on the table.
Training clusters optimize for MFU (model FLOP utilization) across multi-day runs. They use InfiniBand for high-bandwidth all-reduce. They are sized for peak training batch size, not hundreds of concurrent streaming requests. The latency characteristics that matter for training (time to complete a checkpoint) are different from those that matter for inference: time to first token on a user request.
A 1-megawatt cluster running Hopper GPUs generates approximately 180,000 tokens per second at maximum inference volume. The same power envelope on Blackwell B200 hardware produces roughly 10x that output (NVIDIA, 2026).
1 MW of Hopper at inference: ~180,000 tokens/sec. 1 MW of Blackwell at inference: ~1,800,000 tokens/sec. When inference is the workload, generation matters more than the power budget.
Token factory thinking pushes hardware selection toward inference-optimized SKUs, which are not always the same as training-optimal ones. The TTFT vs tokens per second guide explains which throughput metric matters for your application type, and how that changes the right GPU decision.
You do not need to build a data center to run token factory math. GPU cloud makes it computable from a billing dashboard.
Here is a worked example at current packet.ai pricing:
The B200 produces more tokens per watt. The RTX 6000 Pro produces more tokens per dollar at typical utilization for smaller models. The right GPU for your token factory is not the fastest one. It is the one where your model fits in VRAM, your utilization is high enough to amortize the hourly cost, and the memory bandwidth profile matches your workload.
For smaller models (7B-13B), an L40S at $0.92/hr Dedicated or an A100 at $1.43/hr Dedicated often beats both on cost per token because aggressive batching with room to spare matters more than raw throughput. Run the math for your specific workload before committing to hardware.
Pricing verified from packet.ai/pricing as of September 2026.
For multi-node deployments, packet.ai GPU clusters use InfiniBand fabric with pricing typically 30% below retail on 12-month terms. 64-node B200 clusters land around $2.80-$3.20/GPU-hr committed. Monthly commits on any single-GPU tier save up to 20% against the hourly rate.
Most teams reading this are not building data centers. They are renting GPU capacity and trying to make the inference economics work.
The good news: GPU cloud gives you token factory economics without capital expenditure, power procurement, or cooling engineering. On packet.ai, a $19M seed-backed GPU cloud built around AI inference, the relevant decision is which tier matches your workload pattern:
Dedicated PODs: production inference
Dynamic PODs: dev and batch
For teams that want to skip the serving infrastructure entirely, packet.ai Token Factory is a managed LLM inference API launching soon: model serving, request routing, and per-token billing through an OpenAI-compatible endpoint, with no GPU infrastructure to manage. Join the waitlist for early access.
Last reviewed: September 21, 2026. Pricing verified from packet.ai/pricing. For GPU options suited to token factory workloads, browse cluster options or join the Token Factory waitlist, launching soon.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
