Your LLM API bill is not fixed. At 1B output tokens/month, Claude Sonnet 4.6 costs $15,000. Self-hosted Llama 4 70B on an H200 costs $130. Here is the math.
LLM inference cost in 2026 ranges from $0.10 per million input tokens for budget-tier APIs to $0.18 per million output tokens for self-hosted Llama 4 70B on an H100 at batch=8 with vLLM, a 50x spread that depends almost entirely on whether you rent the model or rent the GPU.
Key takeaways
LLM inference cost is now a cost of goods sold line item for most AI products. The question teams face is not whether inference is expensive. It is which pricing model fits their traffic shape. A low-volume agent that fires 10,000 tokens per day belongs on a managed API. A RAG pipeline serving 10 million tokens per day is probably paying 10x too much if it is still on OpenAI.
This post maps the full llm inference cost landscape for 2026: managed API pricing by tier, open-weight hosted API rates, self-hosted math on H100 and H200, and the break-even crossover that determines which path wins at your volume. For the infrastructure guide behind these numbers, see the packet.ai Token Factory for API-first LLM inference.
Every LLM API bills in tokens, not words or requests. A token is approximately 0.75 words in English. A 1,000-word document is roughly 1,300 tokens. Providers publish prices per million tokens (MTok), charged separately for input (your prompt, context, system instructions) and output (the model's response).
Output tokens are always more expensive than input tokens, typically 3 to 5x more, because generation requires a full forward pass through the model for each token, while input is processed in parallel. A workload sending 4,000-token prompts and receiving 200-token responses has 95% of its token volume on the cheap side. A workload generating long code completions or reports has the opposite problem.
Three hidden cost multipliers matter more than the headline rate. First, context overhead: a RAG app sending 4,000-token input payloads per user query (system prompt + retrieved documents + user message) will find that 80 to 90% of its token bill comes from input context written once and repeated on every call. Second, prompt caching: OpenAI and Anthropic both offer cached input at 50 to 90% off standard rates. At an 80% cache hit rate on a fixed system prompt, effective input cost can fall below $0.30 per million tokens even on Claude Sonnet 4.6. Third, batch processing: both OpenAI and Anthropic offer roughly 40 to 50% discounts for offline batch jobs, reducing GPT-5.4 from $2.50 to $1.25 input and Claude Sonnet 4.6 from $3.00 to $1.50 input per million tokens.
Frontier models from OpenAI, Anthropic, and Google are the reference point for capability but not for cost. They command premium rates because they lead on reasoning, coding, and long-context comprehension, on workloads where open-weight alternatives still trail by a measurable margin.
Claude Sonnet 4.6 at $3/$15 per million tokens (input/output) is the production workhorse for teams where quality matters: coding, document analysis, instruction following. At that rate, 100 million output tokens per month costs $1,500. That figure sounds manageable until you model a chat product where each conversation generates 2,000 output tokens per session across 50,000 daily active users. That is 100 million output tokens every day, or $45,000 per month.
The batch discount changes that math significantly. Anthropic's batch API brings Claude Sonnet 4.6 output to $7.50 per million tokens, which is $22,500 per month for the same 100M daily output tokens, assuming the workload tolerates hours-latency turnaround (evaluation runs, data enrichment, content generation pipelines).
Open-weight models are the price floor that frontier providers compete against. The same model weights run on infrastructure owned by Together AI, Fireworks AI, Groq, and DeepInfra, who compete on price and SLA to attract high-volume customers. The result: Llama 4 Maverick at $0.22/$0.88 per million tokens on Together AI, versus Claude Sonnet 4.6 at $3/$15. That is 13x cheaper on input, 17x cheaper on output.
The catch is not quality. Open-weight models now match or beat closed frontier on most non-reasoning workloads (high-volume chat, RAG, classification, summarisation, code completion on well-defined tasks). The catch is infrastructure: these rates assume you trust Together AI or Fireworks AI to maintain SLAs, handle rate limits, and stay available during traffic spikes. For workloads where those risks are acceptable and volume is high, open-weight hosted APIs are the correct choice.
Llama 4 Scout on Together AI costs $0.18/$0.59 per million tokens against Claude Sonnet 4.6 at $3/$15, which is 16x cheaper on input, 25x on output, for workloads where the quality trade-off holds (verified April 2026, source: PECollective LLM API Pricing Comparison).
Self-hosted inference replaces per-token API bills with per-hour GPU bills. The formula is straightforward: (GPU hourly cost) / (tokens per second throughput x 3,600) x 1,000,000 = cost per million tokens. The hard part is getting an honest throughput number for your specific workload.
An packet.ai H100 SXM runs Llama 4 70B at approximately 95 tokens per second single-stream with vLLM at FP16, rising to 380 tokens per second at batch=8 (8 concurrent requests). At packet.ai H100 pricing of $2.50/hr, the math works out as follows.
* H100 figures: AITOT Inference Benchmark (July 2026), vLLM v0.25 at FP16. H200 figures estimated at 1.45x H100 from HBM3e bandwidth ratio (4.8 TB/s vs 3.35 TB/s). Actual throughput varies with prompt length distribution and KV cache pressure.
These numbers are peak throughput at full batch saturation. The honest adjustment is GPU utilisation. A production system running at 40% average utilisation, common for chat workloads with bursty traffic, has an effective cost per million tokens 2.5x higher than the table above. That is why the break-even crossover is not purely a math problem.
At 85% GPU utilisation on an H200 SXM at $2.49/hr, Llama 4 70B self-hosted on packet.ai costs approximately $0.10 to $0.15 per million output tokens, versus $0.88 on Together AI's Llama 4 Maverick hosted API at the same model. The hardware wins at scale, but only if you keep the GPU busy.
The break-even crossover between self-hosting and a managed API depends on three variables: your daily token volume, your GPU utilisation rate, and whether you commit to reserved capacity. The rule of thumb from 2026 benchmarks: the crossover lands at roughly 2 to 5 million tokens per day on reserved GPU capacity over a 12-month window.
Monthly LLM output cost at 1B output tokens/month: self-hosted H200 vs managed API
Assumes 85% H200 utilisation, Llama 4 70B at batch=8, 1B output tokens/month. Self-hosted cost includes GPU only; add engineering overhead for total TCO comparison.
Below 2M tokens per day, the managed API wins on total cost when you factor in engineering overhead. Running your own vLLM stack requires someone to maintain it: version upgrades, OOM debugging, monitoring, and capacity planning. That is easily 10 to 20 hours per month for a production system, which at $150 to $200/hr is $1,500 to $4,000 in hidden cost that does not appear in the GPU bill.
Above 5M tokens per day on a predictable workload, self-hosting on reserved GPU capacity at packet.ai pricing beats every managed API option including the cheapest open-weight hosted alternatives. The arithmetic is not subtle at that scale: $0.13 per million tokens self-hosted versus $0.28 to $0.88 on hosted open-weight APIs means the engineering overhead pays for itself within the first month.
The three variables that move self-hosted LLM inference cost more than GPU choice are batch size, quantisation precision, and GPU utilisation. Getting all three right is the difference between $0.10 and $0.50 per million output tokens on the same hardware.
Batch size is the biggest lever. An H100 running Llama 4 70B at batch=1 costs $0.73 per million output tokens. At batch=8, it costs $0.18, a 4x reduction by filling the GPU. The trade-off is latency: larger batches increase time-to-first-token. For RAG pipelines and batch processing jobs, this is free money. For interactive chat requiring sub-200ms TTFT, batch saturation has a ceiling.
FP8 quantisation on H100 SXM and H200 SXM gives 1.5 to 1.8x throughput improvement over BF16 with minimal quality loss. On an H200 at $2.49/hr, switching from BF16 to FP8 for Llama 4 70B cuts the llm cost per token from approximately $0.21 to $0.13 per million output tokens, a 38% reduction with no additional spend. Enable it in vLLM with --dtype fp8.
GPU utilisation is the honest variable nobody publishes. Peak throughput numbers assume 100% batch saturation and continuous traffic. Real production systems run at 40 to 70% average utilisation due to traffic variance. A workload running at 50% average utilisation has an effective cost per million tokens twice the table value. Run the formula with your real duty cycle, not the vendor's reference benchmark.
Watch out
A production duty cycle of 40% means your real cost per million tokens is 2.5x the peak formula output. Always benchmark with your actual prompt length distribution and concurrency pattern, not the vendor's reference scenario. A workload averaging 8K input tokens behaves nothing like one averaging 500 tokens, even on identical hardware.
The GPU hourly rate is the direct input to self-hosted LLM inference cost. At high utilisation, a 50% difference in GPU rental price translates to a 50% difference in cost per million tokens. No amount of framework tuning closes that gap.
packet.ai H200 SXM at $2.49/hr is the lowest publicly listed on-demand rate for H200 as of July 2026, 44% below AWS and Azure's H200 pricing range. For an 8-GPU H200 cluster running Llama 4 70B at 85% utilisation, that differential saves approximately $25,000 per month compared to equivalent AWS capacity.
The B200 SXM changes the cost-per-token calculation further. At $3.75/hr on packet.ai with roughly 3x the H200 throughput on large models (per MLPerf reporting), the effective cost per million output tokens on a B200 falls below $0.05 for 70B+ models at high batch, below even the cheapest open-weight hosted APIs while running the model locally with full data control.
Last reviewed: July 27, 2026. Deploy self-hosted LLM inference on packet.ai H100, H200, or B200 clusters. Browse available GPU clusters, explore the packet.ai Token Factory for API-first inference, or check current GPU pricing at packet.ai H200 and packet.ai B200.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
