LLM Inference Cost by Model Size in 2026: 8B vs 70B vs 405B

LLM inference cost scales roughly 10x for every 10x increase in active parameter count: self-hosting an 8B model on a single RTX 4090 at $0.39/hr works out to roughly $0.22 per million output tokens at batch=8, while a dense 405B model on 8x A100 at FP8 costs around $3.50 per million output tokens at the same utilisation.

LLM inference cost is the GPU compute and memory consumed every time a model generates a token. It is not a fixed fee per model; it scales with parameter count, request length, and GPU utilisation. Most developers encounter it for the first time as a per-token rate on a pricing page. That number alone tells you almost nothing, because the same model on different hardware and at different batch sizes can cost 2x to 20x more or less. This guide gives you the actual math: VRAM requirements, GPU configurations, and cost-per-million-tokens at realistic batch sizes across the 8B, 70B, and 405B tiers.

For the complete GPU-level break-even analysis and managed API cost comparison, see the companion guide on LLM inference cost in 2026. For a managed inference API with no GPU setup, visit packet.ai Token Factory (pricing launching soon). This post is also part of the LLM inference explainer cluster.

What Is LLM Inference Cost and Why Does Model Size Drive It?

Every token a model generates requires a full forward pass through all of its active parameters. A 70B dense model does roughly 8.75x more arithmetic per token than an 8B model. That arithmetic needs VRAM, compute, and time. The VRAM requirement sets a hardware floor you cannot go below. At FP16 (2 bytes per parameter):

VRAM for weights = Parameters x bytes per parameter

8B  model at FP16 = 8B  x 2 bytes = 16 GB
70B model at FP16 = 70B x 2 bytes = 140 GB
405B model at FP16 = 405B x 2 bytes = 810 GB

Add 15-25% for KV cache and activations at moderate batch sizes.

VRAM for weights is the floor, not the ceiling. At production batch sizes (8 to 16 concurrent requests) and context lengths above 4K tokens, the KV cache adds significant memory on top. A 70B model at BF16 adds approximately 0.31 MB of KV cache per token per context: at 8K context and 16 concurrent requests, that adds roughly 40 GB on top of the 140 GB weight footprint. For a full explanation of how precision formats affect memory, see the FP8 vs FP16 vs BF16 precision guide.

VRAM Requirements and Minimum GPU Configuration by Model Size in 2026

The table below maps model size tiers to minimum GPU configurations available on packet.ai, using the most common quantisation levels at each tier. All prices are from the packet.ai pricing page. For a deeper look at quantisation formats, see the guide to INT8, INT4 and FP8 model quantisation.

The RTX 6000 Pro at $0.66/hr is the most cost-efficient single GPU on packet.ai for models in the 13B to 70B range. Its 96 GB GDDR7 ECC fits a 70B model at 4-bit quantisation on one card, which avoids tensor parallelism overhead entirely.

Self-Hosted Cost per Million Tokens: The Formula and Real Numbers

Self-hosted cost per million tokens depends on three variables: GPU hourly price, throughput in tokens per second, and GPU utilisation. The formula:

Cost per 1M output tokens =
  (GPU $/hr) / (tokens/sec x 3,600 / 1,000,000) / utilisation

Example: A100 at $1.43/hr, 70B model, batch=8, 70% utilisation
At batch=8, throughput reaches ~320 tok/s:
= $1.43 / (320 x 3,600 / 1,000,000) / 0.70
= ~$1.77 per million output tokens

Batch size is the single biggest lever on self-hosted inference cost. An A100 running a 70B model at single-stream (batch=1) generates roughly 45 tokens per second. At batch=8 with continuous batching via vLLM or SGLang, the same GPU reaches 320 to 380 tokens per second, an 8x throughput increase at the same hourly cost. These throughput figures are benchmarked and verified in the A100 vs H100 cost benchmark.

Self-hosted 70B inference on an A100 at $1.43/hr, running at batch=8 and 70% utilisation, costs approximately $1.77 per million output tokens — lower than most hosted 70B APIs at that volume, but only when the GPU stays busy. For VRAM sizing across all model sizes, see the GPU VRAM requirements guide.

packet.ai Token Factory: Managed Inference API (Launching Soon)

packet.ai Token Factory is a managed OpenAI-compatible inference API running on B200 infrastructure. For teams that do not want to manage GPU instances, vLLM configuration, or batch tuning, Token Factory handles the infrastructure and you pay per token.

Token Factory runs on packet.ai's owned B200 infrastructure with an overcommit scheduler targeting 80 to 100% GPU utilisation, versus the 20 to 40% typical at most shared cloud providers. That utilisation advantage is structural: it directly reduces the compute cost per token without any price negotiation or commitment required from you.

For teams that need GPU access today for self-hosted inference while waiting for Token Factory pricing, the cheapest entry points on packet.ai are the RTX 4090 at $0.39/hr (8B to 13B models) and the RTX 6000 Pro at $0.66/hr (13B to 70B at INT4). packet.ai GPU instances are available on-demand with no contract and no minimum commitment; pay by the minute and cancel any time. See how packet.ai compares to other providers in the RunPod alternatives guide.

MoE vs Dense Models: Why Parameter Count Does Not Predict Inference Cost

The most common misconception in model cost analysis is treating MoE and dense models as if total parameter count predicts inference cost. It does not.

A dense 70B model activates all 70 billion parameters on every token. A MoE model like DeepSeek-V3 has 671B total parameters but only activates approximately 37B per token via expert routing. The per-token arithmetic cost sits closer to a 37B dense model than to a 671B dense model. What differs is the VRAM requirement: you still need to load all 671B parameters into GPU memory, which demands 8+ B200s or H200s.

Llama 3.1 405B is the most expensive model to self-host in 2026, not because of total parameters, but because it is the largest dense model with no MoE routing. Every token activates all 405B parameters. DeepSeek-V3 with 671B total parameters costs less per token at inference despite being larger on paper, because only 37B fire per token. For most production quality requirements, a 70B dense model or a well-optimised MoE model will deliver comparable output at a fraction of the 405B GPU cost.

Which GPU to Pick for Each Model Size: A Solo Builder's Cost Guide

Pick the cheapest GPU that fits your model at the quantisation level you need. Do not pay for NVSwitch or high-bandwidth interconnects for single-GPU workloads. All packet.ai GPU instances are on-demand with no contract, no minimum spend, and per-minute billing.

* RTX 6000 Pro 70B INT4 throughput estimated at ~200 tok/s at batch=8 from GDDR7 memory bandwidth (960 GB/s) relative to A100 HBM2e (2 TB/s) with INT4 weight compression factor; verified against llama.cpp GGUF Q4_K_M community benchmarks on GDDR7 96 GB cards.

The RTX 4090 at $0.39/hr is the same GPU that gaming PCs use, running 24/7 in a data centre. It is the cheapest 24 GB GPU on any neocloud, handles 7B to 13B models at FP16 on one card, and requires no NVLink, no tensor parallelism, and no ops overhead. RunPod charges $0.69/hr for the same card.

Frequently asked questions

Last reviewed: August 31, 2026. packet.ai GPU instances start at $0.39/hr for the RTX 4090 (8B models) and $0.66/hr for the RTX 6000 Pro (70B at INT4). No contract, no minimum spend, per-minute billing. For a managed no-ops inference API, packet.ai Token Factory is launching soon.