Rent GPU for AI: VRAM Requirements Guide for Every Major Model (2026)

How much VRAM you need depends on the model you want to run and whether you are doing inference or training. This guide covers the numbers for popular model sizes, precision formats, and workload types - so you can rent the right GPU on the first try.

The Formula: How Much VRAM Does a Model Need?

The VRAM requirement for loading a model has a straightforward formula:

VRAM for weights (GB) = (number of parameters) x (bytes per parameter)

Bytes per parameter by precision:

• FP32 (full precision): 4 bytes per parameter
• FP16 / BF16 (half precision): 2 bytes per parameter
• INT8 (8-bit quantisation): 1 byte per parameter
• INT4 / Q4 (4-bit quantisation): 0.5 bytes per parameter
• FP8: 1 byte per parameter

For a 7 billion parameter model:

• FP32: 7B x 4 bytes = 28 GB
• FP16 / BF16: 7B x 2 bytes = 14 GB
• INT8: 7B x 1 byte = 7 GB
• Q4_K_M: 7B x 0.5 bytes = approximately 4.5 GB (actual GGUF Q4_K_M is closer to 4.1-4.5 GB depending on the model's layer structure)

This gives you the weight memory. Total VRAM requirement = weight memory + KV cache + framework overhead. For inference, add 20-40% to the weight number for a working budget. For training, multiply by 4-8x.

VRAM Requirements by Model Size

"Minimum GPU" assumes inference only (no training), single instance, and moderate concurrency (8-16 concurrent requests at 4K context). For higher concurrency or longer context, scale up accordingly.

KV Cache: The Variable That Changes Everything

KV cache is the memory used to store attention keys and values across the context window. Unlike model weights (which are fixed once loaded), KV cache scales with batch size and context length. For large models at high concurrency, KV cache can exceed the weight memory requirement.

KV cache size per token (approximate formula):

KV cache per token (bytes) = 2 x num_layers x num_kv_heads x head_dim x bytes_per_element

For Llama 3.1 70B (80 layers, 8 KV heads, 128 head dim, FP16):

2 x 80 x 8 x 128 x 2 = 327,680 bytes per token = 0.328 MB per token

At 8K context with 16 concurrent requests:

0.328 MB x 8,192 tokens x 16 requests = 43 GB of KV cache

Total VRAM for Llama 3.1 70B at 8K context, 16 concurrent, FP16: 140 GB weights + 43 GB KV cache = 183 GB. This requires at least two A100 80GB GPUs (160 GB combined, tight) or a single RTX 6000 Pro at 96 GB if you use Q4_K_M quantisation and cap context at 4K.

This is why context length is the variable most people underestimate when sizing GPU memory. Moving from 4K to 32K context quadruples the KV cache requirement.

VRAM for Training

Training requires significantly more VRAM than inference because you need to store gradients, optimiser states, and intermediate activations in addition to the model weights.

For standard full fine-tuning with Adam optimiser in mixed-precision (FP16 forward, FP32 master weights):

• Model weights (FP16): 2 bytes per parameter
• Gradients (FP16): 2 bytes per parameter
• Optimiser states - Adam stores first and second moment (FP32): 8 bytes per parameter
• Master weights (FP32): 4 bytes per parameter
• Activations: varies by batch size and model architecture

Total (excluding activations): 2 + 2 + 8 + 4 = 16 bytes per parameter. For a 7B model: 7B x 16 bytes = 112 GB. This is 8x the inference requirement.

In practice, gradient checkpointing (recomputing activations instead of storing them) reduces VRAM at the cost of compute time. With gradient checkpointing, a 7B full fine-tune on an A100 80GB is feasible at small batch sizes.

LoRA and QLoRA reduce the training VRAM requirement significantly by only training a small set of adapter parameters:

• QLoRA 7B: 4-bit base model (4.5 GB) + adapter parameters + optimiser states = approximately 8-12 GB total
• QLoRA 70B: 4-bit base model (40 GB) + adapter parameters + optimiser states = approximately 48-56 GB total

QLoRA makes 70B fine-tuning feasible on a single A100 80GB. Full fine-tuning of a 70B model requires 8x A100 80GB or equivalent.

GPU Options on packet.ai by VRAM

Worked Examples: Picking the Right GPU

Example 1: Running Llama 3.1 8B inference for a production API

Weight VRAM: 8B x 2 bytes = 16 GB. KV cache at 4K context, 32 concurrent: approximately 4 GB. Total: 20 GB. Recommendation: L40S (48 GB) gives headroom for burst concurrency. RTX 6000 Pro (96 GB) would let you run two instances on one GPU.

Example 2: Running Llama 3.1 70B inference (FP16)

Weight VRAM: 70B x 2 bytes = 140 GB. KV cache at 4K context, 8 concurrent: approximately 8 GB. Total: 148 GB. Recommendation: Two A100 80GB GPUs with tensor parallelism (total 160 GB), or a single B200 (192 GB) which fits the full model with KV cache headroom.

Example 3: Fine-tuning Llama 3.1 70B with QLoRA

Base model in Q4: approximately 40 GB. Adapter parameters and optimiser states: approximately 8-12 GB. Total: 48-52 GB. Recommendation: Single A100 80GB (comfortable fit with gradient checkpointing) or RTX 6000 Pro (96 GB, more KV cache headroom during evaluation steps).

Example 4: Full fine-tuning of Llama 3.1 7B

Parameters x 16 bytes = 7B x 16 = 112 GB. With gradient checkpointing, this becomes feasible at around 80 GB. Recommendation: Single A100 80GB with gradient checkpointing enabled, small batch size (1-2), and ZeRO Stage 2 or 3 if using DeepSpeed.

Example 5: Image generation - SDXL 1.0

SDXL requires approximately 6-8 GB VRAM for the base model at FP16. With refiner and VAE loaded: 10-12 GB. Recommendation: L40S (48 GB) lets you run batches of 4-8 images simultaneously. RTX 6000 Pro (96 GB) allows larger batches or running multiple pipelines in parallel.

Multi-GPU Configurations

For models that exceed single-GPU VRAM (primarily 70B+ FP16 or 405B), you need tensor parallelism or pipeline parallelism across multiple GPUs.

Tensor parallelism splits each layer across GPUs - each GPU holds a fraction of every weight matrix. This requires high-bandwidth interconnect (NVLink preferred, PCIe works but is slower). For A100 pairs on packet.ai, NVLink is available on 8-GPU nodes - check the cluster configuration for your instance type.

Pipeline parallelism assigns different layers to different GPUs. Lower interconnect bandwidth requirement but introduces pipeline bubbles (GPU idle time waiting for the previous stage to finish).

For most inference use cases, tensor parallelism is preferred for latency. For training with very large models, a combination of tensor and pipeline parallelism (3D parallelism) is common.

vLLM handles tensor parallelism automatically with the --tensor-parallel-size flag:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --dtype bfloat16

This splits the 70B model across 2 GPUs. Requires both GPUs to be visible to the process (use CUDA_VISIBLE_DEVICES=0,1).

Quantisation Trade-offs

Quantisation reduces VRAM at the cost of model quality. The trade-off is generally acceptable for inference but less so for training (training typically uses FP16 or BF16).

For production inference, FP16 or BF16 is preferred when VRAM permits. INT8 (via GPTQ or AWQ) is a reasonable second choice with minimal quality impact. Q4_K_M works well for running larger models on single GPUs but should be validated against your specific task before deploying in production.

Quick Reference: Which GPU to Rent

The shortest path to the right GPU:

• 7B model, FP16 inference: L40S (48 GB) - fits with room for KV cache
• 13B model, FP16 inference: L40S (48 GB) - tight; RTX 6000 Pro (96 GB) for more concurrency
• 30B model, FP16 inference: A100 80GB or RTX 6000 Pro (96 GB)
• 70B model, FP16 inference: 2x A100 80GB or B200 (192 GB)
• 70B model, Q4 inference: RTX 6000 Pro (96 GB) - fits on single card with headroom
• QLoRA fine-tuning, 7B: L40S (48 GB)
• QLoRA fine-tuning, 70B: A100 80GB or RTX 6000 Pro (96 GB)
• Full fine-tuning, 7B: A100 80GB (with gradient checkpointing)
• Image generation (SDXL): L40S (48 GB) for batched throughput
• 405B model, Q4: 4x A100 80GB or 2x B200 (384 GB combined)