Fine-Tuning LLMs on GPU Cloud: A100 vs H100, QLoRA Memory Math and Cost Per Run

Fine-tuning a 70B LLM on a single A100 80GB with QLoRA costs between $34 and $51 and takes 24 to 36 hours. H100 SXM (waitlist on packet.ai) trains the same job in approximately 8 to 12 hours due to its higher memory bandwidth and FP8 throughput. This post covers the VRAM math behind those numbers, a GPU-by-GPU cost comparison for four fine-tuning methods, and how to choose between QLoRA, LoRA, full fine-tuning, and PEFT on packet.ai GPU cloud.

Fine-tuning LLMs has become the standard path for adapting base models to specific tasks, domains, and writing styles. The economics changed dramatically in 2024 to 2025: QLoRA made fine-tuning accessible on a single consumer GPU, Unsloth cut memory further, and GPU cloud pricing fell to a point where a meaningful training run on a 70B model costs under $50 rather than $500. This post gives you the VRAM formulas, method comparisons, and real cost math to plan your next fine-tuning run. For a deep dive on FP8 and BF16 precision formats that affect training speed and memory, see the FP8 vs FP16 vs BF16 guide. For deploying the resulting model, see the vLLM deployment tutorial.

Fine-Tuning Methods Compared: Full, LoRA, QLoRA and Unsloth

Four methods dominate LLM fine-tuning in 2026. They trade off memory efficiency against training throughput against final model quality. The correct choice depends on your VRAM budget and whether the task requires maximum alignment or can tolerate a small quality gap.

QLoRA vs LoRA: When the Quality Gap Matters

LoRA fine-tunes the model at full BF16 precision, adding trainable rank-decomposition matrices to frozen base model weights. QLoRA does the same thing but quantises the base model weights to 4-bit NF4 first, then dequantises during the forward pass. The 4-bit quantisation introduces a small but measurable quality gap on tasks requiring precision: instruction following, code generation, and long-form coherence. For classification, sentiment, and domain adaptation on well-defined formats, QLoRA matches LoRA quality within noise.

The practical rule: if your task is evaluated on open-ended output quality (MMLU, HumanEval, your own human evals), start with LoRA on an H100 or H200. If you need to fit the job on a single cheaper GPU and the task tolerates a small gap, QLoRA or Unsloth QLoRA is the right call.

LLM Fine-Tuning VRAM Math: How to Calculate Memory Requirements

VRAM requirements for fine-tuning have three components that add together: base model weights, optimizer states and gradients, and activation memory during the forward pass.

Base Model Weight Memory

Weight memory is parameter count times bytes per parameter. A 70B model at BF16 (2 bytes) = 140 GB. At FP32 (4 bytes) = 280 GB. At 4-bit NF4 (0.5 bytes) = 35 GB. Quantisation dramatically reduces this number, which is why QLoRA exists.

Optimizer State and Gradient Memory

Adam optimizer states add 2 copies of the trained parameters at FP32. For full fine-tuning of a 70B model: 70B parameters x 4 bytes x 2 states = 560 GB, more than the weights themselves. QLoRA and LoRA drastically reduce this because only the adapter layers are trained, not the base model. A 70B model with LoRA rank=16 trains approximately 320M parameters, dropping optimizer state to 2.5 GB.

Activation Memory and Gradient Checkpointing

Activation memory scales with batch size and sequence length. At batch=1 and 2K tokens, activations add roughly 2 to 5 GB for a 70B model during the backward pass. Gradient checkpointing trades compute for memory by recomputing activations during the backward pass rather than storing them. Enable it in Axolotl with gradient_checkpointing: true. This typically adds 10 to 20% training time in exchange for 30 to 60% less activation memory.

A100 vs H100 for LLM Fine-Tuning: Cost and Time Comparison

The A100 80GB and H100 SXM are the two workhorses for LLM fine-tuning. The H100 is not simply a faster A100: the architecture change (Hopper vs Ampere) introduces FP8 Tensor Cores, improved attention kernels, and higher memory bandwidth, all of which change the cost-per-token-trained calculation.

H100 SXM trains 2.2 to 2.5x faster than A100 on QLoRA workloads due to higher memory bandwidth and FP8 Tensor Cores. Once pricing is published for H100 SXM on packet.ai (currently on the waitlist), the faster training time is expected to more than offset the higher per-hour rate for 70B QLoRA jobs. For A100 availability today, packet.ai A100 SXM at $1.43/hr is on-demand. Join the H100 waitlist at packet.ai H100.

QLoRA Setup with Axolotl: Configuration File and Key Parameters

Axolotl is the standard configuration-driven fine-tuning framework for 2026. It wraps Hugging Face Transformers and PEFT with a YAML config file interface that handles dataset loading, tokenisation, LoRA/QLoRA setup, Flash Attention, and checkpointing without custom training loops.

base_model: meta-llama/Meta-Llama-3.1-70B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_4bit: true
load_in_8bit: false
strict: false

adapter: qlora
lora_r: 64
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true

datasets:
  - path: your_dataset
    type: alpaca

dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./qlora-out

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

Key parameters to tune for your specific GPU and model size:

Fine-Tuning Cost by Model Size on packet.ai

Reference costs at packet.ai published GPU rates for a 10K example dataset at 1,024 average token length, 3 epochs. Only on-demand GPUs with published prices are shown.

The L40S 48GB at $0.92/hr is the best value GPU for 70B QLoRA when iteration speed is not the priority. At 20 to 30 hours it costs $18 to $28 per run. If iteration speed matters, H100 SXM is architecturally 2.2 to 2.5x faster and will be available via the packet.ai H100 waitlist.

Unsloth: 2x Faster QLoRA with 60% Less VRAM

Unsloth rewrites key QLoRA kernels in Triton to eliminate the dequantisation overhead that makes standard QLoRA 2 to 3x slower than full BF16 fine-tuning. The result: Unsloth QLoRA runs at approximately the same speed as standard LoRA at BF16 for small models, and 1.5 to 2x faster than standard QLoRA for large models, while using 30 to 60% less VRAM than PEFT QLoRA.

pip install unsloth

# Unsloth drop-in for standard Transformers fine-tuning
from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "meta-llama/Meta-Llama-3.1-70B-Instruct",
    max_seq_length = 2048,
    dtype = None,        # auto-detect
    load_in_4bit = True, # QLoRA mode
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

Unsloth is the default choice for RTX 4090 and L40S fine-tuning runs in 2026. On H100 and H200 where FP8 and Flash Attention 3 are available natively, the Unsloth advantage narrows somewhat, but it still provides meaningful VRAM savings that allow larger batch sizes or longer sequences on the same hardware.

Flash Attention 2 for Fine-Tuning: VRAM and Throughput Impact

Flash Attention 2 implements the attention mechanism using IO-aware tiling that avoids materialising the full attention matrix in VRAM. For fine-tuning, this has two effects: attention VRAM drops by 2 to 4x at sequence lengths above 2K tokens, and attention throughput increases by 15 to 25% due to better memory access patterns on modern GPUs.

Flash Attention 2 is supported natively on H100, H200, and A100 (for FP8 on H100/H200 see the FP8 vs FP16 vs BF16 guide) via PyTorch 2.0+ with the CUDA 12.1 backend. On RTX 4090 (sm_89 architecture), Flash Attention 2 is supported as of PyTorch 2.2. Enable via Axolotl (flash_attention: true) or directly in Transformers (attn_implementation="flash_attention_2").

Frequently asked questions

Last reviewed: 2026-08-06. Prices verified against packet.ai pricing page. Available on-demand today: A100 SXM ($1.43/hr), L40S ($0.92/hr), RTX 4090 ($0.39/hr), B200 ($3.75/hr Dynamic). H100 SXM and H200 SXM: coming soon, no published price. Join the waitlist at packet.ai H100.