QLoRA fine-tuning a 70B model on a single A100 80GB costs $34–$51 and takes 24–36 hours. H100 at $2.50/hr cuts that to 8–12 hours and $20–$30. Here is the full cost math by method and model size.
Fine-tuning an LLM on a cloud GPU costs $3–$51 per run in 2026: a 7B QLoRA job on an A100 80GB at $1.43/hr takes 2–4 hours, while a 70B QLoRA on the same card takes 24–36 hours. Switching to an H100 at $2.50/hr cuts 70B training time to 8–12 hours, with lower total cost despite the higher hourly rate.
Key takeaways
Fine-tuning an LLM means adapting a pretrained model to a specific task using your own data. In 2026, parameter-efficient methods like LoRA and QLoRA have made this viable on a single cloud GPU for models up to 70B. This guide covers which GPU to pick, how much memory each method actually uses, what each run costs on packet.ai, and when upgrading from A100 to H100 reduces your total bill rather than increasing it.
For the full picture of which models fit which GPU at FP16 inference, the packet.ai VRAM requirements guide maps every major model to a minimum GPU and hourly rate.
Most teams reach for fine-tuning the LLM when they should be writing a better system prompt. The wrong choice wastes GPU hours and produces no measurable improvement. Fine-tuning is the correct answer for exactly three situations: you need behavioral consistency at scale (the model must always output a specific JSON schema), you need domain vocabulary the base model does not use correctly, or you are compressing a large model's capabilities into a smaller one for cost at inference time.
RAG is the correct answer when your problem is factual: the model does not know your product documentation, pricing, or case history. Fine-tuning does not reliably inject facts into weights. When the policy changes, you would need to retrain. RAG handles this without retraining.
Fine-tune when
Use RAG instead when
The 2026 production pattern that minimises both GPU cost and output quality risk: fine-tune a 7B-8B open-weight model on behavior and format, then add RAG retrieval for factual grounding at inference time. The fine-tuned 8B with RAG consistently outperforms a general-purpose 70B on narrow tasks while costing roughly 10x less to serve.
LoRA (Low-Rank Adaptation) freezes the base model weights and trains small low-rank adapter matrices. The adapter represents roughly 0.1–1% of total parameters, which means gradients and optimizer states apply only to the adapter, not the full model. QLoRA takes this further: the frozen base model is loaded in 4-bit NF4 quantization, reducing its memory footprint by ~4x, while the LoRA adapters are trained in BF16. The combination makes 70B fine-tuning viable on a single 80 GB GPU.
QLoRA on a 70B model with 4-bit quantization and gradient checkpointing uses approximately 41 GB of VRAM, confirmed by Unsloth benchmarks on Llama 3 70B. That is the difference between needing a $300K 8-GPU cluster for full fine-tuning and a single A100 80GB at $1.43/hr for a QLoRA run.
Note on gradient checkpointing
VRAM figures above assume gradient checkpointing enabled. Without it, add 20–40% to the QLoRA column. Always enable gradient checkpointing on A100 for 34B+ models; it reduces VRAM at the cost of ~15% extra training time, which is a worthwhile trade.
The A100 vs H100 decision for fine-tuning is a total cost calculation, not an hourly rate comparison. The H100 SXM is roughly 2–3x faster than the A100 SXM on transformer training workloads due to its Transformer Engine, FP8 support, and 3.35 TB/s HBM3 bandwidth versus the A100's 2.0 TB/s HBM2e. At $2.50/hr versus $1.43/hr, the H100 costs 75% more per hour. The crossover is at 8–10 hours of training time.
For Llama 3.1 70B QLoRA on a single GPU: the A100 80GB takes 24–36 hours ($34–$51 total at $1.43/hr). The H100 completes the same job in 8–12 hours ($20–$30 at $2.50/hr). The H100 is cheaper on total cost, faster to iterate, and frees the GPU sooner for the next experiment.
Total cost per fine-tuning run. QLoRA, packet.ai pricing July 2026
packet.ai: A100 80GB $1.43/hr, H100 SXM $2.50/hr. Times estimated with Unsloth enabled. Source: computecomparison.com, Spheron benchmarks July 2026.
On a single GPU, the A100 80GB at $1.43/hr on packet.ai beats the H100 on total cost only for short jobs: QLoRA on 7B–13B models that finish in under 6 hours. For 34B and 70B, or any multi-epoch run, the H100's speed advantage flips the economics. The full comparison of A100 and H100 architecture, bandwidth, and production workload benchmarks is in the packet.ai A100 vs H100 deep dive.
Llama 4 Scout (109B total parameters, 17B active per token as a Mixture-of-Experts model) loads all 109B weights at startup, not just the 17B active experts. QLoRA VRAM for Llama 4 Scout is approximately 55–65 GB, which fits an A100 80GB only with aggressive gradient checkpointing and a batch size of 1. Llama 4 Maverick (400B+ MoE) requires multi-GPU. For teams running Llama 4 fine-tuning, plan for H100 with at least 80 GB. See the Llama GPU sizing guide for the full parameter-to-VRAM breakdown across the Llama 3 and Llama 4 families.
Framework choice affects training speed as much as GPU choice on the same hardware. Unsloth rewrites CUDA kernels for attention and gradient computation: on an A100 80GB it delivers 2x the throughput of baseline Transformers/PEFT and reduces VRAM usage by up to 70%, confirmed by the Unsloth project benchmarks across Llama, Mistral, and Qwen models. For single-GPU QLoRA jobs on cloud A100 or H100, Unsloth is the default starting point in 2026.
Axolotl is the correct choice when you move beyond a single GPU. It handles FSDP2 and DeepSpeed ZeRO-3 via a YAML config file, making distributed fine-tuning on 2–8 H100s manageable without custom training loop code. For 70B models requiring multiple GPUs, or for teams running weekly fine-tuning pipelines in production, Axolotl on an H100 cluster on packet.ai is the standard configuration.
packet.ai A100 80GB dedicated starts at $1.43/hr. H100 SXM from $2.50/hr. No minimum commitment on either: spin up for a 4-hour QLoRA run and stop. The table below shows total job cost at packet.ai rates for the most common fine-tuning configurations in 2026.
Managed API costs based on Together AI ($0.30/1M tokens for 7B, $1.20/1M for 70B) and Fireworks ($0.50/1M) at 76.8M training tokens (50K samples x 512 tokens x 3 epochs). DIY on packet.ai GPU is 3–8x cheaper than managed fine-tuning APIs for equivalent dataset sizes, with no per-token billing ceiling.
The packet.ai dedicated GPU cloud gives single-tenant access to the full GPU with no noisy-neighbour interference during a training run and no scheduler preemption mid-epoch. For multi-day fine-tuning pipelines or weekly retraining schedules, dedicated pricing reduces total cost versus on-demand billing.
Last reviewed: 23 July 2026. Fine-tune on A100 80GB from $1.43/hr or H100 SXM from $2.50/hr on packet.ai, no minimum commitment, SSH-ready in under 5 minutes.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
