NVIDIA A100 vs H100 in 2026: Price, Performance and Which GPU Fits Your Workload

On packet.ai, the NVIDIA A100 80GB rents from $1.43/hr and the H100 SXM from $2.50/hr. Hyperstack vLLM benchmarks published in December 2025 show the H100 SXM5 generating 2.88x more aggregate LLM inference throughput than the A100 at high concurrency. For QLoRA fine-tuning under 30B parameters at current pricing, total job cost on an A100 and H100 falls within 10% of each other, making the H100 premium a workload-specific decision rather than a default upgrade.

The A100 vs H100 decision is the most consequential GPU choice for AI teams in 2026. Both GPUs are manufactured by NVIDIA, both carry 80GB of HBM memory, and both run CUDA 12.x workloads with the same PyTorch and Hugging Face training scripts without modification. The differences that determine which GPU is right for your workload are architectural: memory type, memory bandwidth, compute precision format, and the H100's dedicated Transformer Engine, a hardware addition that has no equivalent on the A100.

This guide works through the full specification comparison backed by NVIDIA's official datasheets, real-world LLM inference benchmarks from published provider data, total-cost math for QLoRA and full fine-tuning, and a workload-by-workload decision framework. It also covers how the decision changes when you factor in self-host LLM workloads on rented GPU infrastructure versus using a managed inference API. For broader GPU pricing context, see the packet.ai guide to GPU pricing models.

A100 vs H100 Architecture: What Separates NVIDIA Ampere from Hopper

The H100 outperforms the A100 on transformer-based LLM workloads primarily because of two architectural advances: the Transformer Engine with native FP8 mixed-precision, and HBM3 memory delivering 67% more bandwidth than the A100's HBM2e. The A100 uses 3rd-generation Tensor Cores limited to FP16 and BF16. The H100 uses 4th-generation Tensor Cores with a dedicated Transformer Engine that automatically selects between FP8 and FP16 precision per operation, delivering 3x to 4x higher throughput on models with 30B parameters or more where memory bandwidth is the primary bottleneck.

HBM2e vs HBM3: The Memory Bandwidth Gap That Determines Tokens Per Second

The A100 80GB uses HBM2e at 2.0 TB/s. The H100 SXM uses HBM3 at 3.35 TB/s. Both provide 80GB of total VRAM. Capacity is identical; bandwidth is not, and bandwidth is what determines inference speed.

LLM inference during the autoregressive decode phase is almost entirely memory-bandwidth-bound. For every output token, the GPU must read the entire model weight matrix from VRAM. On Llama 3 70B at 4-bit quantization, that weight matrix is approximately 35 to 40 GB. The HBM bandwidth determines how quickly those 35 to 40 GB of quantized weights can be read per token generated. This is why the H100 generates approximately 2.15x more tokens per second on Llama 3 70B at batch size 1, even though the raw TFLOPS gap between the two GPUs suggests a much larger theoretical advantage. Memory bandwidth is the actual constraint, not compute.

According to NVIDIA's official A100 and H100 specification sheets, the HBM3 to HBM2e bandwidth ratio is 1.675x. Published benchmark data from Spheron (May 2026) confirms that this bandwidth ratio maps directly to single-stream inference throughput: the H100 uses HBM3 at 3,350 GB/s and the A100 uses HBM2e at 2,039 GB/s, and that 1.64x bandwidth difference is the primary driver of the H100's inference speedup on memory-bound attention operations.

FP8 Transformer Engine: The Feature That Changes Production LLM Economics

The H100 Transformer Engine automatically selects between FP8 and FP16 precision per layer during training and inference. FP8 represents model weights and activations in 8-bit floating point rather than 16-bit, halving the memory footprint per activation. For large models running at FP8 instead of FP16, this allows larger batch sizes per GPU and reduces VRAM pressure on KV cache during long-context inference.

NVIDIA's H100 datasheet states the H100 delivers up to 1,979 TFLOPS at FP8 without sparsity. The A100's peak compute at TF32 with sparsity is 312 TFLOPS, and the A100 has no native FP8 support. vLLM supports FP8 inference on H100 via the --quantization fp8 flag. SGLang supports FP8 natively. TensorRT-LLM's H100 engine uses FP8 by default for transformer inference. The practical result at high concurrency on 30B and larger models: the H100 with FP8 delivers 3x to 4x the throughput of the A100 running the same model in FP16.

For workloads that do not use FP8 (QLoRA fine-tuning, small-model inference at low concurrency, development and experimentation), the Transformer Engine advantage disappears and the comparison narrows to memory bandwidth and hourly pricing.

NVLink 3.0 vs NVLink 4.0: When Inter-GPU Bandwidth Becomes the Bottleneck

The A100 SXM4 uses NVLink 3.0 at 600 GB/s bidirectional bandwidth per GPU. The H100 SXM5 uses NVLink 4.0 at 900 GB/s. This 50% inter-GPU bandwidth increase matters specifically for multi-GPU workloads with tensor parallelism, where the cluster must constantly exchange gradient tensors and KV cache activations between cards. For single-GPU workloads including QLoRA fine-tuning and single-card inference, NVLink generation makes no measurable difference. For 8x GPU clusters running pipeline-parallel or tensor-parallel inference on 70B and larger models, the H100's NVLink 4.0 fabric reduces the inter-GPU communication overhead that otherwise caps GPU utilization. PCIe variants of both GPUs operate at PCIe Gen4 bandwidth of 64 GB/s, which makes multi-GPU tensor-parallel workloads impractical; SXM form factor is required for any serious multi-GPU training or inference configuration.

NVIDIA A100 Price vs H100 Price Across Cloud Providers in 2026

The NVIDIA A100 price on packet.ai starts at $1.43/hr for a dedicated instance. The NVIDIA H100 price starts at $2.50/hr and is currently on waitlist. Across the broader GPU cloud market, the A100 80GB ranges from $1.09/hr to $5.07/hr depending on provider, contract type, and availability. The H100 SXM ranges from $2.01/hr to $11.06/hr for the same silicon. The 5x spread on A100 pricing and the 5x spread on H100 pricing for identical silicon is the clearest argument for evaluating specialized GPU cloud providers over hyperscalers.

packet.ai A100 dedicated pricing at $1.43/hr is 35% below Lambda's $2.21/hr (80GB) and 65% below AWS p4d at $4.10/hr for the same 80GB Ampere silicon with the same CUDA Toolkit support and SXM4 form factor.

LLM Inference Benchmark: H100 vs A100 Throughput and Cost Per Million Tokens

For LLM inference, the H100 outperforms the A100 across all model sizes and batch configurations tested in published benchmarks. The throughput advantage ranges from 1.5x on small models at batch size 1, to 2.8x on 70B models at batch size 1, to 3x or higher at batch size 16 with FP8 enabled on the H100. Despite this throughput lead, the cost-per-million-tokens advantage of the H100 is narrower than the throughput ratio suggests, because the H100 also costs 75% more per hour. At batch size 1, the H100 costs $2.48 per million output tokens versus $3.06 on the A100 for Llama 3 70B at packet.ai pricing, a 19% cost advantage.

Why LLM Decoding Is Memory-Bandwidth-Bound: The Physics of Tokens Per Second

LLM inference has two distinct phases. Prefill processes the full input prompt in a single forward pass and is compute-bound. Decode generates output tokens one at a time and is almost entirely memory-bandwidth-bound. During decode, for every output token the GPU must read the entire set of model weight matrices from VRAM to compute the next token probability distribution. At batch size 1, the GPU compute cores are idle most of the time waiting for VRAM reads to complete. This is why the H100's 67% memory bandwidth advantage maps to a roughly 2.15x throughput advantage on Llama 3 70B at batch size 1, rather than the 3x to 6x TFLOPS gap that the raw spec sheets imply. As batch size increases, VRAM reads become more efficient and the H100's compute advantage from FP8 begins to contribute, widening the throughput gap further at high concurrency. vLLM's PagedAttention and continuous batching make batch size management automatic for production inference deployments on both GPUs.

Published Benchmark Results: H100 vs A100 on Llama and Mistral Models

Three independent provider benchmarks confirm consistent throughput ratios. Hyperstack published vLLM benchmark results in December 2025 showing the H100 SXM5 at 3,311 aggregate tokens/second versus the A100 NVLink at 1,148 tokens/second for Llama-class models at high concurrency, a 2.88x advantage. OpenMetal published per-GPU figures showing H100 at 250 to 300 tokens/second and A100 at approximately 130 tokens/second for 70B models in single-GPU deployment. CUDO Compute's April 2026 benchmark using BERT inference as a controlled workload observed a consistent 2x throughput ratio, consistent with published data for Llama 3 70B with TensorRT-LLM optimization. MLPerf inference benchmarks show H100 outperforming A100 by approximately 2x to 3x depending on model size and precision format across multiple inference rounds.

Cost Per Million Output Tokens: H100 vs A100 at packet.ai Pricing

At batch size 1, the H100 ($2.48/M) is 19% cheaper per output token than the A100 ($3.06/M) for Llama 3 70B. At 13B models with batch size 1, the two GPUs reach near cost parity: H100 at $1.16/M vs A100 at $1.17/M. This near-parity at 13B is the clearest data point showing that the A100 matches the H100 on cost-per-token for smaller models, making the A100 the economically rational choice for serving Llama 3 8B, Mistral 7B, and DeepSeek R1 7B at low to moderate concurrency.

packet.ai pricing: A100 at $1.43/hr, H100 at $2.50/hr. Formula: ($/hr divided by tok/s) divided by 3600 multiplied by 1,000,000. Throughput figures are representative estimates based on published provider benchmarks; actual results vary by vLLM version, quantization, and batch configuration.

GPU for LLM Fine-Tuning: QLoRA, LoRA, and Full Fine-Tune Cost Math

For QLoRA fine-tuning on models up to 30B parameters, the A100 and H100 produce nearly identical total job costs at current packet.ai pricing. The H100 finishes faster but costs more per hour, and the two effects largely cancel out. For 70B QLoRA, the H100 saves approximately 12.6% on total cost. For full BF16 fine-tuning at 70B or larger, neither GPU is the right single-card answer: multi-GPU clusters are required, and the H200 SXM or B200 become more relevant options due to higher per-card memory capacity.

QLoRA on 7B to 30B Models: Why the A100 Often Costs Less Per Run

QLoRA (Quantized Low-Rank Adaptation) fine-tuning uses the Hugging Face bitsandbytes library to load base model weights in 4-bit NormalFloat (nf4) format, with LoRA adapter layers training in FP16 or BF16. This reduces VRAM requirements substantially compared to full BF16 fine-tuning. A 7B model with QLoRA requires approximately 6 to 8 GB VRAM for weights plus 2 to 4 GB for gradients and optimizer states. A 13B model requires roughly 10 to 15 GB in the same configuration, well within the headroom of either GPU's 80 GB.

For a 13B QLoRA run using PyTorch with Hugging Face PEFT and Axolotl on packet.ai: the A100 at $1.43/hr completes in 6 to 8 hours at a total cost of $8.58 to $11.44. The H100 at $2.50/hr finishes in 3.5 to 5 hours at $8.75 to $12.50. Total costs are within rounding error of each other. The A100 is available immediately on packet.ai with no waitlist; the H100 requires joining the waitlist. For teams running rapid iteration on 7B to 30B models, the A100 is the pragmatic choice today.

QLoRA on 70B Models: The Total-Cost Calculation Most Teams Get Wrong

A 70B model with 4-bit quantization requires approximately 35 to 40 GB VRAM for model weights plus 5 to 10 GB for LoRA adapter layers and optimizer states, fitting within the 80 GB HBM of both the A100 and H100. Thunder Compute's June 2026 analysis of Llama 3 70B QLoRA runs found that H100 speedups over A100 in QLoRA mode are typically in the 1.5x to 2.5x range, well short of Hopper's full potential, because QLoRA's quantization and dequantization steps add computational overhead that does not fully use the H100's FP8 compute advantage.

At packet.ai pricing of $1.43/hr for the A100 and $2.50/hr for the H100: a 36-hour 70B QLoRA run on the A100 costs $51.48. The same run on the H100 finishes in approximately 18 hours at $45.00. The H100 saves $6.48 per run, a 12.6% total-cost advantage. For teams running 50 or more QLoRA experiments per month, the H100's combined savings in cost and engineering time (faster wallclock results) become meaningful. For teams running fewer than 10 runs per month, the A100's immediate availability without a waitlist is the stronger argument.

Full BF16 Fine-Tuning at Scale: Multi-GPU Cluster Requirements

Full BF16 fine-tuning of a 70B model requires the full weight matrix, gradients, and Adam optimizer states in 16-bit precision. Memory requirement is approximately 140 GB for the base model and optimizer states alone, which exceeds the 80 GB capacity of both the A100 and H100. A minimum of 2x A100 or 2x H100 with tensor parallelism across NVLink is required. For 70B full fine-tuning in production, 8x A100 SXM4 or 4x H200 SXM (with 141 GB HBM3e each) are the standard configurations. packet.ai's GPU cluster options support multi-node A100 configurations with NVLink and InfiniBand interconnect for distributed training at this scale.

NVIDIA A100 80GB: right workloads and wrong workloads

NVIDIA H100 SXM: right workloads and wrong workloads

Self-Hosting LLM Workloads: How the A100 vs H100 Decision Changes at Scale

Teams that self-host LLM workloads on rented GPU infrastructure face a different version of this decision. The question shifts from raw throughput to cost-per-million-tokens at your actual request volume and concurrency level. The A100 wins at low concurrency for smaller models. The H100 wins at production scale for larger models. A managed inference API may be cheapest at low total monthly volume regardless of which GPU powers it.

The Token-Volume Crossover: When Renting a Dedicated GPU Beats Managed APIs

At single-stream inference (batch size 1) on Llama 3 70B, a dedicated A100 at $1.43/hr on packet.ai delivers approximately $3.06 per million output tokens. packet.ai's Token Factory charges $0.10 per million tokens for the same Llama 3.3 70B model. The crossover point where renting a dedicated A100 and running vLLM costs less than Token Factory is approximately 12 million output tokens per month.

Below 12M tokens/month: Token Factory at $0.10/M is cheaper and requires no GPU management. Above 12M tokens/month: a dedicated A100 at $1.43/hr running vLLM with continuous batching delivers lower cost per output token than Token Factory and all major managed inference APIs (Together AI at $0.88/M, Fireworks at $0.90/M, AWS Bedrock at $0.72/M for Llama 3.3 70B). For the H100 at $2.50/hr with higher throughput at production concurrency, the crossover to dedicated hosting is roughly 20 million tokens per month at batch size 16.

vLLM and SGLang on A100 vs H100: Framework Compatibility and Setup Notes

Both the A100 and H100 run vLLM, SGLang, Hugging Face TGI, and TensorRT-LLM using the same Docker container images without modification. CUDA 11.8 is the minimum for H100 SXM5 support; the A100 works from CUDA 11.0. For FP8 inference specifically, vLLM requires the --quantization fp8 flag and CUDA 12.x; this flag is silently ignored on A100, which falls back to FP16. SGLang's FP8 support requires H100 or newer. TGI FP8 support is available from version 2.0. The vllm/vllm-openai Docker image on Docker Hub works on both GPUs without changes; only the --quantization flag and model serving engine choice differ between A100 and H100 deployments.

Best GPU for AI in 2026: Workload Decision Framework

For most AI teams in 2026, the NVIDIA A100 80GB is the best GPU for AI experimentation, QLoRA fine-tuning of models up to 30B parameters, and inference on models up to 13B at low concurrency, at $1.43/hr with no waitlist on packet.ai. The H100 SXM is the best GPU for production LLM inference at scale on 30B to 70B models and for workloads requiring FP8 native precision, at $2.50/hr on packet.ai. Teams that need the highest available memory bandwidth today without a waitlist should consider the H200 SXM at $2.49/hr, which provides 141 GB HBM3e and 43% more bandwidth than the H100 and is available immediately.

For teams that need the highest memory bandwidth currently available on packet.ai without a waitlist, the H200 SXM starts at $2.49/hr with 141 GB HBM3e and 43% more bandwidth than the H100. For frontier model pre-training and Blackwell-generation inference, the B200 starts at $3.75/hr with 192 GB HBM3e and NVIDIA Blackwell architecture at Compute Capability 10.0.

Frequently asked questions

Last reviewed: 17 July 2026. Deploy an A100 80GB from $1.43/hr on packet.ai today, or join the H100 SXM waitlist. For the highest memory bandwidth available now without a waitlist, browse H200 SXM clusters on packet.ai.