HBM3e vs HBM2e: Why Memory Bandwidth Decides LLM Throughput

HBM3e delivers 4.8 TB/s of memory bandwidth on the NVIDIA H200, 140% more than the 2.0 TB/s of HBM2e on the A100, making memory bandwidth the primary driver behind H200 running Llama 2 70B inference 45% faster than H100 at identical batch sizes.

When engineers evaluate GPUs for LLM inference, the conversation usually anchors on TFLOPS, CUDA core count, or Transformer Engine generation. Those specs matter for training and for the prefill phase. For autoregressive token generation, the spec that governs throughput is memory bandwidth. Specifically, the generation of High Bandwidth Memory (HBM) on the die.

The shift from HBM2e to HBM3e is the defining architectural change separating A100-generation infrastructure from the H200 and B200 tier. This post explains the technical differences, shows the inference benchmark impact, and maps each HBM generation to the right workload.

What Is High Bandwidth Memory (HBM) and How Does It Work?

High Bandwidth Memory is a form of stacked DRAM co-packaged with the GPU die on a shared silicon interposer, connected through Through-Silicon Vias (TSVs). Unlike GDDR6 memory sitting on separate chips across a PCB trace, HBM stacks sit microns from the compute die. The interface is wider, latency is lower, and bandwidth per watt is substantially better.

For reference: GDDR6X on a consumer GPU achieves roughly 1.0 TB/s. HBM2e on the A100 SXM doubles that at 2.0 TB/s. HBM3e on the H200 SXM more than doubles HBM2e again at 4.8 TB/s. Each generation achieves this by increasing per-pin signalling speed, not by widening the interface bus. The 1,024-bit interface per stack is constant across HBM2e, HBM3, and HBM3e.

How HBM stacks are built

Each HBM stack is a vertical column of DRAM dies bonded through TSVs. A 12-high HBM3e stack contains 12 DRAM dies and communicates to the GPU via a 1,024-bit interface divided into 16 pseudo-channels. Multiple stacks are placed around the GPU die on the interposer. The H200 uses six HBM3e stacks for 141 GB of total capacity at 4.8 TB/s aggregate bandwidth. The B200, using a dual-die GPU design with more stacks, delivers 192 GB at 8.0 TB/s.

Why HBM outperforms GDDR6X for AI workloads

GDDR6X achieves roughly 1.0 TB/s by using a fast serial interface across a narrow parallel bus (384 bits on a high-end consumer GPU). HBM achieves 2 to 8x more bandwidth by placing memory stacks directly on the same interposer as the compute die and running a 1,024-bit interface per stack. The physical co-location eliminates PCB trace length and reduces operating voltage, which is why HBM3e runs at 1.1V versus GDDR6X's 1.35V while delivering far higher bandwidth per watt. For AI workloads that are bandwidth-limited during inference, this architecture difference is the deciding factor in GPU selection.

HBM3e vs HBM2e: Bandwidth, Capacity, and Efficiency by the Numbers

The table below compares HBM2e, HBM3, and HBM3e across the specifications that directly affect LLM inference performance.

The bandwidth gains come from faster per-pin signalling (9.6 Gb/s on HBM3e versus 3.2 Gb/s on HBM2e), not from widening the bus. HBM3e also supports 12-high die stacks, enabling higher capacity per stack (up to 36 GB) compared to the 8-high stacks common in HBM2e. The interface protocol is backward compatible with HBM3 controllers, standardised under JEDEC JESD238.

HBM generation timeline: HBM2e through HBM4

HBM has gone through four generations in six years. Each generation has increased per-pin speed while keeping the same 1,024-bit interface, a design choice that allows GPU manufacturers to upgrade memory without redesigning the memory controller.

* HBM4 specifications per JEDEC JESD239 and early NVIDIA announcements. GB300 production throughput may differ at general availability.

Why LLM Decode Is Memory-Bound, Not Compute-Bound

LLM token generation is memory-bound because each token requires loading the full model weight matrix and KV cache from HBM before a small number of arithmetic operations, placing the GPU's compute cores idle while waiting for data from memory. Adding TFLOPS does not help; adding bandwidth does.

LLM inference runs in two phases with different bottlenecks. The prefill phase processes the full input prompt in one forward pass and is compute-bound: the GPU executes dense matrix multiplications and TFLOPS set the ceiling. The decode phase generates one token per forward pass and is memory-bound: every token requires loading the complete model weight matrix and accumulated KV cache from HBM before a small amount of arithmetic work.

How batch size shifts the memory-compute balance

At batch size 1 (one inference request), the GPU loads the entire model weight matrix to produce a single token. Arithmetic intensity (FLOPS per byte loaded from HBM) is as low as 1 to 2 FLOP per byte, far below the compute-to-bandwidth ratio of even the A100. At batch size 64, the same weights produce 64 tokens per load, raising effective arithmetic intensity 64x and beginning to stress compute. Memory bandwidth remains the first bottleneck for real-world serving workloads at batch sizes below roughly 64 to 128, covering most production inference deployments.

This is why upgrading from an A100 (HBM2e, 2.0 TB/s) to an H200 (HBM3e, 4.8 TB/s) improves inference throughput by 2 to 3x on 70B parameter models, while the A100-to-H100 step delivers a 1.5 to 2x improvement despite H100 adding substantially more Tensor Core compute through its Transformer Engine and native FP8 support.

HBM Memory Generations Across NVIDIA GPUs: A100 to B200

Each NVIDIA data centre GPU generation since Ampere has shipped with a different HBM generation. The bandwidth progression maps directly to inference throughput capability for large models.

The H100-to-H200 transition is a memory upgrade, not a compute upgrade. Both GPUs share the identical Hopper compute die, the same Tensor Core TFLOPS, and the same Transformer Engine. H200 upgrades the memory subsystem from HBM3 to HBM3e: capacity grows from 80 GB to 141 GB, and bandwidth increases from 3.35 TB/s to 4.8 TB/s. Every inference performance gain H200 delivers over H100 originates in the memory subsystem.

How HBM3e Bandwidth Translates to Real Token Throughput

The bandwidth improvement from HBM2e to HBM3e maps closely to inference throughput gains on large models where decode is the bottleneck. The table below uses measured throughput data from NVIDIA TensorRT-LLM benchmarks.

Source: NVIDIA TensorRT-LLM benchmarks. Llama 2 70B, FP8 precision, max throughput configuration.

The H200 SXM achieves 31,712 tokens per second on Llama 2 70B versus H100's 21,806 tokens per second, a 45% throughput gain with an identical compute die, confirming that HBM3e memory bandwidth is the operative variable.

On Llama 2 13B at single-GPU batch size, NVIDIA's TensorRT-LLM evaluation shows H200 reaching 11,819 tokens/sec, up to 1.9x faster than H100. The improvement is entirely attributable to H200's larger, faster HBM3e memory subsystem.

Teams running Llama 3 70B or Mistral-class models that previously required two H100 GPUs in tensor-parallel configuration often consolidate onto a single H200, reducing NVLink communication overhead and cutting infrastructure cost per served token.

GPU count required by model size and precision

Memory bandwidth determines token throughput. Total VRAM determines whether a model fits at all. The table below shows minimum GPU count to hold model weights, not accounting for KV cache and activation memory overhead during inference.

* H200 holds 70B FP16 weights (approx. 140 GB) within 141 GB VRAM, leaving minimal headroom for KV cache. FP8 is recommended for production 70B inference on H200 at long context lengths.

Estimated cost per million tokens by GPU and tier

Throughput per dollar matters more than throughput alone for inference workloads. The table below shows estimated cost per million tokens on Llama 2 70B, using measured tok/s figures and current or planned packet.ai pricing.

Cost per 1M tokens calculated from tok/s x 3,600 = tokens per GPU-hour, divided into the hourly rate. H100 and H200 prices are planned packet.ai roadmap pricing; verify against the live pricing page before deployment planning. B200 Llama 2 70B throughput not yet independently verified at time of writing.

Choosing the Right GPU for Memory-Bandwidth-Limited Inference

The correct HBM generation depends on model size, context length, and batch size. The guide below maps common inference scenarios to the right GPU tier.

For the highest-throughput inference workloads in 2026, including 405B parameter models, long-context agentic pipelines, and high-concurrency multi-user serving, the B200 SXM is the correct tier. At those scales, both memory bandwidth (8.0 TB/s) and capacity (192 GB) become binding constraints simultaneously, and HBM3e addresses both.

Available now on packet.ai: B200 SXM clusters from $3.75/hr per GPU (Dynamic) or $5.90/hr (Dedicated), no minimum commitment. H200 SXM and H100 SXM are joining the platform shortly. Join the notify list to be first on either. For managed inference without GPU management overhead, packet.ai Token Factory is a per-token LLM inference API launching on HBM3e infrastructure.

HBM3e vs HBM2e: The Bottom Line

Frequently asked questions

Last reviewed: 2 August 2026. GPU availability and pricing verified against the live packet.ai pricing page. For current cluster availability, browse GPU clusters on packet.ai.