The GPU utilization % on your monitoring dashboard measures whether any kernel is executing - not how efficiently the GPU is working. A GPU at 99% utilization can be performing at 10% of its theoretical throughput. Here's what to measure instead.
The GPU utilization percentage on your monitoring dashboard is not what you think it is. It measures whether any kernel is executing - not how efficiently the GPU is working. A GPU at 99% utilization can be performing at 10% of its theoretical throughput.
Key takeaways
The GPU-Util percentage in nvidia-smi (and most monitoring dashboards that pull from it) is defined as:
Percent of time over the past sample period during which one or more kernels was executing on the GPU.
This is a binary metric: kernel running or not running. It says nothing about how efficiently the kernel is using the GPU's compute units. A kernel that launches, does a small memory copy, and waits for data - running a thousand times per second - reports 99% utilization while the GPU's shader processors sit idle for most of each kernel execution.
This definition made more sense when GPU kernels were the primary execution model and the main concern was GPU idle time. For modern deep learning workloads, it is nearly useless as an efficiency metric.
For diagnosing GPU efficiency, the relevant metrics are:
SM Active Cycles (not SM Busy)
nvidia-smi reports SM (Streaming Multiprocessor) utilization, which like GPU utilization, measures "at least one warp active" - not whether the SM was doing useful work. The more precise metric is SM active cycles measured via the CUDA profiling API: the fraction of cycles where an SM had warps actively computing, not just scheduled.
Access via: nvml.nvmlDeviceGetSamples(handle, nvml.NVML_SM_UTIL, ...) or Nsight Systems' SM Warp Occupancy metric.
Memory Bandwidth Utilization
For memory-bound workloads (most LLM inference is memory-bound), the most important efficiency metric is what fraction of the GPU's memory bandwidth you are actually using.
A100 peak bandwidth: 2 TB/s. If your inference workload achieves 800 GB/s, you are at 40% memory bandwidth utilization. This is actually reasonable for many serving configurations - peak bandwidth is a theoretical ceiling that requires extremely careful kernel tuning to approach.
Access via: nvidia-smi dmon -s u (shows memory controller utilization) or Nsight Compute's Memory Throughput metric.
Achieved Occupancy
Occupancy is the ratio of active warps to the maximum number of warps the hardware can support simultaneously. Low occupancy means the GPU's execution units have empty slots that could be filled with more parallel work. For most deep learning kernels, target occupancy is above 60-70%.
Access via: Nsight Compute's Achieved Occupancy metric, or the occupancy calculator in CUDA toolkit.
FLOP Utilization (MFU - Model FLOP Utilization)
For training workloads, the gold standard efficiency metric is MFU: the ratio of observed FLOPS to peak theoretical FLOPS. PaLM achieved 46.2% MFU on TPUs. State-of-the-art LLM training runs on A100s typically land between 35-55% MFU. Below 30% usually indicates a significant inefficiency - data loading bottleneck, suboptimal batch size, or kernel performance issues.
LLM inference has a specific pathology that makes the GPU utilization metric especially misleading:
During autoregressive generation (token-by-token output), each forward pass processes one token at a time (batch size 1 for a single user). The entire forward pass - loading weights from VRAM, computing attention, projecting to logits - takes around 20-30ms on an A100 for a 70B model.
Of that 20-30ms, GPU compute is active for a fraction of the time. Most of the time, the GPU is loading model weights from HBM2 memory into the SM register files. For a memory-bandwidth-bound workload, this is normal - but nvidia-smi still reports the GPU as "utilized" during the memory load time because a kernel is technically executing.
The result: at batch size 1, an A100 serving LLM inference might report 85-95% GPU utilization while achieving only 15-25% of its theoretical token-per-second throughput (compared to what the same model achieves at batch size 32).
The fix: increase batch size. At larger batch sizes, multiple token streams are processed simultaneously, and the GPU's compute units stay busy while memory loads are in flight for other requests. This is why production LLM serving engines (vLLM, TGI, TensorRT-LLM) use continuous batching - to maintain high actual throughput even when individual request latency is not batch-size-constrained.
For LLM inference, measure tokens per second per dollar. This is the efficiency metric that matters: how much useful output per unit of compute cost.
# Measure throughput with vLLM
from vllm import LLM, SamplingParams
import time
llm = LLM(model="meta-llama/Llama-3.1-70b", tensor_parallel_size=1)
prompts = ["Hello, world"] * 32 # batch of 32
params = SamplingParams(max_tokens=100)
start = time.perf_counter()
outputs = llm.generate(prompts, params)
elapsed = time.perf_counter() - start
total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
print(f"Throughput: {total_tokens / elapsed:.0f} tokens/second")
Simultaneously, in another terminal:
# Watch memory bandwidth utilization
nvidia-smi dmon -s u -d 1 | awk '{print $1, $3, $5}'
# Columns: GPU ID, SM util, Memory util
The memory utilization column from nvidia-smi dmon is the memory controller utilization - how busy the GPU's memory interface is. This correlates much better with actual throughput for memory-bound LLM inference than the SM utilization column.
For training, use the palme-style MFU calculation:
# MFU for training
# Approximate FLOPS per token for a transformer: 6 * num_params
# (forward + backward = 6x parameters in total MACs)
num_params = 70e9 # 70B model
batch_tokens = batch_size * seq_len
flops_per_step = 6 * num_params * batch_tokens
step_time = 2.1 # seconds for one training step
observed_flops = flops_per_step / step_time
# A100 peak: 312 TFLOPS (TF32 with sparsity)
peak_flops = 312e12
mfu = observed_flops / peak_flops
print(f"MFU: {mfu:.1%}") # Target: 35-55% for good A100 training efficiency
Small batch size for inference: Covered above. Fix with continuous batching (vLLM, TGI) or by increasing request concurrency.
CPU-bound data loading for training: The GPU is "busy" waiting for the CPU to produce the next batch. Monitor CPU utilization alongside GPU. Fix with more data loader workers (num_workers in PyTorch DataLoader) or a dedicated data loading pipeline.
Gradient accumulation with tiny micro-batches: If you are accumulating gradients over many tiny micro-batches to simulate a larger batch size, each micro-batch forward pass may be too small to keep the GPU occupied. Increase micro-batch size if VRAM allows.
Unoptimised attention kernels: Standard PyTorch attention implementation is not memory-bandwidth-optimal. FlashAttention-2 or -3 typically achieves 2-3x higher memory bandwidth utilization for attention computation on A100 and H100. Install and enable it if you are not using it.
Wrong dtype: Running inference in FP32 instead of FP16 or BF16 halves your memory bandwidth efficiency (twice the bytes moved per value). Check your model's dtype with model.dtype.
If you are trying to choose between GPU SKUs based on utilization data from your current setup, the dashboard utilization percentage will not tell you whether you are using the GPU efficiently. A100 at 90% dashboard utilization is not necessarily closer to its limits than A100 at 50% dashboard utilization.
The better diagnostic: measure tokens/second (for inference) or MFU (for training), then check whether switching to a faster GPU (higher memory bandwidth, more SMs) improves throughput proportionally. If upgrading from A100 to H100 improves throughput by 50% when H100 has 2x the memory bandwidth, your workload is memory-bandwidth-bound and you are not at H100's compute ceiling.
packet.ai publishes GPU benchmarks for LLM inference across its fleet - A100, L40S, RTX 6000 Pro, and B200 - with measured tokens/second at different batch sizes and model sizes. These are more useful for hardware selection than utilization percentages.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
