Bare metal vs virtual machine GPU performance: the benchmark numbers cloud providers quote are single-workload, ideal-condition results. Here is what the data actually shows across inference, training, and multi-node distributed workloads.
GPU passthrough on KVM delivers 98 to 100 percent of bare metal GPU performance on single-node workloads, but real-world virtualization overhead on vGPU-based shared clouds ranges from 4 to 25 percent depending on workload type, and NCCL all-reduce latency in heavily virtualized multi-node setups can balloon by 200 to 400 percent. The decision isn't about bare metal versus VMs in the abstract - it's about matching isolation model to workload profile.
Key takeaways
"Bare metal" and "virtual machine" are often used as if they're a single axis. They're not. There are at least four distinct isolation models in GPU cloud:
1. Full passthrough (PCIe/VFIO): The GPU is attached directly to a single VM. No sharing, no time-slicing. Overhead is driver and PCIe transaction costs: 0-2% on modern hardware. This is what packet.ai's Dedicated tier uses.
2. SR-IOV virtual functions: The GPU exposes multiple virtual functions, each assigned to a separate VM. Compute isolation is good; NVLink fabric and P2P transfers are not shared across VFs. Common in H100 SXM deployments.
3. NVIDIA vGPU (MIG + time-slicing): Multiple VMs share a physical GPU through NVIDIA's driver stack. MIG (Multi-Instance GPU) provides hard partitions; time-slicing provides soft partitions. Both introduce overhead that scales with contention.
4. Container-level isolation (no VM): The workload runs in a container on bare metal, with GPU device access via nvidia-container-toolkit. Zero hypervisor overhead; isolation is cgroup and namespace-based. Weakest security boundary.
The benchmark literature almost universally conflates these. A paper showing "2% VM overhead" is usually measuring passthrough. A blog post showing "20% overhead" is usually measuring time-sliced vGPU under contention. They're describing different things.
For single-GPU inference workloads - running a model that fits entirely in one card's VRAM - the performance difference between bare metal passthrough and a well-configured VM is small.
MLPerf Inference v4.1 results show A100 80GB performance is within 1-3% across bare metal and passthrough VM configurations for ResNet-50 and BERT-Large. The overhead shows up in:
Where you see more separation is p99 tail latency under load. A shared vGPU environment with 4 tenants time-sliced on the same A100 will show inference p99 latency 15-40% higher than dedicated at high QPS, because scheduling jitter compounds at the tail. The median might be within 5%; the 99th percentile is where production SLAs break.
The practical implication: for batch inference, offline scoring, or latency-insensitive workloads, a well-managed vGPU environment can be cost-efficient. For real-time serving with p99 SLAs, dedicated hardware is the only reliable answer.
Single-GPU training shows similar patterns to inference - passthrough overhead is small, vGPU contention overhead is workload-dependent.
The gap widens with memory-bandwidth-bound workloads. Training large language models is heavily bandwidth-bound (VRAM bandwidth, not FLOPS, is often the bottleneck). vGPU time-slicing introduces context-switch overhead that disrupts the prefetch/compute pipeline. Results from IEEE papers on BERT-Large fine-tuning show:
These numbers assume a well-tuned vGPU environment. Poorly configured time-slicing (wrong scheduling policy, incorrect memory reservation) can push overhead above 30%.
This is where isolation model stops being a marginal consideration and becomes the dominant factor in whether a workload is feasible at all.
Distributed training across multiple nodes requires low-latency, high-bandwidth GPU-to-GPU communication. In bare metal NVLink/InfiniBand configurations, NCCL all-reduce bandwidth approaches hardware limits. In virtualized environments:
SR-IOV with InfiniBand passthrough: 18-35% higher all-reduce latency versus bare metal NVLink. This is from actual InfiniBand SR-IOV deployments at scale. The overhead comes from VF interrupt processing and IOMMU pressure on RDMA paths.
vGPU with virtualized networking: 200-400% higher all-reduce latency in congested environments. NCCL is extremely sensitive to jitter; a single slow step causes the entire collective to wait. In a shared environment where other tenants are generating I/O, the variance is unacceptable for large-scale training.
This is why distributed training at scale - 64+ GPUs, multi-node - is almost always run on bare metal or passthrough. Not because of average performance, but because training stability requires predictable step times. A 10% slower average with 2x variance in step time is worse than 20% slower average with 5% variance, because gradient staleness compounds.
packet.ai's Dynamic tier is not vGPU time-slicing. It's a scheduling model where workloads are co-located on the same physical infrastructure but assigned dedicated GPU resources for the duration of their job - no time-slicing, no VRAM sharing.
The isolation model is closer to SR-IOV passthrough than to vGPU: each tenant gets deterministic access to their assigned GPU capacity. The "dynamic" refers to the scheduler's ability to place workloads across the fleet based on utilization patterns, not to sharing GPU resources within a single job.
This matters for benchmarking: comparing packet.ai Dynamic to a cloud provider's "shared GPU" offering is comparing different isolation models. The relevant comparison is dedicated passthrough versus dedicated passthrough, at different price points.
Based on the benchmark literature and production deployment patterns:
Single-GPU inference, latency-sensitive (p99 SLA < 100ms at high QPS): Dedicated bare metal or passthrough VM. vGPU tail latency is unpredictable under load.
Single-GPU inference, batch or async (latency-tolerant): Shared infrastructure with scheduler-enforced isolation (like Dynamic) is cost-efficient. 5-15% overhead on median latency is acceptable; you're optimizing for $/token.
Single-GPU training, model fits in VRAM: Dedicated or passthrough. The overhead of contended vGPU (12-28%) compounds over a multi-hour training run.
Multi-GPU, single-node (NVLink scale): Dedicated. NVLink performance is critical; vGPU and SR-IOV both add meaningful overhead to NVLink P2P transfers.
Multi-node (InfiniBand scale, 64+ GPUs): Bare metal only. No production-grade distributed training framework tolerates the jitter introduced by shared virtualization layers at this scale.
When a cloud provider claims "near bare metal performance" on their GPU VMs, ask:
The benchmark literature is clear: passthrough overhead is small and acceptable for most workloads. Contended vGPU overhead is significant and workload-dependent. Multi-node virtualization overhead is often prohibitive for serious distributed training.
The right question isn't "bare metal vs VM." It's "what isolation model does this provider actually use, and does it match what my workload needs?"
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
