Most teams rent a single GPU and call it a cluster. Real clusters have two layers of interconnect, three parallelism strategies, and one rule that breaks most training jobs: never cross the node boundary with tensor parallelism.
GPU clusters range from a single GPU to 100,000-GPU InfiniBand fabrics costing millions per month. They are networks of compute nodes, each containing one or more GPUs, connected by high-speed fabric so they act as a single pool of accelerated compute. The architecture question is not whether you need a cluster, it is which tier fits your model size and budget.
Key takeaways
The phrase "gpu clusters" covers a lot of ground. An 8xH100 node running vLLM to serve a 70B model is a cluster. So is a 1,024-GPU InfiniBand fabric pre-training a foundation model from scratch. The architecture question is not "do I need a cluster", it is "what kind of cluster, and how big". The answer is determined almost entirely by three numbers: the size of your model in bytes, the throughput target you need to hit, and whether your workload is memory-bound or compute-bound.
This post covers the full stack: what a gpu cluster is at each scale tier, how intra-node and inter-node networking actually works, which parallelism strategy fits which cluster shape, and how to pick the right hardware for the workload. For training specifically, including NCCL tuning and fault-tolerance setup, see the companion post on multi-node GPU cluster training. To browse available cluster configurations directly, see packet.ai cluster options.
A gpu cluster is any group of compute nodes, physical servers or virtual machines, each equipped with one or more GPUs, networked together so workloads can distribute across them. The networking layer is what separates a cluster from a bag of independent GPUs: without high-bandwidth, low-latency interconnect, GPUs cannot exchange gradients or activations fast enough to work cooperatively on a single job.
In practice, GPU clusters in 2026 fall into four distinct scale tiers, each with different interconnect requirements, parallelism strategies, and cost profiles:
The single-node 8-GPU server is the most practical starting point for most teams. An 8xH200 SXM node on packet.ai gives you 1,128 GB of aggregate HBM3e at 38.4 TB/s aggregate bandwidth, controlled by a single NVSwitch fabric, enough to run Llama 4 Maverick at full FP16 precision with full tensor parallelism, no InfiniBand required.
Every GPU cluster, regardless of scale, is built from the same three layers: compute nodes, an interconnect fabric, and shared storage. The design decisions at each layer determine what workloads the cluster can run and at what cost.
Compute layer
Each node is a server with 1 to 8 GPUs. On HGX (SXM) form-factor servers, H100, H200, B200, GPUs sit on a baseboard with direct NVLink connections via NVSwitch. PCIe servers connect GPUs through the CPU, with lower bandwidth.
Interconnect layer
Intra-node: NVLink at 900 GB/s (H100/H200) or 1.8 TB/s (B200) bidirectional. Inter-node: InfiniBand NDR at 400 Gb/s per port or RoCEv2 Ethernet. The two are not interchangeable, NVLink stops at the node boundary.
Storage layer
Local NVMe for checkpoints and hot datasets. Parallel file systems (Lustre, GPFS) or object storage (S3-compatible) for large training datasets. Storage bandwidth is the silent bottleneck, many clusters train fast but checkpoint slow.
The gpu cluster architecture decision that matters most in practice is the form factor of the compute nodes. HGX-based servers (H100 SXM, H200 SXM, B200 SXM) give you NVSwitch-enabled full all-to-all NVLink connectivity between all 8 GPUs on the node. PCIe servers trade that connectivity for easier rack deployment and lower per-node cost, acceptable for inference, problematic for training, where tensor parallelism all-reduce latency determines training step time.
"NVLink vs InfiniBand" is a common search, but the framing is wrong. They are not alternatives, they operate at different layers. A real production cluster uses both simultaneously: NVLink inside each server, InfiniBand between servers.
The bandwidth gap between NVLink and InfiniBand is roughly 18 to 1. A single H100 SXM's NVLink connection to its neighbors runs at 900 GB/s. The InfiniBand NDR link connecting that server to the next runs at 50 GB/s. This is why the node boundary is the most important architectural boundary in a GPU cluster, crossing it costs 18x in bandwidth, which directly translates to 18x longer all-reduce times for the same data volume.
For inference workloads, InfiniBand is rarely justified. KV cache transfers between serving nodes are less latency-critical than gradient all-reduces in training, and standard Ethernet (25 to 100 GbE) is sufficient for most disaggregated inference architectures. The economics flip for training: at training compute steps under 60 seconds, the ratio of communication time to compute time is high enough that InfiniBand NDR is mandatory for cluster efficiency.
The right cluster size is determined by one calculation: total model parameter memory in bytes at your target precision. A 70B parameter model at BF16 (2 bytes/parameter) requires 140 GB. An 8xH100 SXM node gives you 640 GB aggregate, 4.5x overhead for optimizer states, gradients, and activations during training. That fits. A 405B model at BF16 requires 810 GB. That requires multiple 8xH100 nodes, or a single 8xH200 node (1,128 GB aggregate).
For most AI training teams in 2026, the practical entry point is a single 8-GPU node. An 8xH100 SXM node handles full fine-tuning of 7B and 13B models, QLoRA on 70B, and inference serving for models up to Llama 4 Maverick (400B active parameters in MoE configuration). Moving to multi-node adds InfiniBand requirements, NCCL configuration complexity, and checkpoint fault-tolerance overhead. The jump is worth it when the model genuinely does not fit on one node, not before.
For the ai training cluster case that requires multiple nodes, the standard production configuration on packet.ai is 16 to 64 GPUs across 2 to 8 nodes, connected by NVIDIA Quantum-2 InfiniBand NDR at 400 Gb/s per node, with NCCL handling the collective communication layer across nodes.
Choosing between an h100 cluster, h200 cluster, or b200 cluster comes down to three factors: model VRAM requirement, throughput target, and hourly budget. The three GPUs represent three distinct price-performance tiers, not a simple upgrade ladder.
The H100 SXM and H200 SXM share the same GH100 compute die and identical FP8 TFLOPS. The H200 upgrade is entirely about memory: 141 GB HBM3e at 4.8 TB/s versus 80 GB HBM3 at 3.35 TB/s. For workloads that fit in 80 GB, performance is essentially identical. The H200 earns its cost only when 80 GB is the constraint, large models at full precision, long-context inference, or large-batch serving.
packet.ai H200 SXM cluster pricing is coming soon. For context, CoreWeave lists 8xH200 at $50.44/node/hr ($6.30/GPU/hr, verified July 2026) and AWS at $12.29/GPU/hr. Check the packet.ai H200 page for availability and pricing at launch.
The B200 SXM is the current performance ceiling for gpu clusters in 2026. At 192 GB HBM3e per GPU and 8 TB/s memory bandwidth, an 8-GPU B200 node delivers 1,536 GB aggregate VRAM at 64 TB/s aggregate bandwidth. The NVLink 5 fabric doubles intra-node bandwidth to 1.8 TB/s, reducing all-reduce bottlenecks on large models. At $3.75/hr per GPU on packet.ai ($47.2/hr for an 8-GPU node), B200 clusters suit teams running 100B+ models at high batch, or anyone for whom the H200 VRAM ceiling is already a constraint.
GPU cluster rental pricing has two models in 2026: on-demand (hourly, no commitment) and reserved (monthly or annual contract, 20 to 40% discount). Current rates for all GPU configurations are on the packet.ai pricing page. The right choice between the two models depends on whether your workload is bursty or sustained.
On-demand gpu cluster rental on packet.ai has no minimum commitment, no setup fee, and no egress charges. Clusters are provisioned in under 5 minutes and billed per hour. For experimental workloads, short training runs, or burst inference capacity, this is almost always the right pricing model. Reserved capacity makes sense when a cluster runs above 60% utilisation for a sustained period, at that point, a monthly commitment saves 20 to 30% over on-demand rates.
One cost item that catches teams off guard: egress fees. AWS charges $0.09 per GB for data leaving their network. A training run that generates 10 TB of checkpoints and moves them to a different storage system adds $900 in egress. packet.ai has no egress fees, data movement between cluster and storage is included in the GPU hourly rate.
For enterprise-scale GPU cluster procurement across multiple providers, GPUaaS.com aggregates dedicated H100, H200, B200, B300, and GB300 clusters from 20+ vetted providers globally, with flexible commitment terms and sourcing within 24 hours.
A GPU cluster only performs as well as its parallelism strategy. The wrong strategy wastes GPU cycles on communication overhead. The right strategy scales almost linearly with GPU count.
Data parallelism replicates the model on every GPU and splits the dataset. Each GPU processes a different batch; gradients are averaged across all GPUs at the end of each step via all-reduce. DDP is the right choice for any model that fits on a single GPU with room for optimizer states. It scales well across both NVLink (intra-node) and InfiniBand (inter-node) because the all-reduce happens once per step, not once per layer.
Tensor parallelism shards individual model layers across GPUs. The attention heads or FFN layers of a single transformer block are split across 2, 4, or 8 GPUs. TP requires all-reduce at every transformer layer, every forward pass, every backward pass. Because of this, TP must stay within a single node's NVLink domain. Tensor-parallel ranks crossing an InfiniBand link is the single most common cause of poor cluster utilisation in production.
Pipeline parallelism assigns different transformer layers to different GPUs or nodes, passing activations forward and gradients backward through a pipeline. PP tolerates the InfiniBand bandwidth gap better than TP because the communication is sequential (one pipeline stage at a time) rather than simultaneous. PP is the standard strategy for training models that genuinely require more nodes than a single tensor-parallel group.
The standard recipe for a 16-GPU cluster (2 nodes) training a 70B model: tensor parallelism degree 8 within each node (across the 8 GPUs via NVLink), pipeline parallelism degree 2 between nodes (across InfiniBand). Data parallelism is added when more than one replica of the full model fits across the cluster.
Last reviewed: July 29, 2026. Browse available GPU clusters on packet.ai. B200 SXM available from $3.75/hr per GPU. H100 SXM (from $2.50/hr) and H200 SXM (from $2.49/hr) coming soon. See H200 SXM, B200 SXM pricing, and H100 SXM pages for current availability.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
