Multi-Node Training on GPU Clusters: When One Server Stops Being Enough
A single server's NVLink tops out at 8 GPUs. Here's what changes, technically and practically, once a training run needs InfiniBand-connected servers instead.
packet.ai Team
July 24, 2026
A single server's NVLink connects up to 8 GPUs at up to 1.8 TB/s. Once a model or dataset outgrows that, InfiniBand connects multiple servers together, and that transition, not simply a bigger GPU, is what multi-node training actually means.
Key takeaways
NVLink connects GPUs within one server. InfiniBand connects separate servers. They solve different problems and neither substitutes for the other.
Not every GPU supports NVLink. Consumer cards like the RTX 5090 use PCIe only; multi-node scaling requires cards like H100 SXM or B200 built for it.
A real multi-node cluster on packet.ai runs NVIDIA Quantum-2 InfiniBand at 400 Gb/s per node alongside 1.8 TB/s intra-node NVLink, scaling to 1,024+ GPUs.
Multi-node clusters take 2-6 weeks to deploy for InfiniBand cabling and validation, a real planning constraint that doesn't apply to single-node rentals.
packet.ai's own scheduler explicitly steps aside for cluster-scale training: dynamic placement is for single-GPU and small-fleet workloads, while multi-node training runs on dedicated GPUs with InfiniBand.
Most GPU sizing conversations end once a model fits on a single GPU. Multi-node training begins where that conversation ends: once your model or dataset no longer fits on one server, the question shifts from which GPU to how those GPUs talk to each other, and that's a networking problem as much as a compute one.
Multi-Node Training: What Changes When You Add a Second Server
A single node is one physical server, typically 8 GPUs, connected internally by NVLink. Everything about that setup, from software to hardware, assumes the GPUs share the same box. The moment a second server enters the picture, that assumption breaks: those 8 new GPUs aren't on the same NVLink fabric, and reaching them means going over a network instead of an internal interconnect.
This is the actual definition of multi-node training: coordinating GPUs across separate physical machines, not just running more GPUs. A server with 8 GPUs and a rack with 8 separate single-GPU servers are not the same thing, even if the total GPU count matches, because the interconnect between them is fundamentally different.
NVLink vs InfiniBand: Two Different Jobs
These two technologies get conflated constantly, and the confusion causes real planning mistakes. They solve different problems at different scales.
Layer
NVLink
InfiniBand
Connects
GPUs within one server
Separate servers
Typical bandwidth
Up to 1.8 TB/s (5th-gen)
400 Gb/s per node (NDR)
Scope
Single node only
Across the whole fabric
Crosses node boundary?
No
Yes, by design
NVLink does not cross the node boundary under any configuration. Once communication needs to leave one server and reach another, NVLink is irrelevant regardless of how fast it is inside the box, and InfiniBand (or RoCE Ethernet) takes over. This is also why not every GPU is a candidate for multi-node scaling in the first place: consumer cards like the RTX 5090 use PCIe only, with no NVLink support at all, while data-center cards like H100 SXM and B200 are built specifically with NVLink for tight intra-node coupling, which then extends outward via InfiniBand between nodes.
⚡ Note
"NVLink vs InfiniBand" is a common search, but they're not competing options. A real multi-node deployment uses both at once: NVLink inside each server, InfiniBand between them. Asking which one to choose is like asking whether a building needs hallways or elevators.
When One Server Stops Being Enough
A single 8-GPU node is genuinely sufficient for a large share of real training and fine-tuning work. Multi-node becomes necessary at a few specific, recognizable points, not as a vague "when things get big" threshold.
The model no longer fits, even sharded, across 8 GPUs. Frontier-scale models routinely need dozens to hundreds of GPUs just to hold the weights.
Training throughput on one node is the bottleneck, not GPU count. If a run would finish in a reasonable time with more parallel compute, more nodes solve that directly.
You need guaranteed, non-shared performance for a sustained run. packet.ai's own scheduler explicitly steps back here: dynamic, shared placement is for single-GPU and small-fleet workloads, and cluster-scale training runs on dedicated GPUs with InfiniBand instead.
The dataset or checkpoint size demands distributed storage. Multi-node training typically pairs with a parallel file system, since local NVMe on one node isn't shared across the fabric.
What a Real Multi-Node Cluster Looks Like
Rather than describing this abstractly, here's what an actual InfiniBand-connected cluster looks like on packet.ai.
The 2-6 week timeline is worth planning around specifically: InfiniBand cabling, topology validation, and storage provisioning take real time, unlike a single-node rental that's ready in minutes. A typical 64-node B200 cluster lands around $2.80-$3.20/GPU-hr on a 12-month term, roughly 30% below retail per-GPU rates, which is the economic case for committing to a cluster rather than renting many single nodes separately.
Data, Tensor, and Pipeline Parallelism: Which One You Actually Need
Once GPUs are networked together, the model still has to be split across them in a way that makes sense for its size and shape. Three strategies cover most real deployments.
Data parallelism. Every GPU holds a full copy of the model and trains on a different slice of data, syncing gradients after each step. Simple and effective until the model itself no longer fits on a single GPU.
Tensor parallelism. Individual layers are split across multiple GPUs, so no single GPU needs to hold a full layer's weights. Communication-heavy, which is exactly why it benefits most from NVLink's intra-node bandwidth.
Pipeline parallelism. Different layers of the model live on different GPUs or nodes, and data flows through them in sequence like a pipeline. Better suited to crossing node boundaries than tensor parallelism, since it needs less frequent, less bandwidth-intensive communication.
Most production training at scale combines all three: tensor parallelism within a node to exploit NVLink, pipeline parallelism across nodes where InfiniBand is the bottleneck, and data parallelism layered on top to use multiple copies of that combined setup. Which combination fits depends on model size, not a fixed rule, but the general pattern holds: keep the most communication-heavy splitting inside the node, and push the coarser splitting across nodes.
Decision Framework: Four Questions
1. Does your model fit on one 8-GPU node, even sharded? If yes, you likely don't need multi-node at all, and the added networking complexity buys you nothing.
2. Does your GPU choice actually support NVLink? Not every card does. Confirm before assuming multi-GPU scaling, single-node or multi-node, is even on the table.
3. Can your timeline absorb 2-6 weeks of cluster setup? A single node deploys in minutes. A real cluster doesn't, and that lead time needs to be part of the project plan, not a surprise.
4. Which parallelism strategy matches your model's shape? Tensor parallelism inside a node, pipeline parallelism across nodes, data parallelism layered on top. Picking the wrong split wastes the interconnect you're paying for.
Common Mistakes in Multi-Node GPU Training
Treating NVLink and InfiniBand as interchangeable. NVLink never crosses the node boundary. Once you're multi-node, InfiniBand or RoCE is doing that job, not NVLink, no matter how fast NVLink is inside the box.
Choosing a GPU without checking NVLink support. Consumer cards built for single-GPU inference often skip NVLink entirely. Confirm this before planning any multi-GPU architecture around a specific card.
Underestimating cluster deployment lead time. A single GPU rents in minutes. A validated multi-node InfiniBand fabric takes weeks. Plan the timeline around the slower number.
Using the wrong parallelism strategy for the interconnect available. Communication-heavy tensor parallelism across a slower inter-node link wastes the fabric. Match the split to what the interconnect can actually sustain.
Choosing Between One GPU and a Cluster
Once you know whether you need one GPU or a cluster, the next step is choosing the right deployment model.
1
One node is enough: Bare Metal
A full, dedicated 8-GPU node, no virtualization, no sharing, root access. Every GPU comes online together with no cluster coordination needed. See Bare Metal GPU Servers for the fixed-term, prepaid tier.
2
Multiple nodes needed: Clusters
InfiniBand-connected multi-node fabric scaling to 1,024+ GPUs, wholesale pricing around 30% below retail, with a named technical account manager for bring-up. See GPU Clusters for current available configurations.
Both currently-available options run on live silicon: B200 is the largest currently-bookable GPU for either path, with H100 and H200 both listed as coming soon on packet.ai rather than on-demand today. For a deeper look at how packet.ai's scheduling decides between shared and dedicated placement, the technology page covers that directly, including why cluster-scale training always runs on dedicated GPUs rather than shared scheduling. Current rates for every tier are on the pricing page.
If you're sizing the model itself before deciding on infrastructure, packet.ai's VRAM requirements guide covers how model size maps to GPU choice, the step that usually comes before deciding whether one node is enough.
Frequently asked questions
NVLink connects GPUs within a single server, at up to 1.8 TB/s on current hardware. InfiniBand connects separate servers together, typically at 400 Gb/s per node. NVLink never crosses the node boundary; a real multi-node setup uses both at once, NVLink inside each server and InfiniBand between them.
Only once training crosses more than one physical server. If your model and workload fit on a single 8-GPU node, InfiniBand doesn't come into play at all, since NVLink handles everything inside that node. InfiniBand becomes necessary specifically when you need multiple nodes to work together.
On packet.ai, multi-node clusters take 2-6 weeks for InfiniBand cabling, topology validation, and storage provisioning. This is a real planning constraint distinct from single-node rentals, which typically deploy in minutes.
Data parallelism runs full model copies on each GPU, training on different data slices. Tensor parallelism splits individual layers across GPUs, useful when a layer itself doesn't fit on one card, and works best over fast NVLink. Pipeline parallelism splits different layers across different GPUs or nodes in sequence, and tolerates slower inter-node links better than tensor parallelism does.
Technically yes over Ethernet, but it defeats the purpose. Consumer cards like the RTX 5090 have no NVLink support, so they already lack the fast intra-node coupling that makes single-node multi-GPU training work well. Adding a second machine over a standard network on top of that compounds the bottleneck rather than solving it. Multi-node scaling is built around data-center cards with NVLink and InfiniBand specifically because both layers of interconnect need to be fast for it to pay off.
On packet.ai, InfiniBand-connected clusters scale up to 1,024+ GPUs in a single coherent fabric, with larger deployments available on request. A typical 64-node B200 cluster runs around $2.80-$3.20/GPU-hr on a 12-month term.
Last reviewed: July 24, 2026. Technical specifications for NVLink, InfiniBand, and parallelism strategies cross-referenced across multiple independent sources. Cluster pricing, specs, and GPU availability confirmed directly against packet.ai's live pricing and product pages.
