GPU Infrastructure Architecture Guide 2026: NVLink, SXM, Cooling & Data Center Design

1. NVLink vs InfiniBand — When Each Matters

NVLink and InfiniBand solve different problems at different scales. NVLink is a GPU-to-GPU interconnect that operates within a single server node. It provides a unified memory domain across GPUs so that tensor parallelism — splitting a single matrix multiplication across multiple GPUs — can proceed without bottlenecking on a narrow PCIe bus. NVLink 4.0 (Hopper) delivers 900 GB/s total bidirectional bandwidth per GPU. NVLink 5.0 (Blackwell) doubles that to 1,800 GB/s.

InfiniBand is a network fabric that connects servers to each other across racks and even across buildings. When your training job uses data parallelism (each GPU holds the full model, different data shards) or pipeline parallelism (different layers on different nodes), the gradient synchronization and activation transfers happen over InfiniBand. NDR InfiniBand delivers 400 Gb/s (50 GB/s) per port, and NDR400 pushes that to 800 Gb/s per port in late 2025 deployments.

The Jetstream Contrast: If you are fine-tuning a 7B-parameter model on a single 8xH100 server, InfiniBand is not required for single-node training — all communication stays on NVLink. If you are pre-training a 405B model across 512 GPUs spanning 64 nodes, InfiniBand NDR becomes your bottleneck, and the difference between NDR (400 Gb/s) and NDR400 (800 Gb/s) can shave 15-20% off your training time at scale.

Cloud vs colocation for interconnect: Cloud providers bundle InfiniBand into managed GPU clusters (e.g., AWS P5 instances use EFA, not InfiniBand). Colocation gives you direct control over fabric generation and topology. On-prem is unnecessary for most teams — colocation offers the same hardware control without the capital expenditure of building switch infrastructure. NVLink topology is not configurable in cloud instances per NVIDIA's HGX design — you get what the cloud provider deployed.

When evaluating colocation facilities on GYGO Place, confirm which InfiniBand generation the facility supports. Many colocation facilities still run HDR (200 Gb/s) fabric, which creates a 2x bandwidth gap versus NDR.

2. SXM vs PCIe Form Factors — Performance Difference & Use Cases

NVIDIA ships most data center GPUs in two form factors: SXM (Server eXtension Module) and PCIe. The SXM module solders directly onto a custom baseboard designed by NVIDIA (the HGX platform). This direct connection enables higher power delivery (700W for H100 SXM5 vs 350W for H100 PCIe), full NVLink connectivity (18 NVLink 4.0 links vs zero on PCIe), and superior memory bandwidth (3.35 TB/s vs 2.0 TB/s on H100).

PCIe GPUs plug into a standard PCIe Gen5 x16 slot, making them compatible with any server chassis that accepts full-height, full-length cards. They are simpler to deploy, cheaper to procure, and easier to replace. But the performance gap is real: SXM H100 delivers 30-40% more FP8 throughput than PCIe H100 in sustained training workloads because of the higher power envelope and memory bandwidth.

Decision framework: Choose SXM if you are building a dedicated training cluster with 8+ GPUs per node and need maximum throughput. Choose PCIe if you are deploying inference endpoints where each GPU operates independently, or if your facility cannot support the power and cooling demands of SXM modules. SXM is not recommended if your facility only supports air cooling — the 700W TDP per GPU requires liquid cooling or high-density rear-door heat exchangers (per NVIDIA DGX H100 installation guide).

Cloud vs colocation vs on-prem for form factor: Cloud instances abstract the SXM vs PCIe distinction entirely — you cannot choose form factor on AWS or GCP. Colocation lets you specify SXM-only nodes with NVLink, which is critical for training. On-prem SXM deployments are not necessary unless you run 24/7 training at scale exceeding 100 GPUs, where the amortization period justifies the $200K-300K per-node investment. PCIe servers don't require custom baseboards or NVSwitch ASICs, making them viable for smaller on-prem inference setups.

GYGO's GPU purchase marketplace sources both SXM and PCIe variants from partner resellers, priced on request.

3. GPU Cooling Infrastructure: Liquid vs Air, Power Density & PUE Impact

The shift from A100 (400W TDP) to H100 SXM (700W TDP) to GB200 (1,000W+ TDP per GPU module) has made cooling the defining constraint of modern GPU data centers. An 8xH100 DGX server draws 10.2 kW. Pack four of those into a rack and you need 40+ kW of cooling capacity — well beyond what traditional raised-floor air cooling can handle.

Air cooling moves heat from server heatsinks into the data center hot aisle using fans, then exhausts it through CRAC/CRAH units. It works reliably up to 25-30 kW per rack but becomes impractical beyond that threshold. Power Usage Effectiveness (PUE) for air-cooled GPU facilities typically ranges from 1.3 to 1.5, meaning 30-50% of total power goes to cooling infrastructure rather than compute.

Direct liquid cooling (DLC) runs coolant through cold plates mounted directly on GPU dies, removing heat at the source. It supports 80-120+ kW per rack, enabling much denser GPU deployments. PUE drops to 1.05-1.10 because liquid transfers heat 25x more efficiently than air. NVIDIA's GB200 NVL72 requires liquid cooling — there is no air-cooled option for the Blackwell NVL rack.

Configuration negation: Rear-door heat exchangers are not recommended if you plan to exceed 50 kW per rack — they hit their thermal ceiling and cannot scale to GB200 densities (per CoolIT Systems DLC specifications). Immersion cooling is unnecessary for standard H100 SXM deployments; direct-to-chip liquid cooling handles the thermal load at lower complexity. Air cooling doesn't require any plumbing retrofit, but it caps your deployment density at 25-30 kW per rack regardless of available floor space.

Cloud vs colocation vs on-prem for cooling: Cloud providers manage cooling transparently — you cannot choose cooling type on AWS or GCP. Colocation facilities vary widely: some offer DLC-ready racks while others only support air cooling. On-prem liquid cooling requires dedicated mechanical engineering staff and $500K+ in plumbing infrastructure, which is not justified unless you operate dozens of racks continuously.

The Jetstream Contrast: Air cooling is fine for inference-focused deployments with L40S or A100 PCIe cards at moderate density. But if you are deploying H100 SXM or GB200 training clusters, liquid cooling is not a luxury — it is a prerequisite. When browsing GYGO Place facilities, filter for DLC-ready sites if you plan to run anything above 30 kW per rack.

4. Power Density Requirements — kW per Rack for Different GPU Configs

Power density is the single most common reason GPU colocation deals fall through. A facility may have available rack space but lack the electrical capacity to power high-density GPU systems. Understanding your power requirements before contacting facilities saves weeks of back-and-forth.

A standard enterprise server rack draws 5-8 kW. A single NVIDIA DGX H100 system (8xH100 SXM + 2 CPUs + networking) draws 10.2 kW. A DGX SuperPOD with 32 DGX units needs a dedicated 400+ kW power feed. The GB200 NVL72, which packs 72 Blackwell GPUs into a single liquid-cooled rack, draws approximately 120 kW. These numbers exclude networking switches, storage, and cooling overhead.

Cloud vs colocation vs on-prem for power: Cloud pricing bundles power costs into hourly rates, making power density invisible but expensive at scale. Colocation exposes power as a direct cost — you pay per kW provisioned, so over-provisioning wastes money. On-prem facilities give maximum control over power contracts but require negotiating directly with utilities, which is not necessary for deployments under 1 MW total. Dual-feed utility power is not required for most GPU deployments — UPS with generator backup on a concurrently maintainable topology, which Uptime Institute calls Tier III, provides sufficient redundancy.

When calculating your total facility power requirement, multiply compute power by the facility's PUE. A 40 kW compute load in a facility with 1.3 PUE actually draws 52 kW from the grid. At $0.08/kWh, that difference adds $8,400/year in power costs per rack. This is why PUE matters, and why any honest colocation TCO estimate has to include it. Our colocation vs cloud breakdown walks through the rest of the cost stack.

5. GPU Memory Hierarchy — HBM3 vs HBM3e vs GDDR6X

GPU memory technology determines two critical metrics: how much model you can fit on a single GPU (capacity) and how fast you can feed data to the GPU's compute cores (bandwidth). These constraints directly affect which parallelism strategies you need, how many GPUs a workload requires, and ultimately your cost per token or cost per training run.

GDDR6X is the memory technology used in consumer and workstation GPUs like the RTX 4090. It provides 24 GB of capacity at 1.0 TB/s bandwidth. This is sufficient for inference on models up to 13B parameters (quantized) and fine-tuning models up to 7B with QLoRA, but it cannot fit larger models without aggressive quantization.

HBM3 (High Bandwidth Memory, third generation) is used in the H100 and MI300X. The H100 SXM5 provides 80 GB at 3.35 TB/s. The MI300X uses HBM3 stacks to deliver 192 GB at 5.3 TB/s — enough to fit a 70B model in BF16 on a single GPU without quantization. HBM3 achieves its bandwidth advantage by stacking memory dies vertically and connecting them to the GPU via a silicon interposer with thousands of parallel data lanes.

HBM3e is the enhanced version used in the H200 (141 GB at 4.8 TB/s) and GB200 (192 GB at 8.0 TB/s). The "e" denotes higher per-pin data rates, delivering a 43% bandwidth increase over HBM3 at the same stack count. For inference workloads where throughput is memory-bandwidth-bound (autoregressive token generation), the move from H100 (HBM3) to H200 (HBM3e) delivers a proportional throughput improvement without any code changes.

What you don't need: GDDR6X doesn't require HBM-class cooling or power infrastructure — RTX 4090 inference nodes run on standard air cooling at 450W TDP. HBM3e is not necessary for models under 13B parameters; at that scale, GDDR6X provides sufficient bandwidth and capacity (per NVIDIA TensorRT-LLM benchmarks). Upgrading from HBM3 to HBM3e is not recommended if your workload is compute-bound rather than memory-bandwidth-bound — pre-training large models benefits more from additional FLOPS than from extra memory bandwidth.

When renting GPUs on GYGO, memory capacity and bandwidth should drive your selection more than raw FLOPS. A memory-bandwidth-bound inference workload will see more benefit from upgrading H100 to H200 (43% bandwidth gain) than from adding more H100 GPUs.

6. Multi-GPU Topologies — NVSwitch, NVLink Bridge & PCIe Switching

How GPUs communicate within a server node determines the efficiency of multi-GPU parallelism. There are three dominant topologies in 2026 deployments, each with different bandwidth, latency, and cost characteristics.

NVSwitch mesh (SXM systems): NVIDIA's DGX and HGX platforms use NVSwitch ASICs to create an all-to-all mesh between 8 GPUs. Every GPU can communicate with every other GPU at full NVLink bandwidth simultaneously. The DGX H100 has four NVSwitch chips providing 900 GB/s per GPU in a fully non-blocking topology. This is the gold standard for tensor parallelism because no GPU-to-GPU path is slower than any other.

NVLink Bridge (consumer/workstation): The RTX 4090 and some workstation GPUs support NVLink Bridge, which creates a point-to-point link between exactly two GPUs. Bandwidth is 112.5 GB/s (NVLink 3.0 on RTX 4090). This helps for 2-GPU inference or training setups but does not scale beyond a pair. Most consumer motherboards support at most two NVLink-bridged GPUs.

PCIe switching: In PCIe-based multi-GPU servers, GPUs communicate through the PCIe root complex and CPU, adding latency and limiting bandwidth. PCIe Gen5 x16 delivers 64 GB/s per GPU, but actual GPU-to-GPU bandwidth through the CPU switch fabric is often 25-30 GB/s due to NUMA topology and contention. This is adequate for data parallelism (gradient AllReduce) where communication is periodic, but too slow for tensor parallelism where GPUs exchange activations continuously.

The Jetstream Contrast: An 8xH100 PCIe server costs roughly 40% less than a DGX H100. But without NVSwitch, tensor parallelism across all 8 GPUs is not feasible — each GPU must hold the entire model or rely on slow PCIe data paths. For training workloads, the DGX premium pays for itself in throughput. For inference farms where each GPU runs an independent model replica, PCIe servers deliver better cost-per-query.

Over-engineering warning: NVLink Bridge doesn't require NVSwitch ASICs or HGX baseboards — it is a simple physical connector between two adjacent GPUs. NVSwitch is not needed for 2-GPU setups (per NVIDIA NVLink Bridge documentation). PCIe multi-GPU topologies don't require any special interconnect hardware beyond a standard server motherboard with sufficient PCIe lanes. NVLink is not necessary for embarrassingly parallel inference workloads where each GPU serves independent requests with no inter-GPU communication.

Cloud vs colocation topology control: Cloud instances provide fixed topologies — you cannot customize NVSwitch configurations on AWS P5 or GCP A3. Colocation with owned hardware lets you choose NVSwitch mesh vs PCIe based on workload mix. On-prem is not justified for topology control alone; colocation provides the same flexibility without facility management overhead.

7. Data Center Tier Classifications — What They Mean for GPU Workloads

The Uptime Institute's tier classification system (Tier I-IV) defines facility topology: what can be maintained or fail without taking the load down. It assigns no availability percentage to a tier; Uptime Institute removed expected downtime per year from the standard in 2009. For GPU workloads — especially multi-day training runs that checkpoint infrequently — the tier classification of your colocation facility directly impacts your risk of lost compute.

Tier III (Concurrently Maintainable): In Uptime Institute's words, concurrently maintainable with redundant components and redundant distribution paths, so any part can be shut down for maintenance or replacement without impacting IT operation. Most GPU colocation quotes you will see are for Tier III class facilities. This is sufficient for most training workloads that implement checkpoint-restart.

Tier IV (Fault Tolerant): Adds fault tolerance to the Tier III topology, with several independent and physically isolated systems acting as redundant capacity components and distribution paths, so a single unplanned failure does not take the load down. Tier IV facilities cost more to build and lease (Uptime Institute publishes no premium figure) and are justified only for customer-facing inference services with strict SLA requirements or for organizations running training jobs that cannot tolerate interruption.

Configuration negation: Tier II facilities are not recommended for multi-day training runs — an unexpected failure still affects the load, which risks checkpoint loss. Tier IV fault tolerance is unnecessary for batch training workloads that implement checkpoint-restart every 15-30 minutes (per PyTorch distributed training best practices). Dedicated generator capacity is not required if the facility already provides UPS coverage with automatic transfer switches on a concurrently maintainable topology.

For most AI teams, a Tier III facility with strong checkpoint infrastructure is the economically rational choice. Tier IV’s higher cost is only justified if your business model requires sub-minute recovery times or if your compliance framework mandates fault-tolerant infrastructure. Tell GYGO Place the tier, power density, and cooling type you need.

Frequently Asked Questions