HDR InfiniBand Deployment Guide

→ Browse the GPU & AI Infrastructure collection

Building an HDR InfiniBand Fabric: A Cabling & Interconnect Deployment Guide

HDR InfiniBand — 200Gb/s per port — is the interconnect layer behind most NVIDIA DGX A100 and Mellanox ConnectX-6-based AI and HPC clusters still being deployed and expanded today. Getting the compute right is one part of the build. Getting the fabric right — the right switch tier, the right topology, the right cable or transceiver on every link — is what determines whether that compute actually gets fed at line rate.

This guide walks through the pieces: what HDR is, how the hardware fits together, how to cable it correctly, and where deployments typically go wrong.

What "HDR" Actually Means

InfiniBand speed generations are named, not numbered, and it's easy to lose track of which is which:

HDR runs on the QSFP56 form factor, using 4 lanes of 50Gb/s PAM4 modulation. It's the generation NVIDIA shipped as the standard fabric for DGX A100 systems and HGX A100-based clusters, and it remains a large, active install base even as NDR (Quantum-2, 400Gb/s) has become the default for new H100/H200-class builds.

The Three Hardware Pieces

• Host adapters — Mellanox/NVIDIA ConnectX-6 HDR-capable NICs, typically dual-port on DGX A100 nodes, giving each of the system's 8 GPUs its own dedicated 200Gb/s link, a 1:1 GPU-to-NIC ratio.
• Switches — Mellanox Quantum QM8700-series HDR switches, 40 ports of QSFP56 per switch, used at both the leaf and spine tiers.
• Cables and optics — DAC, AOC, or QSFP56 transceivers on MPO fiber, chosen per link based on distance and topology (covered in detail below, and in our dedicated SR4 transceiver vs. DAC comparison).

Topology: Fat-Tree and Rail Optimization

Almost every production HDR deployment uses a non-blocking fat-tree topology: leaf switches connect directly to compute nodes, spine switches connect leaf switches to each other, and the ratio of leaf-to-spine links is sized so there's no oversubscription — every node can talk to every other node at full line rate simultaneously.

Larger GPU clusters typically layer rail optimization on top of the fat tree. Because each GPU has its own dedicated NIC and its own link, the fabric is wired so that "GPU 0" on every node lands on the same leaf switch rail, "GPU 1" on every node lands on a different rail, and so on — aligning the physical cabling with how collective communication (allreduce, all-to-all) actually moves data between GPUs across nodes. Mis-wiring a rail — plugging a node's GPU-0 NIC into the wrong leaf switch — breaks that locality and forces extra switching hops, which shows up as a real, measurable hit to training throughput even though the cluster looks "up."

In a reference deployment like NVIDIA's DGX A100 SuperPOD, this typically means dedicated compute racks holding the DGX nodes, and separate, centralized fabric racks holding the leaf and spine switches, connected by structured cabling rather than one-off point-to-point runs.

Choosing Cable and Transceiver Types by Link

This is where most of the per-link decisions live, and it comes down to distance and how fixed the topology is:

• Node to leaf, same rack (under ~5m): passive DAC cable. Lowest cost, zero additional power draw, lowest latency.
• Leaf to spine, fixed adjacent-rack run (5–100m): AOC cable, a self-contained fiber link with the reach of optics and the plug-and-play simplicity of DAC.
• Any link into a centralized fabric row, or anywhere the physical topology may be repatched, extended, or reconfigured (5–100m): an SR4 transceiver — such as the MMA1T00-HS — on OM4 MPO structured fiber through a patch panel. The transceiver and the fiber plant are independent, so a bad module is a quick swap instead of a full cable replacement, and the fiber plant itself gets reused as the fabric grows.

We cover the reasoning behind that third option — and why it beats AOC even at identical reach — in the SR4 transceiver vs. DAC decision guide.

Structured Fiber Basics for Optics Deployments

If any part of your fabric uses transceivers on structured cabling rather than DAC/AOC, a few things matter more than they might seem to at first:

• Fiber type — HDR SR4 optics run on OM4 (100m) or OM3 (70m) multimode fiber, not single-mode. Confirm your existing fiber plant matches before assuming a transceiver will hit its rated reach.
• Connector — QSFP56 SR4 modules terminate in an MPO-12 UPC connector, using 8 of the 12 fiber positions (4 transmit, 4 receive).
• Polarity — Keep a single, consistent MPO polarity method (Type A, B, or C) across trunks, cassettes, and patch panels. Mismatched polarity is one of the most common causes of a link that won't come up even though every component tests fine individually.
• Slack and labeling — Fabric racks get re-patched more often than compute racks. Build in slack and label runs at both ends — the cost of doing this during initial installation is negligible compared to troubleshooting a mislabeled MPO trunk during a cluster expansion.

Common Deployment Pitfalls

• Breakout mismatches — confirm whether a switch port is configured for a single 200Gb/s link or split into a 4x breakout before ordering cables. The physical cable type differs.
• Rail misalignment — as noted above, cabling that doesn't preserve GPU-to-rail locality silently costs throughput without producing an obvious fault.
• "Unsupported optics" errors — Mellanox/NVIDIA switches and NICs check EEPROM signatures on plug-in. A transceiver that isn't coded for your specific platform can throw errors or run in a degraded/unsupported state even if it's electrically identical to an OEM module. This is a coding issue, not a hardware defect — buy from a vendor that pre-programs and compatibility-tests against your exact platform.
• Assuming AOC and DAC are interchangeable by "just checking the length" — DAC's practical ceiling is far shorter than AOC's, and swapping one for the other at the last minute because a run turned out longer than planned is a common source of order delays.
• No spares plan for fixed-length cables — a bad AOC or DAC cable at 40 meters means holding a 40-meter spare, or waiting on one. This is another point in favor of structured fiber with pluggable optics for larger fabrics — spares are just transceivers, not length-specific cable assemblies.

Compatible Optics and Cables in an HDR Fabric

Every DAC, AOC, and transceiver in an HDR fabric needs to present the correct EEPROM signature to Mellanox/NVIDIA ConnectX-6 adapters and Quantum switches — that's true whether you buy OEM or compatible. A properly coded compatible module is read as a native part with no CLI overrides or suppressed warnings, typically at 60–90% less than OEM pricing, with the same optical specs (wavelength, reach, power draw) because the components are built to the same MSA and IEEE standards. The things worth verifying before ordering: EEPROM coding matched to your exact switch/NIC platform, DDM/DOM support so you can monitor link health in-band, and a warranty that covers the part if it fails.

Frequently Asked Questions

Is HDR InfiniBand still worth deploying, or should I go straight to NDR?
That depends on whether you're expanding an existing HDR (Quantum QM8700 / ConnectX-6) fleet or starting fresh. Expanding an existing HDR fabric with matched HDR gear is almost always simpler and cheaper than mixing generations. New builds without existing HDR infrastructure are more often NDR (Quantum-2) or newer today.

Can HDR and NDR coexist in the same fabric?
Not on the same link — a port and its cable/optic need to match the generation on both ends. Mixed-generation fabrics are typically segmented, with gateways or separate fabric tiers bridging them, not a single flat fabric running two speeds.

What's the practical difference between a leaf switch and a spine switch in an HDR fat tree?
Leaf switches connect directly to compute node NICs. Spine switches only connect to leaf switches, aggregating traffic between racks/pods. Both tiers commonly use the same physical switch hardware (e.g., Quantum QM8700); the difference is purely in how they're wired.

Do I need single-mode fiber for HDR InfiniBand?
No — standard HDR SR4 optics are multimode (OM3/OM4), reaching 70–100m. Single-mode with longer-reach optics is only necessary for runs beyond that, which is uncommon within a single data center fabric.

The Resilient Tec Guarantee

• ✅ Deep expertise in Mellanox/NVIDIA interconnects — we've wired GPU clusters before
• ✅ EEPROM-coded cables and transceivers — no compatibility errors that delay your cluster bring-up
• ✅ Lifetime warranty on all cables and transceivers
• ✅ Talk to a real engineer — not a chatbot — who knows DGX, HGX, and HDR/NDR InfiniBand

Planning or expanding an HDR InfiniBand fabric? Contact Resilient Tec — we'll review your topology and confirm every DAC, AOC, and transceiver is right for your platform before it ships.

See also our GPU & AI Infrastructure Guide, DAC Cables Guide, AOC Cables Guide, and Optical Transceivers Guide.