Scaling past a single 8-GPU DGX H100 or H200 node means moving beyond the 200GbE ConnectX-7 links used on smaller systems like the DGX Spark. DGX H100/H200 nodes use ConnectX-7 adapters running NDR InfiniBand at up to 400Gb/s per port, and connecting two nodes directly (before you introduce a Quantum-2 switch for larger fabrics) requires a 400G passive DAC rated for that platform.
"Note: This guide covers direct node-to-node NDR InfiniBand cabling for small two-node clusters. Larger deployments (3+ nodes) require a Quantum-2 InfiniBand switch rather than direct DAC connections."
The Cable You Need
The DGX H100/H200 cluster ports are OSFP, not QSFP-DD, so you need a straight 400Gb/s OSFP-to-OSFP passive-copper NDR InfiniBand DAC (NVIDIA's 400G OSFP LinkX line). Look for a single-port OSFP-to-OSFP passive DAC such as the MCP4Y10-N001 (1m) or MCP4Y10-N002 (2m). Choose the 1m version for adjacent-rack placement and the 2m version if the nodes sit in different rack positions. Longer passive copper runs increase insertion loss, so don't oversize the length beyond what your rack layout requires.
Plug one end into an OSFP port on the first DGX H100/H200 node and the other end into the matching OSFP port on the second node. No switch is required for a two-node direct connection. Avoid QSFP-DD DACs (for example, the MCP1660-W-series) for this link — those are 400GbE Ethernet cables for Spectrum switches and won't carry NDR InfiniBand on the DGX's OSFP ports. Confirm the exact SKU against NVIDIA's official LinkX OSFP parts list, as cable support is governed by ConnectX-7 firmware.
What Software Do You Need?
After cabling, configure the InfiniBand fabric using NVIDIA's OFED/MLNX_OFED drivers, then verify link state with tools like ibstat and ibping. NCCL handles GPU-to-GPU collective communication across the link, and NVIDIA's official DGX H100/H200 clustering documentation walks through subnet manager configuration (opensm) for small direct-connect topologies.
Important Limitations to Know
- Direct DAC connections without a switch are only practical for 2-node topologies.
- Confirm your specific DGX H100/H200 SKU ships with ConnectX-7 NDR (not HDR) ports before ordering.
- Verify current port and cable compatibility against NVIDIA's official DGX H100/H200 documentation, since specifications can vary by system revision.
Quick Checklist
- Two DGX H100 or H200 nodes with ConnectX-7 NDR ports
- One MCP4Y10-N001 or MCP4Y10-N002 400G OSFP-to-OSFP passive DAC
- MLNX_OFED drivers installed and subnet manager configured
- NCCL and your distributed training/inference framework of choice
Related reading: https://resilient-tec.com/blogs/news/how-to-connect-two-nvidia-dgx-spark-systems-the-cable-you-need-and-why
View the full NVIDIA & Mellanox Interconnect Compatibility Guide: https://resilient-tec.com/pages/nvidia-mellanox-interconnect-compatibility-guide