ethtool tells you the interface is down. mlxlink tells you why. On NVIDIA ConnectX adapters, BlueField DPUs and Quantum switches, mlxlink reads link state, physical state, module diagnostics, error counters and FEC configuration straight from the firmware — which is where the answer usually is. This is the tool to reach for before you swap hardware.
Getting to a device handle
mlxlink ships with NVIDIA Firmware Tools (MFT). It addresses hardware through the mst device layer rather than through network interface names, so the first step is starting that layer:
mst start
mst statusmst status lists the devices it found. You are looking for a path in the form /dev/mst/mt4123_pciconf0 — the number identifies the silicon generation, and the suffix distinguishes multiple cards in the same host. Every mlxlink command takes that path with -d.
If you would rather not install the full MFT package, the open-source mstflint project ships mstlink, which takes the same flags and produces the same output. Everything below applies to either.
The base command
mlxlink -d /dev/mst/mt4123_pciconf0With no query flags you get the operational summary. The fields that matter:
- State — Active means the link is up and usable.
- Physical state — the PHY-layer state machine. This is the field that distinguishes "nothing is plugged in" from "a cable is present and the two ends cannot agree." A port stuck in polling has no partner; a port that reaches an auto-negotiation or training state and stays there has a partner it cannot come to terms with.
- Speed and Width — what actually negotiated, which is frequently not what you expected.
- FEC — the forward error correction mode in force.
- Loopback Mode — should read no loopback. A port left in loopback from earlier testing will look healthy and pass no traffic.
- Auto Negotiation — on or off, and whether that matches the far end.
Physical state plus speed and width usually narrows the fault before you run anything else.
Module diagnostics
mlxlink -d /dev/mst/mt4123_pciconf0 -m-m (or --show_module) reads the transceiver: vendor, part number, serial, revision, temperature, supply voltage, bias current, and transmit and receive power per lane.
This is the fastest way to answer "is light crossing this link." Transmit power present and receive power absent means the fault is in the path or at the far end, not in your module. One lane several dB below the other three on a parallel optic means a contaminated fiber position, not a failing module. The reasoning behind those readings is covered in our guide to reading DOM and working the loss budget.
It is also the fastest way to confirm what is physically installed when you are working remotely and cannot see the port.
Counters and BER
mlxlink -d /dev/mst/mt4123_pciconf0 -c-c (or --show_counters) gives physical layer counters and bit error rate information. Two distinctions matter here.
Raw BER versus effective BER. Raw BER is the error rate on the wire before forward error correction does its work. Effective BER is what survives correction. A high-speed PAM4 link is expected to have a non-trivial raw error rate — that is what FEC exists for. Effective errors are the ones that got through, and on a healthy link that number should be zero.
A link can be up and still be sick. If effective errors are accumulating, the link is passing traffic while corrupting some of it, and applications will see retransmits and stalls rather than a clean failure. This is the failure mode DOM cannot see, because the optical power readings look perfect.
Compare counters against the module and platform documentation rather than against a rule of thumb. What you are looking for is direction of travel: errors that climb while you watch, or that reset to zero after a clean and reseat, are telling you something specific.
FEC configuration
mlxlink -d /dev/mst/mt4123_pciconf0 --show_fecFEC mismatch is one of the most common causes of a link that will not train between equipment from different vendors, and one of the hardest to spot because both ends report healthy optics. --show_fec lists what the port supports and what is configured.
Run it on both ends and compare. If one side is running RS-FEC and the other is on a different mode or none at all, nothing about the fiber will fix it. Our post on FEC mismatch covers the negotiation behaviour in detail.
Signal integrity
mlxlink -d /dev/mst/mt4123_pciconf0 -e
mlxlink -d /dev/mst/mt4123_pciconf0 --show_serdes_tx-e (--show_eye) reports eye opening — the margin the receiver has to distinguish signal levels. --show_serdes_tx shows transmitter parameters. These are the tools for a link that trains but performs badly, and for comparing a suspect port against a known-good one in the same chassis.
On 16nm-generation silicon, add --advanced when displaying SLTP values or the output will be incomplete.
Other queries worth knowing
-
--show_device— general device information -
--show_external_phy— external PHY details where present -
--show_ber_monitor— BER monitoring, though this is not supported on host channel adapters -
--show_links— valid PCIe links, used with--port_type PCIE -
--json— machine-readable output, useful when you are collecting from many hosts
For PCIe-side investigation rather than the network port, add --port_type PCIE. That path also accepts depth, PCIe index and node arguments; without them the tool uses defaults of 0, 0 and 0, and --show_links maps BDF addresses to those coordinates.
Commands that change things
mlxlink can configure as well as query. Two flags are worth knowing and treating carefully:
mlxlink -d <device> -a <UP|DN|TG>
mlxlink -d <device> -s <speeds>-a (--port_state) brings a port up, down, or toggles it. -s (--speeds) sets supported speeds.
On ConnectX-6 and later, and on Quantum switches and later, speed settings require the lane count as part of the value — 50G_1X, 50G_2X, 100G_2X, 100G_4X, 200G_4X. A bare speed will not be accepted.
Five caveats that will save you an outage
- mlxlink is a diagnostic tool, not a monitoring agent. NVIDIA states plainly that it is not intended for continuous polling or background monitoring. Do not put it in a cron loop. Use proper telemetry for trending.
-
Socket-Direct cards need both handles. When you disable a port with
--port_state dnon a NIC connected through Socket-Direct, the port must be disabled on both mst devices representing that physical port. Doing one leaves you in a confusing half-state. -
Do not take down the host-to-unmanaged-switch link. Using
--port_state dnon the port connecting a host to an unmanaged switch is specifically warned against — you may not get it back the easy way. - ConnectX-7 loopback requires a fully down link. On ConnectX-7 and later, loopback configuration only applies when the link is fully down, not merely in a polling state.
- Output goes to two streams. mlxlink prints errors, warnings and notes to stderr. If you are redirecting output to a file and wondering where the warnings went, that is why — redirect both streams.
A triage sequence
-
mst start, thenmst statusto get the device path. - Base command. Read physical state, speed, width, loopback. If loopback is on, you have found it.
-
-m. Is light leaving? Is light arriving? Are all lanes even? -
--show_fecon both ends. Compare. -
-c. Are effective errors accumulating on a link that claims to be up? -
-eand--show_serdes_txagainst a known-good port in the same chassis, if the link trains but performs badly.
Five of those six steps cost nothing and take under a minute. Working them in order is consistently faster than swapping a module and hoping.
On a DGX Spark or GB10: is it the cable?
Most slow-Spark threads on the NVIDIA developer forums are not cable faults. Rule these out before replacing anything.
| What you see | Usually | Check |
|---|---|---|
| About 100–110 Gb/s on a clean 200G link | Only one of the port's two 100G halves in use | Drive both interfaces in parallel; give NCCL both RDMA devices |
| About 12 Gb/s on single-stream iperf3 | The Spark's ARM cores, not the link | Measure with ib_write_bw instead |
About 12 Gb/s on ib_write_bw too |
ConnectX-7 firmware power throttle | Update OS and firmware, then reboot |
| Two-node direct link up, but NCCL fails or SSH routes oddly | Port 1 cabled to port 2 | Cable port 1 to port 1 and port 2 to port 2 |
Effective errors climbing in mlxlink -c, or per-lane errors rising in ethtool -S, after a reseat |
Cable or cage | Move the cable to the other port; if the errors follow it, it is the cable |
If the errors follow the cable, replace it. The NJAAKK-N911 QSFP112 stacking cable is $174 at Resilient Tec and the MCP1650-V00AE30 200G QSFP56 DAC is $84, both new with a lifetime warranty. (Prices checked October 12, 2026.) For switched Spark clusters, see DGX Spark Stuck at 100 Gb/s on a Breakout?
Frequently asked questions
What is the difference between mlxlink and mstlink?
mlxlink ships in NVIDIA's MFT package. mstlink is the equivalent in the open-source mstflint project. They take the same flags and produce equivalent output, so use whichever is already available on the host.
Why does mlxlink need mst started?
It talks to the firmware through a character device rather than through the network stack, which is exactly why it can tell you things ethtool cannot. mst start creates those devices under /dev/mst.
My link is Active but applications are slow. What do I check?
Run -c and look at effective errors. A link can be up and correcting a high raw error rate, which shows up to applications as retransmits rather than as a link failure. Then check negotiated speed and width — a link that came up at half the expected width looks perfectly healthy in every summary view.
Does mlxlink work on switches as well as adapters?
Yes, on NVIDIA Quantum and Spectrum switch platforms as well as ConnectX adapters and BlueField DPUs. Some queries are hardware-dependent — BER monitoring, for instance, is not supported on host channel adapters.
Can I use it to test a link without a partner?
mlxlink supports physical test modes including PRBS generation, which lets you exercise a port and its cable without a working far end. Note that the port has to be configured for physical test mode first, or the tool will refuse.
Will mlxlink tell me if a third-party transceiver is being rejected?
Between the base command's physical state and the module page from -m, you can distinguish a module that is not being read at all from one that is read and reporting healthy optics while the link still refuses to train. That distinction is what you need in order to tell an optical problem from a compatibility one.
Related reading
- My optic won't link: reading DOM and working the optical loss budget
- The dirty connector problem: MPO inspection and cleaning
- Why a 25G or 100G link won't come up: FEC mismatch explained
- MPO-12/APC vs UPC: why your existing fiber won't work with NDR optics
- Is a third-party MMA4Z00-NS or NS400 safe? What to check before you buy
- DGX Spark stuck at 100 Gb/s (or 10) on a breakout? The fixes that get you to 200G