Why Your 200G ConnectX-6 Tops Out Near 100G in a Dell R740xd or R640 (PCIe Gen3 Explained)

A 200G ConnectX-6 such as the MCX653105A-HDAT in a Dell PowerEdge R740xd or R640 will link at 200G and then move roughly 100 to 110 Gb/s. The card isn't faulty. Those servers have PCIe 3.0 slots, and a PCIe 3.0 x16 slot carries about 126 Gb/s each way before protocol overhead. A 200G port needs a PCIe 4.0 x16 slot, and a 400G NDR port needs PCIe 5.0. For a PCIe 3.0 server, the right buy is a 100G card in an x16 slot.

Below: the bandwidth math, what that means for each ConnectX card, how to check your slot, and what to buy for Gen3 and Gen4 servers.

The Bandwidth Math

• PCIe 3.0: 8 GT/s per lane with 128b/130b encoding, about 7.9 Gb/s per lane each way. x16 ≈ 126 Gb/s; x8 ≈ 63 Gb/s.
• PCIe 4.0: double that. x16 ≈ 252 Gb/s, enough for one 200G port.
• PCIe 5.0: double again. x16 ≈ 504 Gb/s, enough for one 400G NDR port.

Those are raw numbers. Packet headers and flow control take another 10 to 15 percent, so a PCIe 3.0 x16 slot delivers about 100 to 110 Gb/s of network traffic in practice.

What That Means Card by Card

• Single-port 100G (MCX515A-CCAT, MCX555A-ECAT, MCX653105A-ECAT): fills a PCIe 3.0 x16 slot. This is the best match for an R740xd or R640.
• Dual-port 100G on PCIe 3.0 (MCX516A-CCAT, MCX556A-ECAT): both ports share the same ~100 Gb/s, in any server. These are good for redundancy or two networks, not for doubling throughput.
• Dual-port 100G on PCIe 4.0 (MCX516A-CDAT, MCX623106AN-CDAT): in a Gen4 slot, both ports can run at the same time. In a Gen3 slot they behave like the PCIe 3.0 cards.
• 200G HDR (MCX653105A-HDAT, MCX653106A-HDAT): about half speed in a Gen3 slot. They need PCIe 4.0 x16 for 200G.
• ConnectX-7 (MCX75310AAS, MCX755106AS, MCX713106AS): these are PCIe 5.0 cards. They work in older slots at reduced throughput, but you're paying for bandwidth the server can't use.

Check the Slot Before You Blame the Card

1. Find the card: lspci | grep Mellanox
2. Read the negotiated link: sudo lspci -vv -s <bus:dev.fn> | grep -E "LnkCap|LnkSta". LnkSta at 8GT/s, Width x16 means PCIe 3.0 x16; Width x8 means half that.
3. Check the riser. On 2U servers some riser slots are physically x16 but electrically x8. Put the NIC in a true x16 slot. In a GPU server, that means planning slots around the GPU, so check your server's riser documentation.
4. Check the bracket. Full-height cards fit R740xd full-height riser slots; the 1U R640 needs low-profile brackets. Ask us and we'll ship the right bracket.

Port-side problems such as FEC, coding and the cable itself are a separate check: see the mlxlink field guide.

What to Buy

• PCIe 3.0 server (R740xd, R640, other Xeon Scalable 1st/2nd gen): one single-port 100G card per server. Ethernet: MCX515A-CCAT (refurbished, $169.95). InfiniBand or flexible: MCX555A-ECAT (refurbished, $149.95). HDR fabric: MCX653105A-ECAT (refurbished, $389.95).
• PCIe 4.0 server (R750, R7525, EPYC 7002/7003): 200G is now usable. HDR: MCX653105A-HDAT (refurbished, $619.95). Dual 100GbE at full rate: MCX516A-CDAT (refurbished, $329.95).
• PCIe 5.0 server: ConnectX-7 for 400G NDR: MCX75310AAS-NEAT (refurbished, $1,049).

Building out a GPU server? See our custom R740xd GPU builds and R640 configurations.

FAQ

Can a 200G ConnectX-6 reach 200 Gb/s in a Dell R740xd or R640?
No. Those servers use PCIe 3.0, and a PCIe 3.0 x16 slot carries about 126 Gb/s each way before protocol overhead, roughly 100 to 110 Gb/s in practice. A 200G card such as the MCX653105A-HDAT will link at 200G but move about half that. You need a PCIe 4.0 x16 slot for 200G.

Is a dual-port 100GbE card worth it in a PCIe 3.0 server?
For redundancy or two separate networks, yes. For more bandwidth, no: both ports share the same x16 slot, so the combined total stays near 100 Gb/s. A single-port card such as the MCX515A-CCAT (refurbished, $169.95) gets the same throughput for less.

What network card should I buy for a PCIe 3.0 server?
A 100G card: the MCX515A-CCAT (refurbished, $169.95) for Ethernet, the MCX555A-ECAT (refurbished, $149.95) if you may run InfiniBand, or the MCX653105A-ECAT (refurbished, $389.95) for an HDR100 InfiniBand fabric. Put it in an x16 slot.

Does the slot width matter as well as the generation?
Yes. A PCIe 3.0 x8 slot carries about 63 Gb/s each way, so even a single 100G port is cut to roughly half. Check the riser: on many 2U servers some slots are x8 electrically even when they are x16 physically.

How do I check what speed my card's PCIe link negotiated?
Run sudo lspci -vv -s <bus:dev.fn> and read the LnkSta line. 8GT/s is PCIe 3.0, 16GT/s is PCIe 4.0, and Width shows the lane count. A 200G card showing 8GT/s x16 is capped near 100 Gb/s.

When does the PCIe 4.0 MCX516A-CDAT make sense?
When the server has PCIe 4.0 slots, such as a Dell PowerEdge R750 or R7525, and you want both 100GbE ports running at once. In a PCIe 3.0 server the CDAT behaves like the CCAT. At Resilient Tec: MCX516A-CDAT (refurbished, $329.95).

Ready to Buy?

Cards sized to the slot you have. Genuine NVIDIA/Mellanox, lifetime warranty, shipping from Knoxville, Tennessee. Prices below checked October 22, 2026.

• MCX555A-ECAT — ConnectX-5 VPI, 1 × EDR / 100GbE — $149.95, refurbished, in stock at Resilient Tec; PCIe 3.0 x16
• MCX515A-CCAT — ConnectX-5 EN, 1 × 100GbE — $169.95, refurbished, in stock at Resilient Tec; PCIe 3.0 x16
• MCX653105A-ECAT — ConnectX-6 VPI, 1 × HDR100 / 100GbE — $389.95, refurbished, in stock at Resilient Tec; PCIe 3.0/4.0 x16
• MCX516A-CDAT — ConnectX-5 Ex, 2 × 100GbE — $329.95, refurbished, in stock at Resilient Tec; PCIe 4.0 x16
• MCX653105A-HDAT — ConnectX-6 VPI, 1 × HDR 200G — $619.95, refurbished, in stock at Resilient Tec; needs PCIe 4.0 x16 for 200G
• MCP1600-C001 — 100G QSFP28 passive DAC, 1 m — $45, new, in stock at Resilient Tec; the cable for a 100GbE link

You Might Also Need

• How to read a ConnectX part number — every MCX suffix decoded
• Custom Dell R740xd GPU servers — build options and real prices
• Dell PowerEdge R640 refurbished — configurations and prices
• 100G QSFP28 DACs for AI homelab clusters — the ConnectX cable guide
• mlxlink and mst field guide — reading a ConnectX link that won't come up

Sources

• NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards User Manual (ordering part numbers)