PCIe lanes, bifurcation and risers for multi-GPU builds

How many PCIe lanes a multi-GPU inference box needs, how boards wire their slots, and which risers and cable adapters keep a Gen4 link clean.

Two GPUs fit on a good desktop board. Past two, the questions change: how many lanes the CPU has, how the board wires them to its slots, how a three-slot card reaches a slot it would otherwise cover, and whether a Gen4 link survives the trip. For llama.cpp inference the lane count matters less than most builders expect. Slot wiring, spacing and signal integrity matter more.

What a lane buys you

Wikipedia’s PCI Express table gives the usable rate per lane, per direction: about 0.985 GB/s at Gen3, 1.969 GB/s at Gen4 and 3.938 GB/s at Gen5. A x16 link at Gen4 moves about 31.5 GB/s; x8 at Gen4 is about 15.75 GB/s, the same as x16 at Gen3; x4 at Gen4, what an OCuLink adapter gives a card, is about 7.9 GB/s, and about 3.9 GB/s at Gen3.

The link runs at the lower of what the card and the slot support. The RTX 3090 is a PCIe 4.0 card, so a Gen5 slot gives it Gen4. On a used card, read the link generation and width under load as part of the used RTX 3090 checklist.

CPU lanes and slot wiring

The CPU sets the total; the board decides where the lanes go.

Consumer AM5. AMD lists the Ryzen 9 9950X with 28 native PCIe 5.0 lanes, 24 of them usable. Two GPUs at x8 each take 16 of them; the rest go to M.2 and USB, depending on the board. The ASRock X870E Taichi in the 48 GB build gives PCIE1, its first CPU-attached slot, all 16 lanes with one card, and runs PCIE1 and PCIE2 at x8/x8 with two (manual PDF page 38). Not every AM5 board does that cleanly: on the ASUS ProArt X870E-Creator, filling its second M.2 slot drops the second GPU slot to x4 (manual PDF page 18). Read the board manual’s lane-sharing table before you pick the board, not after.
Server EPYC. AMD’s EPYC 7002 datasheet gives the EPYC 7302P 128 PCIe 4.0 lanes. The ASRock ROMED8-2T in the 96 GB build wires all seven of its slots as PCIe 4.0 x16 from the CPU (manual PDF page 32), so four cards each get a full x16 link with slots to spare. Even there one slot shares: PCIE2 shares lanes with an M.2 slot, the OCuLink ports and four SATA ports, set by jumpers, so the 96 GB build keeps its GPUs off it. The Supermicro H12SSL-NT alternate has five x16 slots and two x8 slots (manual PDF page 10).

The Catalog’s motherboard table lists each board’s maximum GPU count and the lanes each GPU gets at that count, taken from the manuals.

Bifurcation

Bifurcation splits one physical x16 slot into two x8 links or four x4 links, each of which can feed a GPU through an adapter. The board’s BIOS has to offer it for that slot; Supermicro’s H12SSL manual covers the setting (PDF page 72). techpractice used it on a Supermicro H12SSL-I-O, with risers that “split x16 slots into 2x x8, allowing more GPUs without losing much performance”, and names “up to 7x with bifurcation” as that build’s upgrade path.

You need bifurcation only when you have more GPUs than slots. Four cards on the ROMED8-2T’s seven x16 slots need none. The SlimSAS host card below does need it: it turns one x16 slot into two x8 ports, so the slot must be set to x8x8.

Slot spacing and three-slot cards

The Founders Edition RTX 3090 is three slots wide. Seated directly, it covers the next two slot positions.

Two cards, four positions apart. Measured from the layout drawing in ASRock’s manual (PDF page 11), the Taichi’s two GPU slots sit about 80 mm apart, four positions. Two three-slot cards seat directly with one empty position, roughly 20 mm, of air between them. With three-position spacing they touch: den.dev found on an X570 board that “if you have two 3-slot cards, they will literally be touching each other, with no gap for air”, and moved one card to a vertical mount on a riser. hav4ik went to an open frame for the same reason.
Single-spaced server boards. The ROMED8-2T’s seven slots are single-spaced (manual PDF page 14). Seat a three-slot card there and it buries two slots, so on a board like this every card goes on a riser to a frame. The Catalog records each board’s slot_spacing as single, double or mixed.

Riser types in the Catalog

Ribbon x16 risers. The Thermaltake TT Premium PCI-E 4.0 Extender 300mm and the Lian Li 600 mm riser carry a full x16 link and are both shielded and rated for PCIe 4.0. They are the 96 GB build’s choice: plain, passive, and the card keeps all 16 lanes. Their limit is reach: the ribbon has to run from the slot to wherever the card sits.
SlimSAS 8i, x8 per cable. A 10Gtek host card sits in a bifurcated x16 slot and breaks it out to two SFF-8654 8i ports; two 8i cables, each eight lanes and 50 cm, run to a 10Gtek slot adapter that the GPU plugs into. Together they carry x16, or one cable alone carries x8. The cost is more parts and more connectors in the signal path.
OCuLink, x4. The ADT-Link F9934 cable (25 cm, PCIe 4.0 x4) and the F9G-BK7 dock (a x16 slot fed by x4 over 50 cm) hang an extra card off an OCuLink port, such as the ROMED8-2T’s. On that board the OCuLink ports share lanes with PCIE2 and are off by default; turning them on means moving the PE8_SEL and PE16_SEL jumpers, which takes lanes from PCIE2 (manual PDF pages 9 and 33). The card gets four lanes, which is enough for layer-split inference (below), and OCuLink is not hot-pluggable.

Gen4 signal margin and the Gen3 fallback

Every connector and every centimetre of cable takes margin away from a Gen4 link, and a riser has less than a card seated in its slot. Use the shortest riser that reaches, keep ribbons flat and away from PSU cables, and don’t fold them tightly. The Lian Li riser is twice the Thermaltake’s length, so it has less margin again.

Know what failure looks like. A card that drops off the bus, fails to show up, or makes Linux log PCIe Bus Error lines is the sign. The kernel’s AER documentation describes those reports: correctable errors are logged as warnings, the rest as errors. nvidia-smi -q shows each card’s current link generation and width; read them under load, because the link can drop to a lower speed at idle.

The fix is to force the slot to Gen3 in the BIOS. The ROMED8-2T has a per-slot link-speed setting for each of its seven slots (manual PDF page 58). Gen3 halves the per-lane rate, which, for the reason below, costs layer-split inference very little.

Slot power through a riser

A x16 card may draw up to 75 W from its slot, 66 W of it at 12 V, per Wikipedia’s slot-power figures. With a plain ribbon riser that power still comes from the board, so from whichever PSU feeds the board. That matters in a dual-PSU build: a card whose 8-pin cables hang off the second PSU still takes its slot power from the first. The 10Gtek slot adapter has its own 6-pin power input for the slot; feed it from the same PSU as the card it carries. The level1techs thread on the ROMED8-2T with four 3090s is about exactly this back-feed risk; the PSU sizing guide covers the rules.

Why lane count matters little for llama.cpp

llama.cpp’s default split mode, layer, puts whole layers on each card and runs them in turn. Between cards, only the hidden state crosses: for Llama 3.3 70B that is 8,192 values per token, about 16 KB at 16-bit or 32 KB at 32-bit. At Gen3 x4, the slowest link above, 16 KB takes about 4 µs and 32 KB about 8 µs. For scale, jonready measured 63 tokens/s single-stream for a 27B model on two 3090s under vLLM, about 16 ms a token, and a 70B model is slower than that. Prompt processing moves more at once, but still little: an 8,192-token prompt is about 134 MB of hidden state per card boundary at 16-bit, about 8.5 ms at Gen4 x8 and about 34 ms at Gen3 x4. At 32-bit it is about 268 MB: about 17 ms at Gen4 x8 and about 68 ms at Gen3 x4.

Where lanes do show:

Loading the model. The weights cross the link once. 42.52 GB, the 70B Q4_K_M file, takes at least about 1.4 s at Gen4 x16 and about 11 s at Gen3 x4, before any disk limit; the model storage guide works out the disk side.
Tensor parallelism. vLLM’s documentation says efficient tensor parallelism needs fast communication between GPUs, and suggests pipeline parallelism instead when the GPUs have no NVLink, so its tensor parallel mode leans on the link far more than llama.cpp’s layer split does. jbarrow enabled peer-to-peer transfers on his consumer cards with modified NVIDIA drivers and measured about 26 GB/s between GPUs. jonready’s vLLM figures were measured with NVLink-bridged cards at x16/x8; we have no vLLM figures for x4 links, so if vLLM serving is the goal, don’t plan on them.
Offloading. Layers that don’t fit in VRAM run from system RAM, and how much the link matters then depends on the runtime. We have no measurement for it; the system RAM guide covers what memory bandwidth sets instead.

So for a llama.cpp box, put the effort into slots that seat cleanly, risers that hold Gen4 or a deliberate Gen3, and power; a x4 link for a fifth card is a reasonable trade. For vLLM, count lanes.

Parts in this guide

Store links may earn us a commission at no cost to you. As an Amazon Associate we earn from qualifying purchases.

Part	Buy US	Buy DE	Buy UK
Thermaltake TT Premium PCI-E 4.0 Extender 300mm	Amazon US	Amazon DE	Amazon UK
Lian Li PCI-e 4.0 Riser Cable 600mm (PW-PCI-4-60X)	Amazon US	not available	not available

Sources

Last reviewed 2026-11-12