The 96 GB build

Verdict

For a builder who wants 96 GB of VRAM in one machine to run a 70B model at Q8_0, or at Q4 with its full 128k context, or a 120B-class MoE model such as gpt-oss-120b, all without offloading to system RAM. Four used RTX 3090s on an EPYC board give each card a full x16 PCIe 4.0 link. Power is the hard part: four cards rated at 350 W each, power-limited to 300 W, put up to about 1.57 kW on the wall when all four are busy at once, more than a US 15 A circuit should carry continuously, so this build needs two PSUs and either a lower power limit or a bigger circuit. It costs about twice the 48 GB Tier's GPU outlay (four used 3090s instead of two), plus a server platform, risers, an open frame and a second PSU.

Model fit

Parts list

Store links may earn us a commission at no cost to you. As an Amazon Associate we earn from qualifying purchases.

Basis

Compatibility warnings

• Every card goes on a riser. The ROMED8-2T’s seven slots are single-spaced (ASRock’s layout drawing, manual PDF page 14), and the Founders Edition RTX 3090 is three slots wide, so a card seated directly covers the next two slots. Here all four cards sit on the frame’s GPU rail, each on its own riser from its own slot. ASRock wires all seven slots as PCIe 4.0 x16 from the CPU (manual PDF page 32), so every card gets a full x16 link. Use PCIE1, PCIE3, PCIE5 and PCIE7: all four are x16 from the CPU, every other slot stays empty so the ribbons don’t stack on top of each other at the board, and PCIE2, which shares its lanes (below), stays free.
• No bifurcation needed. Four cards on seven x16 slots leaves three spare, so nothing has to be split. Bifurcation is how techpractice took a different route: on a Supermicro H12SSL-I-O, Maxcloudon risers “split x16 slots into 2x x8, allowing more GPUs without losing much performance”, and that build names “up to 7x with bifurcation” as its upgrade path. You’d only need that here past seven cards. On the H12SSL-NT alternate, which has five x16 slots and two x8 slots, four cards also run at x16 on plain risers; Supermicro’s manual covers its bifurcation settings (PDF page 72) if you later go past five.
• Riser generation and signal integrity. Both risers on the list are rated for PCIe 4.0, but a Gen4 link through 300 mm of ribbon has less margin than a card in the slot, and the 600 mm Lian Li alternate has less again. Measure the run from each slot to its place on the rail before ordering, use the 300 mm Thermaltake wherever it reaches, keep the ribbons flat and away from the PSU cables, and don’t fold them tightly. If a card drops off the bus, fails to show up, or Linux logs PCIe Bus Error lines (AER reports correctable errors as warnings and the rest as errors on the console), set that slot’s Link Speed to Gen3 in the BIOS; the ROMED8-2T has a per-slot setting for each of PCIE1 to PCIE7 (manual PDF page 58). nvidia-smi -q shows each card’s current link generation and width, so check all four after the first boot. Gen3 halves the per-lane bandwidth, which matters little for layer-split inference (see the FAQ). The PCIe lanes and risers guide covers risers and signal margin in more depth.
• ROMED8-2T slot 2 shares its lanes. PCIE2 shares lanes with the M2_1 M.2 slot, the OCU1 and OCU2 OCuLink ports and SATA_4_7, chosen by the PE8_SEL and PE16_SEL jumpers (manual PDF pages 9 and 33). In the default setting PCIE2 runs at x16 and M2_1, SATA_4_7 and both OCuLink ports are off. Leave the jumpers at default and put the SSD in M2_2, which doesn’t share lanes with any slot. If you want M2_1 too, the other setting drops PCIE2 to x8, so keep the GPUs off PCIE2.
• Power connectors. NVIDIA’s Founders Edition takes two 8-pin PCIe cables through the 12-pin adapter in the box, so four cards need eight 8-pin cables, four from each PSU. Run one dedicated cable to each 8-pin input. The AX1600i lists ten 8-pin PCIe connectors and the HX1200i and HX1500i five each. On top of the 24-pin and the board’s 8-pin and 4-pin CPU connectors, ASRock asks for a cable in the board’s 6-pin GFX_12V1 header “when 4 graphics cards are installed” (manual PDF page 38); that comes from the AX1600i too. A partner card with three 8-pin inputs would need six from the second PSU, one more than the HX1200i lists, so count connectors on the exact cards you buy.
• Two PSUs, and how they split. The AX1600i powers the board, CPU, drives and two cards. The HX1200i feeds only the 8-pin inputs of the other two cards; their slot power, up to 75 W each, still comes from the AX1600i through the board and risers. At 300 W per card that puts up to about 1000 W on the AX1600i (250 W system, 2 × 300 W, 2 × 75 W of slot power) and about 450 W on the HX1200i. This site’s PSU estimate, floor((rated W × 0.8 − 250 W) ÷ 300 W per power-limited GPU), gives the AX1600i three GPUs, which is 1280 W of headroom at 80 %, so 1000 W stays inside it. The HX1200i gets two GPUs from the same formula, 960 W at 80 %, well above its 450 W. Corsair rates the AX1600i for 1600 W “on 15A circuits of 115V and higher”, and lists the HX1200i for 100 to 240 V input at 1200 W with no lower rating stated for 115 V; it carries only about 450 W here anyway. The HX1500i alternate gives more headroom on the second unit, but Corsair rates it at only 1200 W on 100 to 115 V input. The Thermaltake dual 24-pin adapter makes the second PSU switch on and off with the first; the level1techs build syncs the same two-PSU split with an Add2PSU relay instead.
• Back-feed between the two PSUs. The level1techs thread is about exactly this risk. Each card draws part of its power through its slot, and with unpowered risers that slot power comes from the board, so from the AX1600i, even for the cards whose 8-pins hang off the HX1200i. One reply reports: “I have actually cooked a pair of 3090s with power flowing the wrong way through them when the system was “shut off””. The thread doesn’t settle how big the risk is, so treat it as real: keep all of a card’s 8-pin cables on one PSU, never power one PSU off or unplug it while the other is on, and if you use a powered riser, feed it from the same unit as its card. The quoted damage happened with the system “shut off”, so the standby case counts too: PSUs keep standby power live while plugged in, so when the machine is off, switch off or unplug both PSUs together, and never leave only one live. The same thread’s poster had an earlier dual-PSU Threadripper build stop powering on after the change, cause never found.
• Keep both PSUs on the same circuit. If they sit on different circuits and one breaker trips, the other PSU keeps feeding half the machine, which is the back-feed case above. Splitting the PSUs over two circuits is a common suggestion for getting round a circuit’s limit; jbarrow was offered it and couldn’t use it, because his apartment has only a few circuits. See the next point for what one circuit has to carry.
• Household circuit. A US 15 A circuit at 120 V carries 1800 W, and a continuous load (one running more than three hours) should stay under 80 % of that, which is 1440 W. This build’s estimate at 300 W per card is up to about 1.57 kW at the wall (arithmetic under Power & noise). That is the upper bound, with all four cards busy at once, as under vLLM tensor parallelism, batched requests or long prompt processing; a single llama.cpp stream with the layer split works the cards in turn and draws less. Plan the circuit for the upper bound anyway: a long run on a shared 15 A circuit is over the 1440 W line before anything else is plugged in. jbarrow ran four power-limited 3090 Tis from one 1650 W PSU on a 1440 W budget and still tripped the breaker: “It only tripped when I was running an OCR job while making tea.” On 120 V you have two options. Limit each card to 250 W, which brings the estimate to about 1.35 kW and leaves about 90 W for anything else on that circuit. Or run it from a 20 A circuit: 20 A × 120 V is 2400 W, and the same 80 % rule gives 1920 W. On 230 V mains there is more room: a UK plug takes at most a 13 A fuse, about 3.0 kW per plug (check the fuse actually fitted in each PSU’s plug), and a German Schuko socket is rated 16 A, about 3.7 kW. Keeping the same 80 % margin still leaves room, but the circuit behind the sockets is shared with other sockets, so keep heaters and kettles off it. Don’t run both PSUs from one power strip or extension lead: they then share that strip’s rating, which is often lower than the circuit’s. Plug each PSU straight into a wall socket on the same circuit, or check the strip’s rating against the total.
• Transient spikes and power-limiting. NVIDIA rates the RTX 3090 at 350 W and asks for a 750 W PSU for a single-card system. Igor’s Lab warns that the card’s short load peaks, at intervals of 1 to 10 ms, can trip a PSU’s protection and shut the system down. Unlimited, four cards plus the 250 W system reserve come to 1650 W from the PSUs and about 1.79 kW at the wall: about 24 % over the 1440 W continuous limit of a US 15 A circuit, and only just under its 1800 W trip rating. Set a limit with nvidia-smi -pl 300 on each card (it needs root), or 250 on a 15 A circuit. The limit does not survive a reboot or a driver reload, so reapply it at boot: Puget Systems does it on a quad-3090 machine with a systemd service that turns on persistence mode (nvidia-smi -pm 1) and then sets the limit. techpractice advises the same 300 W per card. A power limit caps average draw, not the millisecond peaks: jbarrow points out that nvidia-smi -pl 250 “does not set a hard cap at 250W” and recommends also limiting clock speeds, for example with nvidia-smi -lgc 210,1500, to avoid tripping the PSU’s overcurrent protection. That costs some speed. The PSU sizing guide covers transients, dual PSUs and the wall circuit.
• Registered memory only. The ROMED8-2T takes DDR4 RDIMM, LRDIMM or NVDIMM, one DIMM per channel across eight channels (manual PDF page 9), and the H12SSL-NT takes registered ECC DDR4 only. Desktop UDIMMs won’t work. The kit on the list is eight 32 GB RDIMMs, one per channel.
• Headless by design. EPYC has no integrated graphics. Plug a monitor into the board’s own VGA port, driven by the ASPEED AST2500 BMC on both the ROMED8-2T (manual PDF page 10) and the H12SSL-NT, or use the BMC’s remote console over the dedicated IPMI LAN port. That keeps all four 3090s free for models, which the VRAM margins in the FAQ assume. The headless setup guide covers the driver, SSH and BIOS settings for a box like this.
• Open frame. The KC-8GPU and the KC-12GPU alternate are both 360 mm wide, more than the 313 mm Founders Edition is long, but Kingwin states no maximum card length, so check a longer partner card against the rail. Kingwin also doesn’t say how many PSUs its mount holds, so plan where the second unit sits before you order; the KC-12GPU alternate (640 mm long, 550 mm tall, against the KC-8GPU’s 880 mm and 380 mm) doesn’t say either. Both frames take ATX boards, and both boards here are ATX. Skip the KC-6GPU-SS: at 285 mm wide it is narrower than the card is long.
• CPU cooler. The NH-U14S TR4-SP3 is the only SP3 cooler in the Catalog, and Noctua now marks it discontinued, so stock may run out. Its 165 mm height doesn’t matter on an open frame. It is the cooler techpractice runs on its EPYC 7402P.

Power & noise

Estimated 24/7 energy use, with every assumption stated:

• Load: all four cards power-limited to 300 W (the per-GPU figure in this site’s PSU estimate), plus 250 W for the rest of the system (the same estimate’s system reserve). That gives 4 × 300 W + 250 W = 1450 W delivered with all four cards busy, the upper bound (vLLM tensor parallelism, batched requests, prompt processing); a single llama.cpp stream works the cards in turn and draws less. Of that, up to about 1000 W comes from the AX1600i (board, two cards, and the slot power of the other two, up to 75 W each) and about 450 W from the HX1200i (the 8-pin inputs of two cards). Both stay inside this site’s PSU estimate (1280 W and 960 W at 80 %).
• Idle: Igor’s Lab measured about 16 W for one RTX 3090 at idle, so 64 W for four. The rest of the system at idle is an assumption, not a measurement: 100 W for the EPYC, eight RDIMMs, the BMC, the SSD and fans. That gives about 164 W at idle: 132 W from the AX1600i and 32 W from the HX1200i.
• At the wall: at load the AX1600i runs at about 62 % of its rating (1000 W of 1600 W) and the HX1200i at about 38 % (450 W of 1200 W). On 115 V, 80 Plus requires Titanium (the AX1600i) to reach 94 % at 50 % load and 90 % at 100 %, and Platinum (the HX1200i) 90 % at 20 % and 92 % at 50 %. Interpolating between those points, this estimate assumes 93 % and 91 %: 1000 W ÷ 0.93 ≈ 1,075 W plus 450 W ÷ 0.91 ≈ 495 W, about 1,570 W at the wall. At idle both units run below 10 % load, where 80 Plus sets no Platinum figure and requires 90 % for Titanium; this estimate assumes 90 % for both, giving about 182 W. The real idle draw is probably higher: Igor’s Lab measured a desktop idle, and a headless inference box usually runs with persistence mode on and a model held in VRAM, which keeps the cards in higher power states.
• Duty cycle: an assumption for a personal inference box: 4 hours of generation and 20 hours idle each day.

Arithmetic: 4 h × 1.570 kW ≈ 6.3 kWh, plus 20 h × 0.182 kW ≈ 3.6 kWh, gives about 9.9 kWh a day at the wall. That is about 300 kWh a month (× 30.4 days) and about 3,620 kWh a year (× 365). Because the load figure is the all-cards-busy upper bound, a box serving one llama.cpp stream at a time will use less. At 250 W per card the load figure drops to about 1.35 kW and the day to about 9.1 kWh. For comparison, the full 1.570 kW load held around the clock would be 24 h × 1.570 kW ≈ 38 kWh a day, nearly four times as much. Multiply by your own tariff; the duty cycle moves the result far more than any part choice here. The power and noise guide explains the method and its weakest inputs.

Noise: this is the loudest Tier. Four 3090 coolers and two PSUs sit in an open frame with no damping at all, and the frame’s fan mounts come empty. The upside of the frame is spacing: the cards don’t blow into each other the way the 48 GB Tier’s stacked pair does, which should help them run cooler at the same power limit. jbarrow, whose four 3090 Tis share a room with him, writes that under full load the room “can get quite warm”; he powers the rig down when it gets too loud or hot and runs bulk inference overnight. Put it in a room you don’t sit in. We have no noise measurement for this build.

FAQ

Has anyone built exactly this list? No, not that we know of. The three Basis builds each share part of it:

• level1techs uses the same board and the same dual-PSU split: an AX1600i with the board and two cards, a 1200 W unit (an EVGA P2) with the other two, synced by an Add2PSU relay, four 3090s on unpowered PCIe 4.0 x16 risers in an open rack. Its CPU is an EPYC 7532. The thread was written before the machine was first powered on, and it never reports the build running.
• techpractice runs four 3090s, 256 GB of registered DDR4, the same 990 Pro SSD and the same Noctua cooler on a Supermicro H12SSL-I-O (the sibling of the H12SSL-NT alternate) with an EPYC 7402P. It reaches the cards through bifurcated x8 risers in a 12-GPU mining case, and advises limiting each card to 300 W. Its author reports it “very stable for over a year (since Aug 2024)”. Its PSU count is inconsistent: a heading says two EVGA 1600 W units, the text says three.
• jbarrow uses the same board with 256 GB, an EPYC 7532 and one 1650 W Thermaltake PSU, but his four cards are RTX 3090 Tis, rated at 450 W each, not 3090s. It is the build that tripped a 15 A breaker.

None of the three reports LLM tokens per second, so every Model fit row is an estimate from file sizes and model configs.

How was the fit worked out? The VRAM sizing guide explains the method. The four cards hold 4 × 24,576 MiB, about 103.1 GB. Llama 3.3 70B has 80 layers, 8 key/value heads and a head size of 128, so a 16-bit KV cache takes 80 × 8 × 128 × 2 (keys and values) × 2 bytes = 327,680 bytes per token. At 32,768 tokens that is about 10.7 GB; with the 74.98 GB Q8_0 file that comes to about 85.7 GB, leaving roughly 17 GB for the CUDA context and compute buffers on four cards. At 64k the cache doubles to about 21.5 GB and the margin drops to about 6.6 GB, which is borderline. At the full 131,072 tokens the cache is about 43.0 GB; with the 42.52 GB Q4_K_M file that is about 85.5 GB. gpt-oss-120b has 36 layers, 8 key/value heads and a head size of 64, but only 18 layers use full attention; the other 18 use a 128-token sliding window. The full-attention layers take 18 × 8 × 64 × 2 × 2 = 36,864 bytes per token, about 4.8 GB at 131,072 tokens; the sliding layers need about 5 MB. With the 63.39 GB MXFP4 file that is about 68.2 GB. Even if every layer kept a full-length cache, it would be about 9.7 GB, about 73 GB in total.

Why EPYC and not the 48 GB Tier’s AM5 board? Lanes and slots. Four cards at x16 need 64 lanes; the EPYC 7302P has 128 PCIe 4.0 lanes and eight memory channels, while the 48 GB Tier’s AM5 board runs two GPUs at x8/x8. The 7302P has 16 cores, which is plenty to feed GPUs. The 7313P alternate has the same core count and lanes on the newer EPYC 7003 generation.

Why two PSUs and not one big one? This Tier follows the level1techs split: two units, each inside this site’s PSU estimate, that both run from US 115 to 120 V mains. Corsair rates the AX1600i for 1600 W from 115 V up and lists the HX1200i for 100 to 240 V input with no lower 115 V rating stated, and the second unit carries only about 450 W. One unit would have to cover 4 × 300 W + 250 W = 1450 W at 80 % of its rating, so about 1,810 W (1450 W ÷ 0.8). The Catalog’s 2000 W and 2200 W units (Super Flower Leadex Platinum 2000W, Seasonic PRIME PX-2200) take 200 to 240 V input only, so they won’t run from a standard US outlet. On 230 V mains one of them would remove the back-feed risk entirely, which is the trade-off: two units work on 115 V but need the sync and back-feed care in the warnings.

Do I need NVLink or P2P? Not for llama.cpp. Its default split mode, layer, puts whole layers on each card, so only the hidden state crosses between cards: for Llama 3.3 70B that is 8,192 values, about 16 KB per token at 16-bit or 32 KB at 32-bit, which a x16 link moves without strain, even at Gen3. Tensor parallelism in vLLM exchanges data inside every layer and benefits from GPU-to-GPU transfers; jbarrow enabled peer-to-peer on his consumer cards with modified NVIDIA drivers and measured about 26 GB/s between GPUs. NVLink is no option here anyway: RTX 3090 bridges join two cards only, at a fixed slot spacing (NVIDIA sold the GeForce bridge in 4-slot spacing only, per Puget Systems), so they don’t fit four cards on risers.

Why 256 GB of system RAM? Eight 32 GB RDIMMs fill all eight channels, one DIMM each, for full memory bandwidth. 256 GB holds the largest model files above in the page cache with room to spare, and lets you offload layers of a model that doesn’t fit in 96 GB; the system RAM guide and the model storage guide cover both. Read the used RTX 3090 checklist before buying the four cards, and test each one under load while you can still return it. If two cards are enough, the 48 GB build is simpler on every point above.