System RAM for offloading models, 128 to 256 GB

The cleanest local setup keeps the whole model and its cache in VRAM, which is how every Build Tier on this site sizes its Model fit rows. System RAM still matters twice over. It holds the model file while it loads, and it is where llama.cpp puts whatever doesn’t fit on the cards. When that happens, the part of the model in RAM runs at the speed of system memory, which is a fraction of a GPU’s. This guide covers when that happens, how fast it can be, and how much RAM to buy.

When a model spills into RAM

llama.cpp decides how much goes on the GPUs with a few options, all from its server README:

• -ngl / --n-gpu-layers is the maximum number of layers kept in VRAM: an exact number, auto or all, with auto the default. Layers past that number run on the CPU, from system RAM.
• --fit, on by default, adjusts the options you left unset so the model fits the device memory, keeping a margin of 1024 MiB per device by default (--fit-target). That is convenient, and it means a model that is slightly too big for your cards loads anyway, with some layers quietly left in RAM. Watch the load log, or set -ngl yourself, if you want to know.
• --cpu-moe keeps all mixture-of-experts weights in RAM; --n-cpu-moe N keeps those of the first N layers there. --n-cpu-ffn N does the same for the dense feed-forward weights of the first N layers in a dense model.

For a dense model, a layer in RAM is read in full for every token. For a mixture-of-experts model, only the experts the router picks are read. gpt-oss-120b’s config lists 128 experts per layer and 4 used per token, so with --cpu-moe each token reads 4 of 128 experts’ weights from RAM while attention and the shared weights stay on the cards. That is why experts are the first thing to move off the GPUs, and why a 63.39 GB MXFP4 file can be worth trying on the 48 GB build, whose two cards hold about 51.5 GB. We have no measurement of the speed that gives on any build here.

Bandwidth sets the speed

Offloaded layers are bounded by memory bandwidth, and peak bandwidth is simple arithmetic: channels × transfer rate in MT/s × 8 bytes, since each channel carries 64 bits of data (Wikipedia’s DDR5 article works its bandwidth figures the same way). From the Catalog’s specs:

AMD’s EPYC 7002 datasheet gives the same 204.8 GB/s per socket, which confirms the arithmetic. For scale, the RTX 3090’s own memory moves 936 GB/s, as the Tiers give it. These are peaks; real reads come in lower, and we have no measured figure for either platform.

A ceiling, worked: Llama 3.3 70B at Q8_0 is a 74.98 GB file over 80 layers, about 0.94 GB a layer. On the 48 GB build, roughly 48 layers fit on the two cards with an 8k cache, which leaves 32 layers, about 30 GB, in RAM. Every token has to read those 30 GB once:

• at 89.6 GB/s, at least 0.33 s a token, so no more than about 3 tokens a second;
• at 57.6 GB/s, with four DIMMs, at least 0.52 s, no more than about 1.9;
• at 204.8 GB/s on eight channels, at least 0.15 s, no more than about 6.8.

That is an upper bound from arithmetic, not a measurement: it ignores the GPU’s share of each token, compute on the CPU and every other overhead. What it shows is the ratio: the eight-channel platform’s ceiling is about 2.3 times the two-channel one’s, and for generating tokens from RAM-resident layers, bandwidth is likely to matter more than core count. Prompt processing is different: for layers on the CPU it is bound by compute rather than bandwidth, so long prompts can be slow on an offloaded model. We have no measurement of either here.

Two DIMMs or four on AM5

AMD lists the Ryzen 9 9950X’s maximum memory speed as DDR5-5600 with two DIMMs (one or two ranks) and DDR5-3600 with four. Both are still two channels; the extra DIMMs add capacity, not channels. Going from the 128 GB kit’s two 64 GB DIMMs to the board’s 256 GB limit with a second kit therefore cuts peak bandwidth from 89.6 to 57.6 GB/s, about 36 % less, at least at AMD’s listed speeds. The 128 GB kit also runs 5600 only with its EXPO profile enabled. The X870E Taichi takes up to 256 GB in its four slots.

So on AM5, 128 GB in two DIMMs is the sweet spot: enough for any single model file on this site’s Tiers, at full speed. If you need 256 GB and offload speed matters, that is a reason for a server platform, not a second kit.

On EPYC the rule is one DIMM per channel. The ROMED8-2T takes registered DDR4 in eight channels, one DIMM each, which is why the 96 GB build uses the 256 GB kit of eight 32 GB RDIMMs: half as many DIMMs would mean half the channels and half the bandwidth. For more, eight of the 64 GB RDIMM give 512 GB at the same speed.

How much RAM a model needs

Size RAM as model file + the operating system and whatever else runs on the box. Why the whole file, even when most of it lives on the GPUs:

• llama.cpp memory-maps the model by default (--load-mode auto: mmap, unless a device does not support it). The file’s pages pass through the Linux page cache, which the kernel documentation describes as keeping file data in memory to avoid disk reads on later accesses.
• Layers left on the CPU are read from those cached pages for every token. If RAM is too small to keep them, the kernel drops pages and has to read them again from the drive, which is an order of magnitude slower: the Samsung 990 PRO on the Tiers reads at 7,450 MB/s against 89.6 GB/s for two channels of DDR5-5600. The model storage guide has the drive figures.
• --load-mode mlock forces the system to keep the model in RAM rather than swapping or compressing it. It is the option to try if a long-running server slows down after other jobs have used the memory. Locking needs a raised memlock limit: LimitMEMLOCK=infinity in the server’s systemd service, or ulimit -l unlimited in the shell that starts it. Without that, the lock fails.

Page cache counts as reclaimable, so a full cache is not a problem: free shows it as buff/cache, and its “available” column estimates what can still be handed to new programs without swapping.

On the Tiers’ own models, with every figure in decimal GB as file listings use it: the 128 GB kit is two 64 GiB DIMMs, about 137.4 GB. The largest file is Llama 3.3 70B at Q8_0, 74.98 GB, which on the 48 GB build leaves about 62 GB for everything else, before the OS; gpt-oss-120b at 63.39 GB leaves about 74 GB. On the 96 GB build every Model fit row runs entirely in VRAM, so its 256 GB is there to keep several model files cached at once and to try models bigger than 96 GB. We have no measurement of how much the OS and a server process need on top; leave a few GB more than you think.

So for this site’s builds: 128 GB in two DIMMs on AM5, and 256 GB in eight on EPYC. Less than the model file is the case to avoid.

Parts in this guide

Store links may earn us a commission at no cost to you. As an Amazon Associate we earn from qualifying purchases.

Related

• RAM
• CPU
• Motherboard
• Kingston FURY Beast 128GB (2x64GB) DDR5-5600 (KF556C36BBEK2-128)
• A-Tech 256GB (8x32GB) DDR4-3200 ECC RDIMM Kit (2Rx4)
• A-Tech 64GB DDR4-3200 ECC RDIMM (2Rx4)
• AMD Ryzen 9 9950X
• AMD EPYC 7302P
• ASRock X870E Taichi
• ASRock Rack ROMED8-2T
• 48 GB build
• 96 GB build

Last reviewed 2026-11-12