How much VRAM a local model needs

A model fits on a card when three things fit together: the weights, the KV cache for the context you want, and what the runtime needs to work. The first is the size of the file you download. The second you can work out from the model’s config. The third you can only estimate, so leave a margin for it. Each Build Tier on this site, Starter, 24 GB, 48 GB and 96 GB, was sized this way, and the worked examples below are the Tiers’ own numbers.

The formula

VRAM ≈ model file + KV cache + runtime overhead.

• Model file. For a GGUF model, the weights take about the size of the file. Quantisation is what sets that size: Hugging Face’s GGUF reference lists Q4_K at 4.5 bits per weight and Q6_K at 6.5625, against 16 for F16. Take the size from the file listing of the exact quantisation you will run, not from the parameter count.
• KV cache. For every token in the context, each layer stores a key and a value vector for each key/value head: layers × KV heads × head size × 2 (keys and values) × bytes per value × tokens. The first three come from the model’s config.json (num_hidden_layers, num_key_value_heads, and head_dim, or hidden_size ÷ num_attention_heads where head_dim is missing). Bytes per value is 2 for llama.cpp’s default 16-bit cache: the server README lists f16 as the default for both --cache-type-k and --cache-type-v.
• Runtime overhead. The CUDA context, compute buffers and anything else on the card, such as a desktop. We have no measurement that pins this down per model, so the Tiers treat whatever is left after file and cache as the margin for it, and call a margin under about 2 GB per card borderline.

Two unit traps. Card capacity is in binary units: a 24 GB RTX 3090 holds 24,576 MiB, which is about 25.77 GB in the decimal units that file listings use. And the context you set is the cache you pay for: llama.cpp’s --ctx-size defaults to 0, meaning the model’s own maximum, so set it to what you need. Its --fit option, on by default, adjusts settings you left unset to fit the card’s memory, which can hide that it shrank something for you.

Worked examples

All three use a 16-bit KV cache and are estimates, not measurements.

12 GB: Qwen2.5 14B at Q4_K_M, 8k context (Starter build, on the RTX 3060 12 GB). The config gives 48 layers, 8 KV heads and a head size of 128 (5,120 ÷ 40 heads). That is 48 × 8 × 128 × 2 × 2 = 196,608 bytes per token, so 8,192 tokens take about 1.61 GB. With the 8.99 GB file, about 10.6 GB, against the card’s 12,288 MiB (about 12.88 GB): roughly 2.3 GB of margin. At 16k the cache is 3.22 GB and the margin falls to about 0.7 GB, too tight to count on.

24 GB: Qwen2.5 32B at Q4_K_M, 8k context (24 GB build, one RTX 3090; if you buy it used, run the used RTX 3090 checklist, which also confirms the card reports its full 24,576 MiB). 64 layers, 8 KV heads, head size 128: 262,144 bytes per token, about 2.15 GB at 8,192 tokens. With the 19.85 GB file, about 22.0 GB against 25.77 GB, leaving about 3.8 GB. At 16k the cache is 4.29 GB and the margin is about 1.6 GB, borderline.

48 GB: Llama 3.3 70B at Q4_K_M, 8k context (48 GB build, two RTX 3090s). 80 layers, 8 KV heads, head size 128: 327,680 bytes per token, about 2.68 GB at 8,192 tokens. With the 42.52 GB file, about 45.2 GB against 2 × 25.77 ≈ 51.5 GB, leaving about 6 GB. That margin has to cover two cards’ runtime overhead and an uneven split between them (see multi-GPU below). At 16k the cache doubles to 5.37 GB and the margin drops to about 3.6 GB, about 1.8 GB per card, which the 48 GB build calls borderline.

The same file at the full 131,072-token context needs about 43.0 GB of cache on its own, more than the weights. That is why the 96 GB build lists it at 128k: about 85.5 GB in total, against about 103.1 GB on four cards. At long contexts the cache, not the weights, can be what pushes a build up a Tier.

Quantising the KV cache

llama.cpp can store the cache in lower precision: the README lists q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1 for both -ctk and -ctv. The block layouts in ggml-common.h set the bytes: a q8_0 block holds 32 values in 34 bytes (32 one-byte values and a 16-bit scale), about 1.06 bytes per value, and a q4_0 block holds 32 values in 18 bytes, about 0.56. Against 2 bytes for f16, that is about 53 % and 28 % of the cache size.

For Llama 3.3 70B at 16k on the 48 GB build, that turns the 5.37 GB cache into about 2.85 GB at q8_0, and the margin from about 3.6 GB to about 6.2 GB. At 128k on the 96 GB build, q8_0 takes the cache from 43.0 GB to about 22.8 GB.

The cost is output quality, and we have no measurement of how much for any model here. Treat a quantised cache as a lever to try, compare the answers on your own prompts, and keep in mind that every Model fit row on this site assumes the 16-bit default.

Sliding-window and mixture-of-experts models

Sliding window. Some models give only part of their layers a full-length cache. gpt-oss-120b has 36 layers, 8 KV heads and a head size of 64, but only 18 layers use full attention; the other 18 look back 128 tokens. The full layers take 18 × 8 × 64 × 2 × 2 = 36,864 bytes per token, about 4.8 GB at 131,072 tokens, and the sliding layers about 5 MB in total at their 128-token minimum (llama.cpp may hold somewhat more). Counting all 36 layers as full would give about 9.7 GB. With the 63.39 GB MXFP4 file, the real figure comes to about 68.2 GB at 131,072 tokens, which the 96 GB build fits with room to spare. So read config.json for a layer-type list or a sliding-window size before you size the cache. llama.cpp’s --swa-full option, off by default, keeps a full-size cache for the sliding layers too.

Mixture of experts. A model like Qwen3 30B-A3B uses 3B parameters per token, but all of its experts still have to be loaded: its Q4_K_M file is 18.63 GB, and that is what the card holds. The cache follows the attention layout, not the active parameters: 48 layers, 4 KV heads and a head size of 128 give 98,304 bytes per token, about 3.22 GB at 32k, for about 21.9 GB in total on the 24 GB build. If an MoE file doesn’t fit, llama.cpp’s --cpu-moe keeps all expert weights in system RAM and --n-cpu-moe N keeps those of the first N layers there, which trades VRAM for speed; we have no figure for how much speed. The system RAM guide covers what sets that speed and how much RAM it takes.

Splitting across several GPUs

llama.cpp’s default split mode, layer, splits layers and KV cache across the GPUs, so each card holds whole layers and their share of the cache. File plus cache divides across the cards; the overhead does not, because each card has its own CUDA context and compute buffers. That is why the per-card rule above applies to each card, not to the total: the 48 GB build’s 6 GB at 8k is about 3 GB per card. Which link the cards share matters much less for this split than it seems; the PCIe lanes and risers guide works through why. Each extra card also adds to the power budget, which the PSU sizing guide covers.

Layers don’t always divide evenly, and the card that drives a desktop has less room than the others. --tensor-split (-ts) sets the share each GPU gets, as proportions such as 3,1, so you can move layers off the fuller card.

vLLM works differently. Its --gpu-memory-utilization setting is the fraction of each GPU’s memory vLLM may use, and its documentation notes that a lower value takes memory away from the KV cache. On the jonready build behind the 48 GB Tier, two 3090s at 0.85 held a KV-cache pool of 61,152 tokens for Qwen3.6-27B at AWQ Int4, shared by all concurrent requests. Size a vLLM box by total pool tokens across all users, not by one request’s context. llama.cpp’s server does something similar with several slots (--parallel): with the slot count left on auto they share one unified cache, so concurrent requests divide the context between them.

Checking it on the card

The arithmetic gets you to the right card; the card itself has the last word. After the model loads, nvidia-smi shows memory used against total for each GPU. Then send the longest prompt you expect at the context you set and read it again. If a card is close to full, shorten the context, quantise the cache, or move layers with -ts before you trust it with long jobs.

Parts in this guide

Store links may earn us a commission at no cost to you. As an Amazon Associate we earn from qualifying purchases.

Related

• GPU
• Starter 12 GB build
• 24 GB build
• 48 GB build
• 96 GB build

Last reviewed 2026-11-12