How Much VRAM Do You Actually Need for Local LLMs?
Card shopping for local models goes wrong the same way every time: someone reads "70B needs 40GB," buys a 24GB card, then discovers the model technically fits at Q2 but crawls, and the output quality is worse than the 32B they could have run at Q4.
This page is the arithmetic behind that decision — what fits, what spills to system RAM, and what the unlock does and doesn't change.
Three things take up VRAM, not one
- Weights — fixed by model size and quantization.
- KV cache — grows with context length, and it's what kills long-context runs on "it fits" setups.
- Runtime overhead — CUDA context, compute buffers, tokenizer, and whatever the frontend holds. Budget 1–2GB and stop pretending it's free.
The KV cache formula, per token:
bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
For a typical 8B model (32 layers, 8 KV heads, head_dim 128, FP16 cache) that's about 128 KB per token — roughly 1GB at 8k context. For a 70B with 80 layers it's about 320 KB per token, so 2.5GB at 8k. That's why a model that "fits" at 512 tokens OOMs at 8k, and why quantizing the KV cache to Q8 roughly halves it.
What fits, in practice
GGUF sizes below are approximate weight footprints. The verdict column already includes a 4k-context KV cache and ~1.5GB overhead.
| Model | Quant | Weights | 12GB card | 24GB card |
|---|---|---|---|---|
| 8B class (Llama 3.1 8B) | Q4_K_M | ~5GB | Yes, comfortably | Yes, room for 32k ctx |
| 8B class | Q8_0 | ~8.5GB | Yes, tight at long ctx | Yes |
| 13B class | Q4_K_M | ~8GB | Yes, tight | Yes |
| 13B class | Q8_0 | ~14GB | No | Yes |
| 30–32B class | Q4_K_M | ~19–20GB | No | Yes, short context |
| 70B class | Q2_K | ~26GB | No | No — over budget with KV |
| 70B class | Q4_K_M | ~40GB | No | No — needs 2 cards |
| 120B+ class | Q4_K_M | ~70GB+ | No | No — 2–3 cards |
The honest version of this table: 24GB is the sweet spot for 32B at Q4, or 8–13B at high quality with long context. 12GB is comfortable for 8B and painful above it.
What about offloading?
llama.cpp and most frontends will happily split a model between VRAM and system RAM. It runs. It's also slow — the layers that live in RAM get hit every token, over PCIe, at DDR bandwidth. A 70B at Q4 on a 24GB card with 16GB offloaded often lands in the 1–2 tokens/second range versus 6–10 for a 32B fully resident. Fit the model to the card; don't fit the card to the model.
Image models, for comparison
| Workload | VRAM at 1024² | 12GB | 24GB |
|---|---|---|---|
| SD 1.5 | ~4–6GB | Yes | Yes |
| SDXL (fp16) | ~6–8GB | Yes, tight with refiner | Yes |
| FLUX.1-dev (fp8) | ~12–16GB | Marginal / offload | Yes |
| FLUX.1-dev (fp16) | ~22–24GB | No | Barely, no headroom |
Where the GPU unlock fits into this
It doesn't change any number in the tables above. The unlock affects how fast the tensor math runs, not how much memory the model occupies. A 70B that doesn't fit in 24GB still doesn't fit after unlocking — it just fails faster.
The place it does help is throughput after the model is already resident: prompt processing and generation on cards that fit their model. See the benchmarks for the measured gap, and the safety page before assuming it's supported on your card.
Common questions
Is 12GB enough in 2026? For 8B and small image models, yes. It stops being comfortable around 13B at high quality or 32B at any quantization.
Does more VRAM beat faster compute? For LLM inference, almost always — capacity decides whether you run at all, throughput decides how pleasant it is.
Can two cards combine VRAM? For tensor-parallel inference, yes, if the model supports it and the interconnect is decent. Two 12GB cards do not behave like one 24GB card for simple layer offload, and NVLink on consumer cards is gone.
Does the unlock add memory? No. It never has. That's the single most common misconception about this tool.