How Much VRAM Do I Need for Local LLMs?

By LLM Hardware team · Methodology · Updated October 24, 2026

For a local LLM at 4-bit (Q4_K_M), plan on about 0.6 GB of VRAM per billion parameters, plus 2 GB. That is our formula, parameters × bytes per parameter × 1.2 + 2 GB, with Q4_K_M at 0.5 bytes. So 8 GB runs 7B to 8B models (8B: 6.8 GB), 12 GB runs 14B (10.4 GB), 24 GB runs 27B to 32B (up to 21.2 GB), and a 70B needs about 44 GB. Long context adds KV cache on top: about 8.6 GB for a 32K window on Qwen3-32B. Estimates as of 19 October 2026.

Checked 17 October 2026. The formula matches the VRAM calculator. Dory 1.0 4B at Q4 is 4.4 GB and fits 8 GB. Granite Switch 4.1 8B at Q4 is 7.3 GB and fits 8 GB. Granite Switch 4.1 30B at Q4 is 20.0 GB and fits 24 GB, not 16 GB.

VRAM by Model Size and Quantization

Each cell is params × bytes × 1.2 + 2, rounded to one decimal. The 1.2 factor already includes a short context. Longer windows are the add-on in the next section, and on the VRAM calculator.

Ollama and LM Studio

Pull the tag that matches the VRAM you actually have. In LM Studio, download the Q4_K_M GGUF of the same size.

• 8 GB: ollama pull qwen3:8b then ollama run qwen3:8b (6.8 GB). Q8 of the same model is 11.6 GB and spills.
• 12 GB or 16 GB: ollama run qwen3:14b (10.4 GB). Do not load the Q8 tag (18.8 GB) or qwen3:32b (21.2 GB).
• 24 GB: ollama run qwen3:32b (21.2 GB). There is no current qwen3:34b tag.

After the model loads, run ollama ps. GPU% under 100 means part of the model is in system RAM.

The Formula

Example: 13B at Q4_K_M = 13 × 0.5 × 1.2 + 2 = 9.8 GB. The old weight-only sum (13 × 0.5 + 2 = 8.5 GB) left out the 1.2 factor.

Context on top of the formula

The published formula already includes a short context inside the 1.2 factor. The VRAM calculator adds the same extra for a longer window on every model size: 0 GB at 4K, 2 GB at 16K, 6 GB at 32K, and 22 GB at 128K. That add-on is approximate KV cache beyond the short window. It is not a second copy of the weights.

A 32B Q4_K_M at 4K is 21.2 GB. The same model at 32K is 21.2 + 6 = 27.2 GB, which misses a 24 GB card. Check the context control on the calculator before you buy for long documents.

KV cache math: exact per-model numbers, cache quantization, and parallel chats

The flat add-on above is a shortcut. The real KV cache size depends on the model's layer count and attention heads, which are published in each model's config.json on Hugging Face. Per token it is:

KV bytes per token = 2 (K and V) × layers × KV heads × head dim × bytes per value

At the default f16 cache that is 2 bytes per value. Two worked examples from the official configs:

• Qwen3-8B: 36 layers, 8 KV heads, head dim 128 → 147,456 bytes per token. A full 32,768-token window is about 4.8 GB at f16.
• Qwen3-32B: 64 layers, 8 KV heads, head dim 128 → 262,144 bytes per token. A full 32,768-token window is about 8.6 GB at f16.

So the flat +6 GB at 32K is generous for an 8B model and too small for Qwen3-32B. For a 32B model at a full 32K window, plan on roughly 21.2 + 8.6 ≈ 30 GB at the default cache. That is a 32 GB card such as the RTX 5090, not a 24 GB card.

Quantizing the KV cache

Both main runtimes can store the cache at lower precision. The Ollama FAQ documents OLLAMA_KV_CACHE_TYPE: f16 is the default, q8_0 uses about half the memory with a very small precision loss, and q4_0 uses about a quarter with a loss that may be more noticeable at long context. It needs Flash Attention and applies to every model the server loads. In llama.cpp the same switch is --cache-type-k / --cache-type-v (-ctk / -ctv) on llama-server.

With a q8_0 cache, Qwen3-32B Q4_K_M at 32K comes to roughly 21.2 + 4.3 ≈ 25.5 GB. That still misses a 24 GB card. At q4_0 it is about 23.3 GB, which loads on 24 GB with almost no headroom. For long documents on 24 GB, a 27B-class model such as Qwen3.8-27B (18.2 GB at Q4) with a q8_0 cache is the safer pick.

Parallel requests multiply the cache

Serving more than one chat at a time, for example an Open WebUI box shared by a family or an agent that runs tool calls in parallel, multiplies the cache. The Ollama FAQ says memory scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. In llama-server the slot count is --parallel. Two parallel 32K chats on Qwen3-32B need about 17.2 GB of f16 cache on top of the weights. Size for the number of users first, then pick the GPU in the best GPU for LLMs guide or the 24 GB GPU guide.

MoE models: total parameters set the size, CPU offload can move the experts

A Mixture-of-Experts model has to store every expert, so size it on the total parameter count, not the active one. Qwen3.6-35B-A3B is 35B total and 3B active, so Q4 is 35 × 0.5 × 1.2 + 2 = 23.0 GB, a tight fit on 24 GB. Only about 3B parameters are read per token, which is why it generates much faster than a dense 32B.

If the whole model does not fit, llama.cpp can keep the expert weights in system RAM and the rest on the GPU. The llama-server README lists --cpu-moe (all MoE weights on the CPU) and --n-cpu-moe N (the experts of the first N layers on the CPU). That lets a 16 GB card run a 35B-A3B model with the rest in RAM. Expect it to be slower than a full GPU fit, because those experts are read from system memory. A dense model gains far less from offload, since every weight is used for every token.

Q4 vs Q8: Is Quality Affected?

Q4_K_M is the sweet spot for most users. It cuts VRAM roughly in half compared to Q8 while retaining 95-98% of the model's quality. The difference is imperceptible in normal use. Q8 is only worth it if you need maximum accuracy for tasks like math or coding benchmarks.

What Can I Run on My GPU?

Recommended GPUs by VRAM

Checked 17 October 2026. The 4090 page said currently unavailable, so no price is shown. The 4060 page showed Buy New with only 1 left. The 4060 Ti page said currently unavailable. Check current price on Amazon.

Links to Amazon are affiliate links. No extra cost to you.

Frequently Asked Questions