Best GPU for Running LLMs Locally (2026), by VRAM Tier

Q: What is the best GPU for running LLMs locally in 2026?

The RTX 4090 (24 GB) fits 32B and 34B at Q4_K_M (21.2 GB and 22.4 GB). The RTX 5090 (32 GB) fits 32B at Q5_K_M (26.0 GB) and 34B at Q4_K_M. It does not fit 34B at Q8 (42.8 GB). A 70B model at Q4_K_M is 44 GB, so it needs two 24 GB cards or 64 GB of unified memory. On 17 October 2026 the ASUS TUF RTX 5090 OC (B0DS2X13PH) page showed See All Buying Options and no new buy box. Check current price on Amazon.

Q: Can I run a good LLM on a budget GPU?

An 8B model at Q4 is 6.8 GB and fits an RTX 4060. A 14B model at Q4 is 10.4 GB and misses 8 GB. Check current price on Amazon.

Q: Is the RTX 3090 still worth buying used for LLMs?

Both cards have 24 GB, so a 32B model at Q4 (21.2 GB) fits either one and a 70B model at Q4 (44 GB) fits neither. Listed bandwidth is 936 GB/s on the 3090 and 1,008 GB/s on the 4090. Listed TDP is 350W versus 450W. Check current price on Amazon.

Q: Is a Mac better than a GPU PC for running LLMs?

It depends on model size. For 7B–34B models, a GPU PC with an RTX 4090 is faster and cheaper. For 70B models, the NVIDIA DGX Spark (128 GB) is the top NVIDIA desktop option and the Mac Studio M4 Max (64 GB) is the top Apple option. For CUDA/TensorRT-LLM workflows, the DGX Spark wins. For macOS-native tools (MLX, Ollama on Apple Silicon), the Mac Studio wins.

Q: What is the difference between the RTX 4070 and RX 7900 XTX for LLMs?

The RTX 4070 has 12 GB and the RX 7900 XTX has 24 GB. The 4070 has the easier software path (CUDA, llama.cpp, Ollama). The 7900 XTX doubles the VRAM, enough for 34B models at Q4_K_M (22.4 GB), but AMD ROCm needs more setup. If you want maximum VRAM and are comfortable with AMD, the 7900 XTX wins. If you want the easiest setup, the RTX 4070 wins.

Q: How much VRAM do I need for a 70B model?

A 70B model at Q4_K_M is 70 × 0.5 × 1.2 + 2 = 44 GB. That does not fit a 24 GB or 32 GB consumer GPU, including the RTX 5090. Two 24 GB cards (48 GB) or 64 GB of unified memory can hold it. CPU offload runs, and tokens per second collapse.

Q: Does the RTX 5090 support LLM inference?

Yes. The RTX 5090 (32 GB GDDR7, ASIN B0DS2X13PH) works with Ollama, llama.cpp, and LM Studio. On 17 October 2026 the ASUS TUF OC page showed See All Buying Options and no new buy box. 32 GB fits a 34B model at Q4_K_M (22.4 GB) and a 32B model at Q5_K_M (26.0 GB). It does not fit 34B at Q8 (42.8 GB) or 70B at Q4_K_M (44 GB). Pull qwen3:32b for the 24–32 GB tier. Check current price on Amazon.

Q: Is 16 GB enough for Qwen3.8-27B or Gemma 4 26B-A4B?

Not as a comfortable fit. Qwen3.8-27B is a 27B dense model under Apache-2.0. The formula is 27 × 0.5 × 1.2 + 2 = 18.2 GB at Q4_K_M. Unsloth’s UD-Q4_K_XL file was 17.6 GB on 3 October 2026, before KV cache. Gemma 4 26B-A4B is 25.2B total and 3.8B active, also Apache-2.0. Count the total: 25.2 × 0.5 × 1.2 + 2 = 17.1 GB. Unsloth recommends 16–18 GB at 4-bit, and their UD-Q4_K_XL file was 17.01 GB. A 24 GB card is the discrete tier. Aleph Alpha Kolibri-1 is a different class: about 78 GB of FP8 weights, with a card minimum of datacenter GPUs. Check current price on Amazon.

By LLM Hardware team · Methodology · Updated October 24, 2026

The best GPU for running LLMs locally is the one with enough VRAM for your model. At Q4_K_M, a 14B model needs about 10.4 GB and a 20B about 14.0 GB, so 16 GB covers both. A 27B needs 18.2 GB and a 32B 21.2 GB, so they need 24 GB: a used RTX 3090 or an RTX 4090. The RTX 5090's 32 GB adds room for long context on a 32B. A 70B at Q4 is about 44 GB, more than any single graphics card here. Memory bandwidth then decides speed. Estimates as of 19 October 2026.

Key facts

Qwen3.6-27B: Apache-2.0, 27B. Q4 formula 18.2 GB. Unsloth Q4_K_M file 16.82 GB. A 24 GB card, not 16 GB.
Qwen3.6-35B-A3B: 35B total, 3B active. Q4 formula on the total is 23.0 GB. Unsloth UD-Q4_K_M file 22.13 GB. Tight on 24 GB.
RTX 5090 price: 32 GB misses 70B Q4 (44 GB) and LFM2.5-24B Q8 (30.8 GB fits). Check current price on Amazon.
Checked: 2026-10-13

Checked 2026-10-13. Author: LLM Hardware team. This site does not store or show Amazon prices. Check current price on Amazon.

Manufacturer MSRP (launch list price)

These are NVIDIA’s launch MSRPs from NVIDIA’s own pages, checked 17 October 2026. This is not Amazon’s price. Current prices are on Amazon, and partner cards often list higher.

Manufacturer MSRP (launch list price), checked 17 October 2026
GPU	Manufacturer MSRP (launch list price)	Official source
GeForce RTX 5090	$1,999	NVIDIA: starting at $1,999, 30 Jan 2025
GeForce RTX 5080	$999	NVIDIA: starting at $999, 30 Jan 2025
GeForce RTX 5070 Ti	$749	NVIDIA: starting at $749, February 2025
GeForce RTX 5070	$549	NVIDIA: starting at $549, February 2025
GeForce RTX 5060 Ti 16 GB	$429	NVIDIA: 16 GB starting at $429, 16 Apr 2025
GeForce RTX 4090	$1,599	NVIDIA Newsroom: starting at $1,599, 12 Oct 2022

Checked 17 October 2026 against the linked manufacturer pages. Not Amazon’s price; current prices are on Amazon.

Best for: choosing a GPU by the largest model you want to run: 8 GB for 8B, 12–16 GB for 14–20B, 24 GB for 32B, 32 GB for 32B at Q5.

Key specs
8 GB	8B at Q4_K_M (6.8 GB)
12 GB	14B at Q4_K_M (10.4 GB)
16 GB	20B at Q4_K_M (14.0 GB)
24 GB	32B at Q4_K_M (21.2 GB)
32 GB (RTX 5090, 1792 GB/s)	32B at Q5_K_M (26.0 GB) (source)
70B at Q4	44 GB — two 24 GB cards or 64 GB unified memory

RTX 4090 vs RTX 5090
	RTX 4090	RTX 5090
VRAM	24 GB	32 GB
Bandwidth	1008 GB/s	1792 GB/s
Best 32B quant	Q4_K_M (21.2 GB)	Q5_K_M (26.0 GB)
NVIDIA PSU	850 W	1000 W

Pairs with: PSU, RAM and case sizing for an LLM build · VRAM calculator · Can I run it? · How much VRAM do I need?

Checked 17 October 2026. Qwen3.8-Flash-Next UD-Q4_K_XL is 111.33 GB and does not fit 32 GB. Check current price on Amazon.

Citeable facts

Key fact: On 17 October 2026 the ASUS TUF RTX 5090 32 GB OC (B0DS2X13PH) did not show a new buy box. 32 GB still does not fit Qwen3.8-Flash-Next or a 70B model at Q4.

How we got it: Amazon product page. The page showed See All Buying Options and no new buy box. 70 × 0.5 × 1.2 + 2 = 44 GB. Unsloth UD-Q4_K_XL for Flash-Next summed to 111.33 GB on 9 October. Check current price on Amazon.

Checked: 2026-10-17

Key fact: Qwen3.8-27B at Q4_K_M needs 18.2 GB, so a 24 GB card fits a short context and a 16 GB card does not.

How we got it: 27 × 0.5 × 1.2 + 2 = 18.2. Unsloth UD-Q4_K_XL file was 17.6 GB on 2026-10-03. Ollama listed qwen3.8:27b at 18 GB.

Checked: 2026-10-03

Key fact: Gemma 4 26B-A4B needs 17.1 GB at Q4 when you count all 25.2B parameters, not the 3.8B that are active.

How we got it: 25.2 × 0.5 × 1.2 + 2 = 17.1. Google card and Unsloth 16–18 GB 4-bit note, checked 2026-10-03.

Checked: 2026-10-03

Methodology — parameter math, quantization bytes, and the source list.

The right GPU for local LLM inference depends entirely on which models you want to run. An entry-level RTX 4060 is enough for 7B models at Q4 quantization. Running 70B without a multi-GPU setup requires Apple Silicon with 64 GB+ of unified memory. This guide covers every major budget tier and the exact models each card can handle.

October 2026: what 16 GB and 24 GB actually load

Qwen3.8-27B is the dense laptop-class model that still wants a 24 GB card. Qwen’s card lists 27 billion parameters, a 262,144-token native context, and Apache-2.0. The short-context formula is 27 × 0.5 × 1.2 + 2 = 18.2 GB at Q4_K_M. On 3 October 2026 Unsloth’s GGUF repo showed UD-Q4_K_S at 15.4 GB, UD-Q4_K_M at 16.5 GB, and UD-Q4_K_XL at 17.6 GB. The Q8_0 file was 29 GB, and the Q8 formula is 34.4 GB, so Q8 is a 32 GB-and-up problem. Ollama’s library listed qwen3.8:27b at 18 GB. A 16 GB card can see the smaller file and still have nowhere to put KV cache. Skip 16 GB cards for this model.

Gemma 4 is the other local family worth mapping. Google’s gemma-4-26B-A4B-it card lists 25.2B total parameters and 3.8B active, Apache-2.0, text and image. Active parameters change speed. Memory follows the total, because every expert is resident: 25.2 × 0.5 × 1.2 + 2 = 17.1 GB. Unsloth’s own table says 16–18 GB at 4-bit. Their UD-Q4_K_XL file was 17.01 GB, and Google’s QAT Q4_0 GGUF was 14.44 GB. Ollama listed gemma4:26b at 16–19 GB. The edge siblings are the ones that fit a laptop: E2B and E4B. Unsloth’s Q4_K_M files were 3.11 GB and 4.98 GB, and their memory notes are 4 GB and 5.5–6 GB. Tags: gemma4:e2b and gemma4:e4b. The dense 31B is 30.7B on the card, formula 20.4 GB at Q4, which is a short-context fit on 24 GB and tight if the OS is sharing the pool.

Reflection Beam at Q4 is 302.8 GB and does not fit a 32 GB card. Granite Switch 4.1 8B at Q4 is 7.3 GB and fits an 8 GB card. Granite Switch 4.1 30B at Q4 is 20.0 GB and needs 24 GB. Dory 1.0 4B at Q4 is 4.4 GB and fits 8 GB. Kolibri-1, released 3 October 2026 by Aleph Alpha, is not in this shopping list. The card says about 78 GB of FP8 weights and a minimum of two A100 80 GB GPUs, two H100s, or one H200, B200, or B300. A Q4 sketch on 78.1B is about 48.9 GB, which still misses a 24 GB or 32 GB gaming card, and they did not publish that quant as a consumer download. If a page calls DeepSeek-V4-class or GLM-5.3-class MoEs “local,” read the gigabytes. They are workstation or API models, not a 24 GB upgrade.

If you are buying a discrete card for Qwen3.8-27B, 24 GB is the tier that matches the 18.2 GB estimate with a little KV left. The catalog card is the MSI RTX 4090 Gaming X Trio. On 17 October 2026 that listing said currently unavailable. Qwen3.6-27B is the same 18.2 GB Q4 case as Qwen3.8-27B (Unsloth Q4_K_M file 16.82 GB). Qwen3.6-35B-A3B is 23.0 GB at Q4 (UD-Q4_K_M file 22.13 GB), so 24 GB is tight and 32 GB is the comfortable card. Extra 5090 memory is for that file and for Q5, not because 27B Q4 requires 32 GB. Granite 4.2-8B (Q4_K_M file 5.35 GB) fits the 8 GB and 12 GB cards. Granite 4.2-30B (Q4_K_M file 17.72 GB) does not fit 16 GB. Check current price on Amazon.

MSI RTX 4090 Gaming X Trio, 24 GB — check the live price

ASIN B0BG94PS2F. Amazon Associates link. The live price is on that page, not locked in here.

Run the same numbers in the VRAM calculator, then match a tag on the Ollama cheat sheet or a Mac pool on the Apple Silicon guide. The tier table with E2B through 31B is on what you can run.

Quick picks by model size

7B models — RTX 4060 8GB — cheapest option that works well
14B at Q4 (10.4 GB) — RTX 4060 Ti 16GB or the RTX 5070 12GB . 14B at Q8 is 18.8 GB and misses both.
Qwen3.8-27B at Q4 (18.2 GB) — a 24 GB card. Command: ollama run qwen3.8:27b. Gemma 4 26B-A4B is 17.1 GB on its 25.2B total: ollama run gemma4:26b.
32B at Q4 (21.2 GB) — RTX 4090 24GB or RX 7900 XTX 24GB. Command: ollama run qwen3:32b.
Mac buy-now (Oct 2026) — Mac Studio M5 Max 36GB, then the 24GB Pro minis. The M4 Max 64GB Amazon listing was unavailable; 64GB+ configures at Apple.

Running 70B, and the cheapest new 16 GB card

A 70B model at Q4_K_M is 70 × 0.5 × 1.2 + 2 = 44 GB, and at Q8 it is 86 GB. No GeForce card holds either, so the 70B options are:

Two 24 GB cards (48 GB). Two RTX 3090s ran Llama 3 70B Q4_K_M at 16.3 tok/s and two RTX 4090s at 19.1 tok/s in the XiongjieDai llama.cpp runs (repo last updated May 2024). llama.cpp splits the layers across both cards.
NVIDIA RTX 6000 Ada, 48 GB. 48 GB GDDR6 with ECC at 300 W max power (NVIDIA spec page, read 14 October 2026). It holds 70B Q4 (44 GB) at a short context, with little room for a long one.
NVIDIA RTX PRO 6000 Blackwell, 96 GB. 96 GB GDDR7 with ECC, 1,792 GB/sec, 600 W max power (NVIDIA spec page, read 14 October 2026). It fits 70B at Q8 (86 GB) on one card.
Unified memory. A 64 GB+ Mac or a 128 GB Strix Halo mini PC fits 70B Q4 more slowly. See Apple Silicon for LLMs and Strix Halo mini PCs.

RTX 5060 Ti 16 GB. It is the cheapest new NVIDIA card with 16 GB: 448 GB/sec on the 16 GB and 8 GB versions (NVIDIA 50-series compare page) and 180 W total graphics power (NVIDIA spec page), both read 14 October 2026. 16 GB holds qwen3:14b at Q4 (10.4 GB) but not a 27B at Q4 (18.2 GB). Buy the 16 GB version, not the 8 GB one.

When another option is better: the RTX 5070 Ti and RTX 5080 list 896 and 960 GB/sec on the same NVIDIA compare page, about twice the 5060 Ti, so they generate faster on the same 16 GB models. If 70B is only an occasional need, a hosted API avoids buying a 48 GB card at all.

Head-to-head: 5070, 5070 Ti, 5080, 3090, and RX 9070 XT

These four comparisons used to be separate pages. The fit numbers below are the same VRAM formula used on the rest of this site. No new tokens-per-second run was added. Check current price on Amazon.

RTX 5070 vs RTX 5070 Ti

12 GB at 672 GB/s versus 16 GB at 896 GB/s. CUDA cores are 6,144 versus 8,960. Board power is 250 W versus 300 W. 14B Q4 (10.4 GB) fits both. 20B Q4 (14.0 GB) fits 16 GB only. 14B Q8 (18.8 GB) fits neither. Sources: NVIDIA GeForce 50-series compare page and the RTX 5070 family spec page, as of 2026-10-14.

ASUS TUF OC RTX 5070 — check current price on Amazon

ASIN B0DS6S98ZF. Amazon Associates link. The live price is on that page, not locked in here.

MSI Gaming Trio RTX 5070 Ti — check current price on Amazon

ASIN B0DV9GMDLR. Amazon Associates link. The live price is on that page, not locked in here.

RTX 5070 Ti vs RTX 5080

Both cards are 16 GB. Bandwidth is 896 GB/s versus 960 GB/s, about 7% on the spec sheet, not a new tokens-per-second run. CUDA cores are 8,960 versus 10,752. Board power is 300 W versus 360 W. Both fit 14B Q4 (10.4 GB) and 20B Q4 (14.0 GB). Neither fits Qwen3.8-27B Q4 (18.2 GB), Holo4-27B Q4 (formula 18.2 GB, file 16.88 GB), or Holo4-35B-A3B Q4 (formula 23.0 GB, file 21.30 GB). Sources: NVIDIA compare page, RTX 5070 family spec (300 W), and RTX 5080 spec (16 GB, 360 W), as of 2026-10-14.

MSI Gaming Trio RTX 5070 Ti — check current price on Amazon

ASIN B0DV9GMDLR. Amazon Associates link. The live price is on that page, not locked in here.

ASUS TUF RTX 5080 — check current price on Amazon

ASIN B0DQSMMCSH. Amazon Associates link. The live price is on that page, not locked in here.

RTX 3090 vs RTX 5070 Ti

24 GB at 936 GB/s versus 16 GB at 896 GB/s. Board power is 350 W versus 300 W. 32B Q4 (21.2 GB) fits 24 GB only. 14B Q4 (10.4 GB) fits both. Sources: NVIDIA Ampere GA102 whitepaper, RTX 3090 spec (24 GB, 350 W), 50-series compare page, and RTX 5070 family spec (16 GB, 300 W), as of 2026-10-14.

RTX 3090 — check current price on Amazon

ASIN B08HR6ZBYJ. Amazon Associates link. The live price is on that page, not locked in here.

MSI Gaming Trio RTX 5070 Ti — check current price on Amazon

ASIN B0DV9GMDLR. Amazon Associates link. The live price is on that page, not locked in here.

RX 9070 XT vs RTX 5070

16 GB at up to 640 GB/s (AMD spec; an earlier draft used 896 GB/s) versus 12 GB at 672 GB/s. 14B Q4 (10.4 GB) fits both. 20B Q4 (14.0 GB) fits 16 GB only. 14B Q8 (18.8 GB) fits neither. No RX 9070 XT Amazon ASIN was verified, so this section does not link one. Sources: AMD RX 9070 XT spec (16 GB, up to 640 GB/s, 304 W), NVIDIA compare page, and NVIDIA 5070 family spec (12 GB, 250 W), as of 2026-10-14.

ASUS TUF OC RTX 5070 — check current price on Amazon

ASIN B0DS6S98ZF. Amazon Associates link. The live price is on that page, not locked in here.

GPU Comparison Table

Checked 17 October 2026. The 4090 and 4060 Ti pages said currently unavailable. The 4060 page showed Buy New with only 1 left. The 5090 page showed See All Buying Options. Check current price on Amazon.

Links to Amazon are affiliate links. No extra cost to you.

GPU	VRAM	Price	Max model	Quantization	Best for	Buy
RTX 4060 8GB	8 GB	Check current price on Amazon	qwen3:8b (6.8 GB)	Q4_K_M	Not 14B (10.4 GB). Check current price on Amazon.	Buy
RTX 4060 Ti 16GB	16 GB	Check current price on Amazon	20B Q4 (14.0 GB)	Q4_K_M	Not 14B Q8 (18.8 GB). Check current price on Amazon.	Buy
RTX 4070 12GB	12 GB	Check current price on Amazon	qwen3:14b (10.4 GB)	Q4_K_M	Fits Phi-4 Q4 (10.4 GB) and d1-3B Q4 (3.9 GB). Not 27B. Check current price on Amazon.	Buy
RTX 5070 12GB	12 GB	Check current price on Amazon	qwen3:14b (10.4 GB)	Q4_K_M	Not 16 GB and not 27B Q4. Check current price on Amazon.	Buy
RTX 5070 Ti 16GB	16 GB	Check current price on Amazon	20B Q4 (14.0 GB)	Q4_K_M	Not Qwen3.8-27B (18.2 GB). Check current price on Amazon.	Buy
RTX 4070 Ti Super 16GB	16 GB	Check current price on Amazon	20B Q4 (14.0 GB)	Q4_K_M	That is not a new buy box. Not 27B Q4 (18.2 GB). Check current price on Amazon.	Buy
RX 7900 XTX 24GB	24 GB	Check current price on Amazon	34B	Q4_K_M	Budget 24 GB VRAM (AMD)	Buy
RTX 3090 24GB	24 GB	Check current price on Amazon.	32B Q4 (21.2 GB)	Q4_K_M	Not 70B Q4 (44 GB). Check current price on Amazon.	Buy
RTX 4080 16GB	16 GB	Check current price on Amazon	20B Q4 (14.0 GB)	Q4_K_M	Not Qwen3.8-27B (18.2 GB). Check current price on Amazon.	Buy
RTX 4090 24GB	24 GB	Check current price on Amazon	Qwen3.8-27B Q4 (18.2 GB)	Q4_K_M	Fits 32B Q4 (21.2 GB). Not 70B Q4 (44 GB). Check current price on Amazon.	Buy
RTX 5090 32GB	32 GB	Check current price on Amazon	32B Q5 (26.0 GB)	Q5_K_M	On 2026-10-17 the page showed See All Buying Options and no new buy box. Not 70B Q4 (44 GB) and not Flash-Next. Check current price on Amazon.	Buy
Mac Studio M4 Max 64GB	64 GB	Prior generation	48–64GB tier	Q4 / higher precision 27B	Prior generation. Not a buy-now pick.	—
Mac Studio M4 Max 128GB	128 GB	Prior generation	large MoE quant	aggressive Q4	Prior generation. Not a buy-now pick.	—
NVIDIA DGX Spark	128 GB	Check current price on Amazon	200B	Q4_K_M	Best desktop NVIDIA 70B–200B inference	Buy

"Max model" is the largest parameter count that fits in VRAM at the listed quantization with at least 1 GB headroom. Tokens/sec ranges (in the tier breakdown below) are cross-referenced with the XiongjieDai community llama-bench runs and the Home GPU LLM Leaderboard. Use the VRAM Calculator for exact figures. Compare any two options side-by-side on the Compare page.

Budget Tier Breakdown

Budget tier

Budget Pick

RTX 4060 8GB

Entry-level local inference, 7B chat & coding models

Check current price on Amazon

VRAM

8 GB

Max model

7B at Q4_K_M

Speed

not listed

Pros

+ Cheapest GPU for usable LLM inference
+ Low power draw (115 W TDP)
+ Fits Qwen3 8B, Llama 3.1 8B, DeepSeek-R1-Distill-8B at Q4_K_M

Cons

- 8 GB VRAM — cannot run 13B+ models
- No headroom for Q8 on 7B

Buy on Amazon

The RTX 4060 is the go-to entry point for local LLMs in 2026. It runs any 7B model at Q4_K_M quantization with a comfortable 1–3 GB of headroom. You cannot run 13B or larger models without slow CPU offloading, but a well-tuned 7B model handles most everyday tasks.

Value tier

Best Value 16GB

RTX 4060 Ti 16GB

Best VRAM-per-dollar — 16 GB CUDA. Listing extract 2 October 2026.

Check current price on Amazon

VRAM

16 GB

Max model

20B Q4 (14.0 GB) / 14B Q4 (10.4 GB)

Speed

20–35 t/s

Pros

+ 16 GB VRAM — more than the RTX 4070 12 GB
+ Fits qwen3:14b at Q4_K_M (10.4 GB) and 20B Q4 (14.0 GB)
+ Full CUDA support, low power draw (165 W TDP)

Cons

- Slower than RTX 4070 Ti / 4090 at same model size
- Limited by slower memory bus (256-bit vs 192-bit on 4070)
- Cannot run 34B+ models

Buy on Amazon

16 GB fits qwen3:14b at Q4_K_M (10.4 GB) and a 20B model at Q4_K_M (14.0 GB). It does not fit 13B at Q8 (17.6 GB) or 14B at Q8 (18.8 GB). In LM Studio, download the Q4_K_M GGUF. Run ollama run qwen3:14b, then ollama ps. Check current price on Amazon.

Mid-range tier

Mid-Range Sweet Spot

RTX 4070 12GB

Fast 7B at Q8, 13B at Q4, best CUDA support

Check current price on Amazon

VRAM

12 GB

Max model

13B at Q4_K_M

Speed

30–50 t/s

Pros

+ Excellent inference speed for the price
+ Full CUDA support — widest framework compatibility
+ Runs 13B at Q4_K_M with ~3 GB headroom

Cons

- 12 GB limits you to 13B at Q4 — no 34B
- Small step up in VRAM vs RTX 4060

Buy on Amazon

RTX 4070 Ti Super 16GB

Best CUDA 16GB value — 2.3x faster than 4060 Ti at same VRAM

Check current price on Amazon

VRAM

16 GB

Max model

20B Q4 (14.0 GB) / 14B Q4 (10.4 GB)

Speed

50–100 t/s at 7B Q4

Pros

+ 672 GB/s bandwidth — faster than the RTX 4060 Ti at the same 16 GB
+ Fits 20B at Q4_K_M (14.0 GB) and qwen3:14b (10.4 GB)
+ Does not fit 14B at Q8 (18.8 GB)

Cons

- More expensive than the 4060 Ti — paying for speed, not VRAM
- Still limited to 16 GB — cannot run 34B+
- 285 W TDP — higher power than 4060 Ti

Buy on Amazon

RX 7900 XTX 24GB

34B models on a budget, large VRAM at low cost

Check current price on Amazon

VRAM

24 GB

Max model

34B at Q4_K_M

Speed

15–25 t/s

Pros

+ 24 GB VRAM — best VRAM-per-dollar
+ Handles 34B models at Q4_K_M
+ llama.cpp + Ollama ROCm support

Cons

- ROCm software less mature than CUDA
- Slower than NVIDIA at same VRAM size
- Some tools require manual ROCm setup

Buy on Amazon

The RTX 4070 (12 GB) fits 13B at Q4_K_M (9.8 GB) and qwen3:14b (10.4 GB). It does not fit 14B at Q8 (18.8 GB). The RTX 4070 Ti Super (16 GB) fits 20B at Q4 (14.0 GB), not 13B at Q8 (17.6 GB). The RX 7900 XTX (24 GB) fits qwen3:32b at Q4 (21.2 GB). AMD ROCm needs more setup than CUDA. Check current price on Amazon.

Used-market tier

Best Used Value

RTX 3090 24GB (used)

Budget 24 GB VRAM build — same capacity as RTX 4090

Check current price on Amazon

VRAM

24 GB

Max model

34B at Q4_K_M

Speed

15–22 t/s

Pros

+ 24 GB VRAM at roughly one-third the price of a new RTX 4090
+ Full CUDA support — works with all frameworks
+ Can run same model sizes as RTX 4090

Cons

- 25–30% slower inference than RTX 4090
- Used market — no warranty, condition varies
- Higher power draw (350 W TDP)

Buy on Amazon

The RTX 3090 is the used-market standout for 2026. You get 24 GB of CUDA VRAM — the same as the RTX 4090 — on eBay or local resale markets. Inference speed is about 25–30% lower than the RTX 4090 due to the older memory architecture, but you can run the same models. If you want 24 GB VRAM without paying flagship prices, this is the best-value move.

Performance tier

Performance Pick

RTX 4080 16GB

Check current price on Amazon.

Check current price on Amazon

VRAM

16 GB

Max model

20B Q4 (14.0 GB)

Speed

35–55 t/s at 13B

Pros

+ 716 GB/s memory bandwidth — faster tokens/s than 4060 Ti at same VRAM
+ Same 16 GB as 4060 Ti but significantly faster inference
+ Full CUDA support, Ada Lovelace architecture

Cons

- Same 16 GB VRAM ceiling as the RTX 4060 Ti
- Much pricier than the 4060 Ti for ~1.5–2x the speed
- RTX 5080 offers higher bandwidth at a higher price

Buy on Amazon

The RTX 4080 is the speed pick in the 16 GB tier. You get the same 16 GB of VRAM as the RTX 4060 Ti but with 716 GB/s bandwidth — roughly 2.5x more throughput. That translates to noticeably faster tokens per second on 13B–20B models. If inference speed matters more than VRAM capacity, the 4080 is worth the premium over the 4060 Ti. For max VRAM on a budget, go with the 4060 Ti instead.

Flagship tier

Best Consumer GPU

RTX 4090 24GB

Best single-GPU fit for Qwen3.8-27B Q4 (18.2 GB). Check current price on Amazon.

Check current price on Amazon

VRAM

24 GB

Max model

34B Q4 (22.4 GB) / 32B Q4 (21.2 GB)

Speed

not listed

Pros

+ 24 GB fits 34B at Q4 (22.4 GB)
+ Full CUDA support across all frameworks
+ Qwen3.8-27B Q4 (18.2 GB) fits

Cons

- Premium price
- Still cannot run 70B models without CPU offloading
- 450 W TDP — high power draw

Buy on Amazon

The RTX 4090 is a 24 GB card. A 34B model at Q4 is 22.4 GB and fits. A 13B model at Q8 is 17.6 GB and fits. A 13B model at FP16 is 33.2 GB and does not fit. A 70B model at Q4 is 44 GB and does not fit. Check current price on Amazon.

Top tier

Top of the Line

RTX 5090 32GB

32 GB GDDR7. Does not fit 70B Q4 (44 GB) or 34B Q8 (42.8 GB).

Check current price on Amazon

VRAM

32 GB

Max model

32B Q5 (26.0 GB) / 34B Q4 (22.4 GB)

Speed

35–55 t/s at 13B

Pros

+ 32 GB GDDR7 — fits 32B at Q5_K_M (26.0 GB)
+ ASUS TUF OC, ASIN B0DS2X13PH. Check current price on Amazon.
+ Fits ollama run qwen3:32b (21.2 GB) with more headroom than 24 GB

Cons

- Most expensive consumer GPU
- Costs more than the 4090 but is meaningfully more capable
- 575 W TDP — requires high-end PSU

Buy on Amazon

The RTX 5090 is 32 GB of GDDR7. On 17 October 2026 the ASUS TUF OC (B0DS2X13PH) page showed See All Buying Options and no new buy box. 32 GB fits a 34B model at Q4_K_M (22.4 GB) and a 32B model at Q5_K_M (26.0 GB). It does not fit Qwen3.8-Flash-Next (UD-Q4_K_XL 111.33 GB), 34B at Q8 (42.8 GB), or 70B at Q4_K_M (44 GB). Use ollama run qwen3:32b. In LM Studio, the matching file is Qwen3 32B Q4_K_M or Q5_K_M, not Q8. Check current price on Amazon.

Workstation tier

Best for 70B

Mac Studio M4 Max (64GB), prior generation

Prior generation. ASIN kept. Not a buy-now pick.

Prior generation

VRAM

64 GB unified

Max model

70B at Q4_K_M, 34B at Q8

Speed

not measured here

Pros

+ 64 GB unified memory — runs 70B at Q4_K_M with 27 GB headroom
+ Silent, low power (300 W max)
+ Excellent llama.cpp Metal performance
+ No driver issues — just works

Cons

- Expensive for the GPU performance
- Slower than RTX 4090 on 13B–34B models
- Tied to macOS and Apple ecosystem

Mac Studio M4 Max (128GB), prior generation

Prior generation. ASIN kept. Not a buy-now pick.

Prior generation

VRAM

128 GB unified

Max model

70B at Q8, 34B at FP16

Speed

not measured here

Pros

+ 128 GB — fits 70B at Q8 (near-lossless quality)
+ Silent, energy efficient (300 W max)
+ Best option for macOS-native LLM tools (MLX framework)

Cons

- Far more expensive than the NVIDIA DGX Spark for same VRAM
- Slower AI compute than DGX Spark for inference workloads
- Tied to macOS and Apple ecosystem

NVIDIA DGX Spark

70B–200B NVIDIA inference, CUDA/TensorRT-LLM stack

Check current price on Amazon

VRAM

128 GB unified (LPDDR5X)

Max model

70B at Q8, 200B at Q4

Speed

~8 t/s at 70B Q4, ~4 t/s at 70B Q8

Pros

+ 128 GB unified memory — far cheaper than the Mac Studio 128GB
+ 1 PFLOPS FP4 AI compute — purpose-built for LLM inference
+ Full CUDA support: TensorRT-LLM, llama.cpp CUDA, vLLM
+ Can dual-link two units for 256 GB to run 405B models

Cons

- ~8 t/s at 70B is slower than RTX 4090 at 34B (memory bandwidth limited)
- No GPU display output — pure compute device
- New product: limited community benchmarks at launch

Buy on Amazon

For 70B and larger models, the NVIDIA DGX Spark is the current desktop NVIDIA option in this table. It runs 70B at Q8 and 200B at Q4. The Mac Studio M4 Max 64 GB and 128 GB rows are prior generation. Their ASINs stay in the catalog and they are not buy-now picks. Current Mac Studio memory steps are on the hardware index. For CUDA and TensorRT-LLM workflows, the DGX Spark is the NVIDIA choice.

How to Choose the Right GPU

A 24 GB RTX 4090 fits a 34B model at Q4 (22.4 GB). It does not fit that model at FP16, and it does not fit a 70B model at Q4 (44 GB). A 12 GB RTX 4070 fits a 13B model at Q4 (9.8 GB) and misses a 14B model at Q8 (18.8 GB). Check current price on Amazon.

1.
Decide which models you want to run. The model size — not inference speed — is the primary constraint. A 7B model at Q4 is 6.2 GB by our VRAM formula. A 70B model at Q4 is 44 GB. Use the VRAM Calculator to check any specific model.
2.
Choose VRAM first, speed second. A faster GPU that cannot fit your model is useless. Prioritize having enough VRAM with at least 1–2 GB headroom for KV cache. Then consider inference throughput.
3.
Prefer NVIDIA if software maturity matters. CUDA is better supported across Ollama, llama.cpp, LM Studio, vLLM, and most fine-tuning tools. AMD ROCm works for inference but requires more configuration and has less community troubleshooting content.
4.
Consider the used market for 24 GB VRAM. A used RTX 3090 gives you the same 24 GB VRAM as the RTX 4090 at around 25–30% lower inference speed. If budget is tight and you can accept slower throughput, it is a legitimate option.
5.
For 70B, go Apple Silicon or multi-GPU. No single consumer GPU card fits 70B at Q4_K_M. The Mac Studio M4 Max (64 GB) is the practical single-device option. Dual RTX 4090s (48 GB) work via llama.cpp tensor splitting but require a compatible motherboard and higher power draw.

Quantization and VRAM: Quick Reference

Every "max model" figure in this guide assumes Q4_K_M quantization unless noted. Quantization determines how many bits each model weight uses — and therefore how much VRAM the model occupies.

Format	Bits/weight	Bytes/param	Quality loss	Typical use
Q4_K_M	4-bit	~0.5	~1–3%	Default for consumer GPUs — best VRAM efficiency
Q5_K_M	5-bit	~0.625	~0.5–1%	Slight quality improvement over Q4 with moderate VRAM cost
Q8_0	8-bit	~1.0	<0.1%	Near-lossless — use when VRAM allows
FP16	16-bit	2.0	Reference	Full precision — fine-tuning, research

Use the VRAM Calculator to compute exact memory requirements for any model size and quantization combination.

Frequently Asked Questions

What is the best GPU for running LLMs locally in 2026?

The RTX 4090 (24 GB) fits 32B and 34B at Q4_K_M (21.2 GB and 22.4 GB). The RTX 5090 (32 GB) fits 32B at Q5_K_M (26.0 GB) and 34B at Q4_K_M. It does not fit 34B at Q8 (42.8 GB). A 70B model at Q4_K_M is 44 GB, so it needs two 24 GB cards or 64 GB of unified memory. On 17 October 2026 the ASUS TUF RTX 5090 OC (B0DS2X13PH) page showed See All Buying Options and no new buy box. Check current price on Amazon.

Can I run a good LLM on a budget GPU?

An 8B model at Q4 is 6.8 GB and fits an RTX 4060. A 14B model at Q4 is 10.4 GB and misses 8 GB. Check current price on Amazon.

Is the RTX 3090 still worth buying used for LLMs?

Both cards have 24 GB, so a 32B model at Q4 (21.2 GB) fits either one and a 70B model at Q4 (44 GB) fits neither. Listed bandwidth is 936 GB/s on the 3090 and 1,008 GB/s on the 4090. Listed TDP is 350W versus 450W. Check current price on Amazon.

Is a Mac better than a GPU PC for running LLMs?

It depends on model size. For 7B–34B models, a GPU PC with an RTX 4090 is faster and cheaper. For 70B models, the NVIDIA DGX Spark (128 GB) is the top NVIDIA desktop option and the Mac Studio M4 Max (64 GB) is the top Apple option. For CUDA/TensorRT-LLM workflows, the DGX Spark wins. For macOS-native tools (MLX, Ollama on Apple Silicon), the Mac Studio wins.

What is the difference between the RTX 4070 and RX 7900 XTX for LLMs?

The RTX 4070 has 12 GB and the RX 7900 XTX has 24 GB. The 4070 has the easier software path (CUDA, llama.cpp, Ollama). The 7900 XTX doubles the VRAM, enough for 34B models at Q4_K_M (22.4 GB), but AMD ROCm needs more setup. If you want maximum VRAM and are comfortable with AMD, the 7900 XTX wins. If you want the easiest setup, the RTX 4070 wins.

How much VRAM do I need for a 70B model?

A 70B model at Q4_K_M is 70 × 0.5 × 1.2 + 2 = 44 GB. That does not fit a 24 GB or 32 GB consumer GPU, including the RTX 5090. Two 24 GB cards (48 GB) or 64 GB of unified memory can hold it. CPU offload runs, and tokens per second collapse.

Does the RTX 5090 support LLM inference?

Yes. The RTX 5090 (32 GB GDDR7, ASIN B0DS2X13PH) works with Ollama, llama.cpp, and LM Studio. On 17 October 2026 the ASUS TUF OC page showed See All Buying Options and no new buy box. 32 GB fits a 34B model at Q4_K_M (22.4 GB) and a 32B model at Q5_K_M (26.0 GB). It does not fit 34B at Q8 (42.8 GB) or 70B at Q4_K_M (44 GB). Pull qwen3:32b for the 24–32 GB tier. Check current price on Amazon.

Is 16 GB enough for Qwen3.8-27B or Gemma 4 26B-A4B?

Not as a comfortable fit. Qwen3.8-27B is a 27B dense model under Apache-2.0. The formula is 27 × 0.5 × 1.2 + 2 = 18.2 GB at Q4_K_M. Unsloth’s UD-Q4_K_XL file was 17.6 GB on 3 October 2026, before KV cache. Gemma 4 26B-A4B is 25.2B total and 3.8B active, also Apache-2.0. Count the total: 25.2 × 0.5 × 1.2 + 2 = 17.1 GB. Unsloth recommends 16–18 GB at 4-bit, and their UD-Q4_K_XL file was 17.01 GB. A 24 GB card is the discrete tier. Aleph Alpha Kolibri-1 is a different class: about 78 GB of FP8 weights, with a card minimum of datacenter GPUs. Check current price on Amazon.

GPU-specific guides

Comparison — 16 GB vs 24 GB

RTX 4060 vs RTX 4070

Comparison — 8 GB vs 12 GB

RTX 4060 vs Arc B580

Comparison — CUDA vs Arc, budget tier

RTX 4070 vs 4080 vs 4090

Comparison — 12 GB vs 16 GB vs 24 GB

RTX 3090 vs 4070 Ti Super

Comparison — used 24 GB vs new 16 GB

Mac mini vs a discrete GPU

Unified memory compared with VRAM

RTX 3090

24 GB

RTX 3080

10 GB · used · fast bandwidth

RX 7900 XTX · ROCm guide

AMD RX 7900 XTX

24 GB · best AMD value

Mac Studio M5 Max 48GB

48 GB unified memory

AMD RX 9070 XT for LLMs

New RDNA 4 GPU: 16 GB GDDR6, matches RTX 5080 throughput.

AMD vs NVIDIA for LLMs

ROCm vs CUDA — which platform to choose

Best Budget GPU for LLMs

Top budget picks for local AI

Model hardware guides

Qwen3

0.6B–32B · dense and MoE

Llama 3.3 70B

44 GB at Q4_K_M · ollama llama3.3:70b

DeepSeek R1

Distill 7B–70B · reasoning

Gemma 3

1B–27B · Google open models

Phi-4

14B · 10.4 GB at Q4_K_M · phi4:14b

Mistral

7B–22B · VRAM efficient

Llama 4 Scout

109B total params · 67.4 GB at Q4_K_M

Find hardware for a specific model or check exact VRAM requirements.

VRAM Calculator Compare GPUs Browse All Models

Related Guides

VRAM Calculator

Same formula: 32B Q4 is 21.2 GB and 70B Q4 is 44 GB.

How Much VRAM Do I Need?

Q4, Q5, Q8, and FP16 for 7B through 70B.

Best Budget GPUs

The RTX 5070 is 12 GB, not 16 GB.

Best 24 GB VRAM GPU

Cards that fit qwen3:32b at 21.2 GB.

Ollama Cheat Sheet

ollama run qwen3:14b and how to read ollama ps.

LLM Quantization Guide

Q4_K_M is 0.5 bytes per parameter. Q8 is 1.0.

Sources & methodology

VRAM figures on this page come from the site’s VRAM formula and model cards. The sitewide formula and the full source list are on the methodology page. For this guide specifically we used:

XiongjieDai GPU-Benchmarks-on-LLM-Inference — community llama-bench runs across 30+ consumer and workstation GPUs, source of the per-card tokens-per-second numbers.
Home GPU LLM Leaderboard — ranked head-to-head leaderboard used to sanity-check the tier ordering at each VRAM bucket.
Hardware Corner GPU ranking — independent local-LLM GPU ranking; used as a tiebreaker where the other two disagreed by more than ~10%.
llama.cpp llama-bench discussion — the upstream benchmarking thread the community numbers all derive from.

Spot a number that does not match the linked source? Contact us and we will update the guide.

Sources & last reviewed

Last reviewed October 18, 2026 by the LLM Hardware team. The date changes only when the numbers or sources on this page change.

Vendor spec pages

NVIDIA GeForce 50-series compare page — memory bandwidth for RTX 5090 / 5080 / 5070 Ti / 5070 / 5060 Ti.
NVIDIA GeForce RTX 5090 specs — 32 GB GDDR7.
NVIDIA GeForce RTX 5080 specs — 16 GB GDDR7.
NVIDIA GeForce RTX 5070 family specs — RTX 5070 12 GB, RTX 5070 Ti 16 GB.
NVIDIA GeForce RTX 5060 family specs — RTX 5060 Ti 16 GB / 8 GB, 180 W.
NVIDIA GeForce RTX 4090 specs — 24 GB GDDR6X.
NVIDIA GeForce RTX 4060 family specs — RTX 4060 Ti 16 GB / 8 GB, RTX 4060 8 GB.
NVIDIA GeForce RTX 3090 / 3090 Ti specs — 24 GB GDDR6X on both cards.
AMD Radeon RX 7900 XTX specs — 24 GB GDDR6.
NVIDIA RTX 6000 Ada specs — 48 GB GDDR6 ECC, 300 W.
NVIDIA RTX PRO 6000 Blackwell specs — 96 GB GDDR7 ECC, 1,792 GB/sec, 600 W.
Apple Mac Studio tech specs — M5 Max 32-core GPU 36GB at 460GB/s; M5 Max 40-core GPU at 614GB/s for 48GB, 64GB, or 128GB; M5 Ultra 96GB at 1.2TB/s (checked 2026-10-18).

Benchmarks

XiongjieDai GPU-Benchmarks-on-LLM-Inference (GitHub) — llama.cpp tok/s for RTX 3090 / 4090 single and multi-GPU (repo last updated May 2024).

Model cards

Qwen/Qwen3-14B model card (Hugging Face) — 14.8B parameters; the 14B row.
Qwen/Qwen3-32B model card (Hugging Face) — 32.8B parameters, 32,768-token native context; the 32B row.
Qwen/Qwen3.6-27B model card (Hugging Face) — 27B-class, Apache-2.0; the 24 GB tier example.
Qwen/Qwen3.6-35B-A3B model card (Hugging Face) — MoE, 35B total / 3B active; we size on the total.
meta-llama/Llama-3.3-70B-Instruct model card (Hugging Face) — 70B parameters; the 70B row (44 GB at Q4).

VRAM estimates use params × bytes per parameter × 1.2 + 2 GB (Q4_K_M ≈ 0.5 bytes), explained on the methodology page. Found a mismatch? Tell us on the contact page.