VRAM Fitting Guide: Local LLM Quantization Benchmarks for 8-32GB GPUs

A 16GB GPU can run a surprisingly serious local model. I went through 38 models and quantizations and measured the actual peak VRAM usage at a 16K prompt, then added 768 MiB of headroom to see what genuinely fits on 8, 12, 16, 24 and 32GB cards. The results make the VRAM tradeoffs pretty interesting. On a 16GB card: Qwen3.8-27B UD-IQ3_XXS 92.7% board score 45.0 GPQA 12.8GB peak Qwen3.8-27B UD-IQ2_M 91.5% 11.3GB peak Qwen3.8-27B UD-IQ2_XXS 90.8% 10.1GB peak Gemma 4 12B Q6_K 87.6% 11.6GB peak gpt-oss-20b Q4_K_M 87.4% 14.5GB peak The wild part is that a heavily quantized 27B Qwen is still scoring around 5 points higher than these smaller models on the board metric. But there’s a catch. The harder GPQA-Diamond benchmark exposes some of the quality lost by pushing the quantization down. The 3-bit Qwen scores 45.0, while the 6-bit version reaches 49.0. So yes, lower-bit quantization can make the model fit. It doesn’t make the quality tradeoff disappear. The 12GB tier is also interesting. The Qwen3.8-27B UD-IQ2_XXS fits at 16K and still scores 90.8. But 8GB is where things get ugly. Nothing dense on this particular board fits fully resident at 16K. And there’s another trap with Q4_K_M. The Qwen3.8-27B Q4_K_M model file is only 15.9GB, so you might assume it should be fine on a 16GB card. Not at 16K context. Peak usage reaches 17.3GB. That’s the difference between “the model fits” and “the workload fits.” If you’re running out of VRAM, shortening the context is one option. Another is changing the KV cache precision. But the measurements here suggest using Q8 KV rather than Q4 KV if performance matters. Q4 KV can cost roughly 17–29% of decode speed depending on the model.