Best Qwen3.5 4B quant for a 8GB GPU
Short answer: Q8_0 from unsloth . Of the 8 Qwen3.5 4B files we tested that fit fully in 8GB with a 8192-token context, it has the smallest diff from the BF16 model (mean KLD 0.0019, same top token 98.3%).
- File size
- 4.17 GB
- KLD (mean)
- 0.0019
- Same top token
- 98.3%
- Tool calls
- 86.5%
- Generation
- 114 t/s full GPU
- File size
- 3.54 GB
- KLD (mean)
- 0.0037
- Same top token
- 97.1%
- Tool calls
- 86.5%
- Generation
- 130 t/s full GPU
- File size
- 1.81 GB
- KLD (mean)
- 0.2425
- Same top token
- 78.6%
- Tool calls
- 76.0%
- Generation
- 187 t/s full GPU
Other budgets: 12GB16GB24GB ยท all 8 Qwen3.5 4B quants
How we picked
We estimate VRAM need as file size + f16 KV cache for 8192 tokens + about 0.6GB of compute buffers, and keep 0.5GB free for your desktop. Among files that fit, we rank by mean KL divergence against the BF16 model's logits, then by tool-call score. If you need a longer context, step down one quant level or quantize the KV cache (-ctk q8_0 -ctv q8_0). Methodology.
GPU too small? Rent one.
Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.
Per-second GPU pods. RTX 4090 / 5090 / A100 / H100 for big quants or BF16 reference runs.
Link coming soonMarketplace GPUs, often the cheapest way to try a 24-48GB card for an evening.
Link coming soon