Models
Models up to ~32B parameters. Up to ~14B runs fully on the 12GB benchmark GPU. 20-32B models run with a GPU+RAM split: their diff scores are exact, and their speeds are labeled as split.
Qwen3.5 4B
Qwen/Qwen3.5-4B · apache-2.0
- 8GB Q8_0 (unsloth, KLD 0.0019)
- 12GB Q8_0 (unsloth, KLD 0.0019)
- 16GB Q8_0 (unsloth, KLD 0.0019)
- 24GB Q8_0 (unsloth, KLD 0.0019)
Granite 4.2 8B
ibm-granite/granite-4.2-8b · apache-2.0
- 8GB Q4_K_M (bartowski, KLD 0.0729)
- 12GB Q8_0 (bartowski, KLD 0.0082)
- 16GB Q8_0 (bartowski, KLD 0.0082)
- 24GB Q8_0 (bartowski, KLD 0.0082)
Qwen3.5 9B
Qwen/Qwen3.5-9B · apache-2.0
- 8GB Q5_K_M (bartowski, KLD 0.0093)
- 12GB Q8_0 (unsloth, KLD 0.0025)
- 16GB Q8_0 (unsloth, KLD 0.0025)
- 24GB Q8_0 (unsloth, KLD 0.0025)