Model Quantization - GGUF Quant Levels and Memory Sizing on Ollama

Status: Active
Last Updated: 2026-08-26
Category: AI/ML - Phase 2: Local LLMs
Prerequisites: ollama-setup, model-selection-guide
Time: 1-2 hours
Tags: quantization, gguf, vram, ollama, memory-planning

Summary

What quantization actually does to a model, how to read GGUF quant tags like q4_K_M, how to size VRAM/RAM for a given model + context combination, and which quality tradeoffs matter for homelab workloads.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Decode GGUF quant level names and pick sensible defaults
✅ Estimate disk and runtime memory for any model+quant before pulling it
✅ Balance weights size against context (KV cache) within your hardware budget
✅ Choose quants deliberately in Ollama instead of accepting :latest

Context / Why This Matters

model-selection-guide picks the model family; quantization decides whether it fits your hardware and how much quality you trade away. On a homelab server this is the difference between an 8B model running fully on GPU at usable speed versus spilling to system RAM and crawling. It also interacts directly with context sizing (context-and-tokens): weights and KV cache share the same memory pool.

Implementation / Core Content

What Quantization Does

Base LLMs store weights as 16-bit floats. Quantization compresses each weight to fewer bits using grouped scaling constants, shrinking file size and memory roughly proportionally to bits-per-weight. GGUF is the file format used by llama.cpp/Ollama; the tag encodes bits per weight and the quantization scheme family.

Reading q4_K_M:

q4 ≈ 4 bits per weight
K = k-quants: block-wise schemes with better accuracy than old-style q4_0/q4_1 at the same size
M = medium mix of quant precision within blocks (variants: S small, M medium, L large)

Common Levels Compared (8B-class reference)

Tag	bpw	~Size	Quality vs fp16	Use
q8_0	8.5	~8.5 GB	~indistinguishable	When RAM allows; reference/testing
q6_K	6.6	~6.6 GB	near-lossless	Great default if it fits
q5_K_M	5.7	~5.7 GB	very close	Good balance
q4_K_M	4.8	~4.9 GB	minor loss	The community default; best size/quality
q4_0	4.0	~4.4 GB	noticeably worse	Avoid when K variants exist
q3_K_M / iq3	3.3	~3.5 GB	clear degradation	Small GPUs only
q2_K / iq2	~2.6	~2.8 GB	often broken coherence	Last resort

Rules of thumb validated across community evals:

q4_K_M is the default choice. Below q4, quality drops faster than size shrinks.
Smaller models need higher quants. A 14B at q4 usually beats an 8B at q8 while using similar memory — prefer bigger-model-lower-quant over smaller-model-higher-quant.
IQ (importance-matrix) quants (iq3_xxs, iq4_xs etc.) squeeze extra quality from low bitrates; useful on tight budgets, slightly slower on some CPUs.
Never ship anything below q3 to users who will notice "the AI got dumber".

Sizing Memory: Weights + Context

Runtime footprint = weights + KV cache (grows with context) + compute buffers (~0.5–1 GB).

Planning table (weights at q4_K_M):

Model	Weights	@8k ctx total*	@16k	@32k
3B	~2.0 GB	~3 GB	~4 GB	~6 GB
8B	~4.9 GB	~6 GB	~9 GB	~13 GB
14B	~9.0 GB	~11 GB	~15 GB	~21 GB
32B	~20 GB	~22 GB	~27 GB	~35 GB

*approximate, includes buffers; exact KV cost varies by architecture. Rule: check both numbers before pulling, then verify empirically with nvidia-smi (GPU) or RSS monitoring during a real generation.

Hardware guidance:

GPU VRAM	Comfortable target
6–8 GB	8B q4 with ≤8k ctx, or 3B q6 with more headroom
12 GB	8B q6/q8, or 14B q4 with modest ctx
16 GB	14B q4-q5, or 8B q8 with 16k ctx
24 GB	32B q4, or 14B q6 with long context

Anything exceeding VRAM falls back to CPU layers — expect 5–20x slower decode. Partial offload is a graceful degradation mode, not a plan.

Picking Quants With Ollama

Ollama tags map to quants; many models publish several:

ollama pull llama3.1:8b            # default tag = q4_K_M-class
ollama pull llama3.1:8b-instruct-q8_0
ollama pull llama3.1:8b-instruct-fp16     # rarely worth it locally

# inspect what you have
ollama show llama3.1:8b            # prints quantization + parameter info

Naming convention: <model>:<size>-instruct-q<bits>_<scheme> — e.g. qwen2.5:14b-instruct-q5_K_M. If a tag doesn't specify a quant, assume the default is q4_K_M-ish; confirm with ollama show.

Embedding models are tiny (nomic-embed-text ~270 MB) and quantization is not a concern there — spend your budget on chat models.

Decision Procedure

Note usable VRAM (nvidia-smi), subtract ~1 GB OS/desktop overhead.
Pick the largest parameter count whose q4_K_M weights fit with room for your target context's KV cache (table above).
If headroom remains after measuring real usage, step up: more context first (usually more valuable) or higher quant second.
If nothing fits: smaller model at q4 beats same-size model at q2; consider CPU-only inference for batch/offline jobs where latency doesn't matter.
Benchmark once with a representative prompt: tokens/sec via --verbose, and output quality against your prompt-engineering eval set.

Practical Examples

Example 1: Choosing for a 12 GB GPU

Target: local assistant with ~8k context.

14B q4_K_M ≈ 9 GB weights + ~1.5 GB KV/buffers ≈ 10.5 GB → fits, but tight with desktop apps open.
8B q6_K ≈ 6.6 GB + 1.5 GB ≈ 8 GB → comfortable, leaves room for embedding model loaded simultaneously.

Pick llama3.1:8b-instruct-q6_K for interactive use; keep 14b q4 as an option for harder offline tasks run overnight.

Example 2: Verifying Real Usage

# terminal 1
ollama run llama3.1:8b-instruct-q6_K --verbose
>>> summarize this long log...      # note eval rate (tok/s)

# terminal 2, during generation
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1

If used ≈ total and tok/s collapses, you're partially offloaded — reduce context or quant. Record the numbers in the model's ops notes so future sizing uses measurements, not guesses.

Example 3: A/B Testing a Quant Step

import ollama
from eval_prompt import CASES   # golden set from prompt-engineering.md

for model in ["llama3.1:8b-instruct-q4_K_M", "llama3.1:8b-instruct-q6_K"]:
    score = sum(
        classify(ollama, model, text) == expected for text, expected in CASES
    )
    print(model, f"{score}/{len(CASES)}")

If q4 and q6 tie on your tasks, keep q4 and bank the memory as context headroom.

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Generation suddenly much slower mid-session	Context growth pushed past VRAM → CPU offload	Lower num_ctx or quant; watch nvidia-smi trend
Model gives incoherent/garbled answers	Aggressive quant (q2/q3) on small model	Move up to q4_K_M minimum; try bigger-model-lower-quant instead
"Model requires more system memory than available"	Weights + KV exceed RAM	Smaller quant/model; reduce num_ctx before load
Disk full of unused variants	Pulled many quant tags experimentally	`ollama rm` stale tags; document the chosen one per service
Same model name behaves differently across hosts	Different implicit quant under same short tag	Always record full tag (`-q6_K`) in configs and docs
Higher quant shows no quality gain	Task below model's capability floor either way	Spend memory on context or a larger model instead
Embedding search degraded after "optimizing" embed model	Swapped to a quantized/different embedding model	Keep embeddings unquantized & consistent; see embeddings-vector-db

Next Steps / Ops Actions

Rebalance freed memory into longer context: context-and-tokens
Validate candidate models on real prompts: prompt-engineering
Full model-family comparison: model-selection-guide
Monitor GPU/memory trends over time: netdata-basics

Sources & Related

External references consulted:

https://github.com/ggerganov/llama.cpp/discussions/2094 (k-quants overview)
https://huggingface.co/docs/hub/en/gguf
https://github.com/ollama/ollama/blob/main/docs/import.md
Community quant benchmarks (r/LocalLLaMA summaries; treat as directional)

Change Log

2026-08-26

Initial creation by KB build session.