Model Selection Guide - Choosing the Right Model for Your Hardware and Task
Status: Active
Last Updated: 2026-08-14
Category: AImL - Phase 1: LLM Fundamentals
Prerequisites: ollama-setup
Time: 2 hours
Resources: [███░░░░░░░] 30%
Tags: models, llama, mistral, phi, qwen, gemma, benchmarks, hardware-requirements, selection
Summary
The open-weight ecosystem moves fast and picking wrong wastes hours of download time and gigabytes of disk. This lesson gives you a stable decision framework: match model families to tasks, sizes to your RAM/VRAM, and quantization levels to your quality tolerance.
🎯 What You'll Learn
- ✅ Know the major open model families (Llama, Mistral, Qwen, Gemma, Phi) and their personalities
- ✅ Reason about model size trade-offs: speed vs quality vs memory
- ✅ Size a model to your actual hardware before downloading
- ✅ Choose task-specific variants (code, math, instruct vs base)
- ✅ Evaluate candidates yourself instead of trusting hype
1. The Major Model Families
All run great in Ollama. "Personalities" below are community experience, not gospel — always test on your own workload.
| Family | Sizes (typical tags) | Character | Best at |
|---|---|---|---|
| Llama (Meta) | 3B, 8B, 70B (llama3.1, llama3.2) |
The default generalist; huge fine-tune ecosystem | General chat, instruction following |
| Mistral/Mixtral | 7B, Nemo 12B, MoE 8x7B (mistral) |
Efficient, European-hosted options, permissive licenses for some | Fast general use, EU data-residency concerns |
| Qwen (Alibaba) | 0.5B–72B (qwen2.5) |
Excellent multilingual, strong math/code at each size tier | Code, math, CJK languages |
| Gemma (Google) | 2B, 9B, 27B (gemma2) |
Polished instruction tuning, strong for size | Chat quality per parameter |
| Phi (Microsoft) | 3.8B, 14B (phi3, phi4) |
Trained on curated "textbook" data; punchy reasoning at tiny sizes | Logic/reasoning where RAM is tight |
| DeepSeek-R1 distills | 7B–70B (deepseek-r1) |
Reasoning models that emit chain-of-thought before answers | Math, logic puzzles; slower outputs |
Specialist categories you'll also meet:
- Code models:
qwen2.5-coder,codellama,starcoder2— autocomplete, code review, SQL generation - Embedding models:
nomic-embed-text,mxbai-embed-large— not chatbots! They turn text into vectors (embeddings-explained) - Vision models:
llava,llama3.2-vision— accept images (multi-modal-models)
2. Model Size Trade-offs
The Triangle
Quality
/\
/ \
/ \
/ \
Speed ------ Memory footprint
You optimize two at the expense of the third:
| Size | Generation speed (GPU) | Speed (CPU) | Quality tier | Fits in |
|---|---|---|---|---|
| 1–4B | Instant | Fast | Simple tasks, routing, extraction | Anything with 8GB RAM |
| 7–9B | Real-time | Tolerable | Good general assistant | 8–16GB RAM or 6GB+ VRAM |
| 12–15B | Real-time | Slow-ish | Solid reasoning, better code | 16GB RAM or 12GB VRAM |
| 27–35B | Fine | Painful | High-quality analysis | 24GB VRAM or 32GB+ RAM |
| 70B | OK (high-end GPU) | No | Near-frontier | 48GB VRAM / 64GB+ RAM |
Rules That Hold Up
- Start at 7–8B. It's the community sweet spot: good enough to be useful, small enough to iterate quickly.
- Go bigger only when you can articulate what the small model got wrong. Vague "it felt dumb" is not a reason to download 40GB.
- A specialized 7B beats a general 70B on narrow domains more often than people expect.
- CPU-only? Stay ≤ 14B, prefer 7–8B Q4.
- Leave headroom. The KV cache grows with context length — an 8B model at 128k context wants far more than its 5GB weight size. Budget ~1.25–2× the listed model size for comfortable operation.
Quantization: The Free Lunch (Mostly)
Same model, different precision levels (full details in model-quantization):
| Tag suffix | Bits | Relative size | Quality loss |
|---|---|---|---|
q4_K_M (Ollama default) |
~4.5 | ~30% of fp16 | Minor — usually imperceptible |
q5_K_M |
~5.5 | ~38% | Very minor |
q8_0 |
8.5 | ~53% | Negligible |
| fp16 | 16 | 100% | None (reference) |
Practical default: Q4_K_M unless you're benchmarking or have VRAM to burn.
3. Hardware Requirements Matching
Work out your budget first:
# RAM
free -h
# GPU VRAM (NVIDIA)
nvidia-smi --query-gpu=name,memory.total --format=csv
# Apple Silicon unified memory
sysctl hw.memsize
Then apply this table (Q4, leaving room for context):
| You have | Max sensible chat model | Also fits alongside |
|---|---|---|
| 8GB RAM, no GPU | 3B (llama3.2:3b, qwen2.5:3b) | nothing else heavy |
| 16GB RAM, no GPU | 8B (llama3.1:8b, mistral:7b) | small embedding model |
| 32GB RAM, no GPU | 14B (qwen2.5:14b), maybe 27B slowly | embedding + vector DB |
| 12GB VRAM GPU | 14B fully on GPU | — |
| 24GB VRAM GPU | 27–32B fully on GPU | embedding model too |
| 48GB+ VRAM | 70B class | multiple services |
What Happens if you overshoot: Ollama partially offloads to CPU (slow), swaps (agonizing), or refuses to load with OOM. Nothing breaks permanently, but you lose an hour. Check before pulling.
4. Task-Specific Selection
Match the variant to the job:
| Task | First pick | Alternative | Why |
|---|---|---|---|
| General chat/assistant | llama3.1:8b |
gemma2:9b |
Balanced instruction following |
| Coding help | qwen2.5-coder:7b |
deepseek-coder-v2 |
Trained specifically on code |
| Math/logic | deepseek-r1:8b (reasoning) |
qwen2.5:14b |
Chain-of-thought improves accuracy |
| Summarization/classification at scale | llama3.2:3b / phi3:mini |
any 3–4B | Latency and cost dominate |
| RAG over your docs | chat model + nomic-embed-text |
mxbai-embed-large |
Embedding model choice matters as much as chat model |
| Multilingual | qwen2.5 family |
llama3.1 |
Best coverage incl. Asian languages |
| Image understanding | llama3.2-vision:11b |
llava:13b |
See multi-modal-models |
Instruct vs base models: always choose instruct/chat variants (the default tags on ollama.com are). Base models just continue text — llama3.1:8b is instruct; raw base weights appear as things like llama3.1:8b-text and will frustrate you.
5. Evaluating Candidates Yourself
Benchmarks (MMLU, HumanEval, etc.) saturate and leak into training data. Trust them only for coarse ranking. For your use case, build a mini eval:
# 1. Write 10-20 prompts representing REAL tasks you'll run
cat > eval-prompts.txt <<'EOF'
Summarize this incident report in 3 bullets: <text>
Classify this support email: billing|bug|how-to: <text>
Write a systemd unit for a Python app logging to journald.
Extract all dates and amounts from: <invoice text>
EOF
# 2. Run each against candidate models
for m in llama3.1:8b qwen2.5:7b gemma2:9b; do
echo "=== $m ==="
while IFS= read -r p; do
ollama run "$m" "$p" > "out_${m}_$(echo "$p" | md5sum | cut -c1-6).txt" 2>&1
done < eval-prompts.txt
done
# 3. Read the outputs side by side. Score: correct? concise? fast enough?
Also measure speed objectively:
curl -s -w '\ntotal: %{time_total}s\n' http://localhost:11434/api/generate \
-d '{"model":"llama3.1:8b","prompt":"Count from 1 to 50.","stream":false}' \
| jq -r '.eval_count, .eval_duration'
# tokens/sec = eval_count / (eval_duration / 1e9)
Keep a simple scorecard per task. Ten minutes of structured testing beats ten forum threads.
6. A Sensible Starter Lineup
For a typical homelab server with 16–32GB RAM:
ollama pull llama3.1:8b # general workhorse
ollama pull qwen2.5-coder:7b # coding sessions
ollama pull llama3.2:3b # fast/bulk tasks
ollama pull nomic-embed-text # embeddings for RAG (Phase 3-4)
ollama list
Total disk: ~20GB. This covers 90% of what the rest of this course builds.
🛠️ Troubleshooting & Common Issues
| Symptom | Likely cause | Fix |
|---|---|---|
| Downloaded model won't load, OOM | Bigger than available RAM/VRAM after context | Smaller variant, lower quant tag (e.g., :8b-q4_0), or upgrade RAM |
| Model output looks like autocomplete gibberish | Pulled a base/text variant | Use instruct/chat tags |
| Responses in wrong language (e.g., Chinese) | Qwen defaults by prompt language detection | Set system prompt: "Always respond in English." |
Reasoning model outputs <think> blocks |
DeepSeek-R1 style CoT | Expected; strip in post-processing or prompt "think briefly" |
| Great benchmark scores, terrible on your tasks | Benchmark ≠ your distribution | Build the mini-eval from §5 |
| Model ignores tool/function instructions | Small model weak at function calling | Use larger model for agent work (agent-frameworks) |
🔗 Related
- Prev: ollama-setup
- Next: ollama-cli-basics
- Later: embeddings-explained, model-quantization, fine-tuning-basics
- Sizing context limits: context-and-tokens
📝 Change Log
- 2026-08-14 — Initial publication. Covers README outline: Llama/Mistral/Phi-3 families, size trade-offs, speed vs quality, hardware requirements, task-specific models.