Context and Tokens - Context Windows, Cost, and Management

Status: Active
Last Updated: 2026-08-26
Category: AI/ML - Phase 2: Local LLMs
Prerequisites: llm-introduction, ollama-api
Time: 2 hours
Tags: tokens, context-window, num_ctx, summarization, truncation, retrieval

Summary

What tokens and context windows actually are, how to do the latency and capacity math for a local Ollama host, and the practical strategies — truncation, sliding windows, summarization, retrieval — for staying inside the window without losing the plot.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Estimate token counts for English text, code, and logs
✅ Compute whether a workload fits a given context window and RAM budget
✅ Predict prompt-size vs generation-speed tradeoffs on your hardware
✅ Choose between truncation, summarization, and RAG for context management

Context / Why This Matters

Every LLM call in python-llm-integration sends an entire conversation (or document) as input. Whether that call succeeds, how fast it returns, and how much RAM your homelab server burns all come down to one number: total tokens in the context window. On local hardware this bites harder than in the cloud — there is no elastic scaling when num_ctx pushes KV-cache memory past what your GPU has.

Context management strategy also directly shapes architecture: it's the reason rag-pipeline exists instead of just "paste everything into the prompt".

Implementation / Core Content

Tokens: The Unit of Everything

Models don't read characters; they read tokens produced by a tokenizer (BPE variants). Rules of thumb:

Text type	Approx tokens
English prose	~~4 chars/token (~~0.75 words/token)
Code / JSON / YAML	denser — often ~3 chars/token
Log lines	highly variable; timestamps and UUIDs tokenize poorly (5–10+ tokens each)
Non-English text	frequently 2–4x more tokens than English

Count precisely rather than guessing:

ollama run llama3.1:8b --verbose "your prompt here"   # prints eval count in stats

# via API: the response metadata includes counts
import ollama
r = ollama.chat(model="llama3.1:8b", messages=[{"role": "user", "content": s}])
print(r["prompt_eval_count"], r["eval_count"])   # prompt tokens, generated tokens

The Context Window

The context window = system + history + retrieved documents + few-shot examples + your question + the model's own output, all counted together. If total exceeds num_ctx, Ollama silently truncates the oldest context (or the request misbehaves) — output quality collapses without an error message. Default num_ctx in Ollama has historically been small (2048/4096); set it explicitly.

Choosing num_ctx:

ollama.chat(model="llama3.1:8b", options={"num_ctx": 8192}, ...)

Memory Math (Why Bigger Windows Hurt)

Attention needs a KV cache that grows with context length. Rough sizing for common models:

Model	Weights (Q4)	KV cache @4k ctx	KV cache @16k	KV cache @32k
llama3.1:8b	~4.9 GB	~1 GB	~4 GB	~8 GB
qwen2.5:14b	~9 GB	~1.6 GB	~6.5 GB	~13 GB

(Exact numbers vary by architecture/GQA; treat as planning figures.) So doubling context can cost more RAM than the model itself. A 16GB GPU running an 8B Q4 at 32k context is right at the edge — spill to CPU and speed drops 5–20x. See model-quantization for weight-side savings and nvidia-smi during load tests to measure real usage.

Latency Math

Two distinct phases:

Prompt processing (prefill): parallel across prompt tokens — fast even on CPU, very fast on GPU. Typically thousands of tokens/sec.
Generation (decode): strictly sequential, one token at a time. This dominates for long outputs. Typical local rates: 40–120 tok/s on consumer GPUs, 3–15 tok/s CPU-only.

total ≈ prompt_tokens / prefill_rate + output_tokens / decode_rate

Example: 6,000-token prompt + 300-token answer, GPU at ~2000 tok/s prefill, ~60 tok/s decode: 6000/2000 = 3s prefill + 300/60 = 5s decode ≈ 8s. The same call with only 800 prompt tokens: ~0.4s + 5s ≈ 5.5s — long prompts are cheap-ish, but long outputs are the expensive part. Cap output (num_predict) whenever you can.

Strategy 1: Truncation (Cheapest)

Hard-cut the oldest history or least-relevant text.

def fit_context(messages, max_turns=6):
    # keep system prompt + last N turns; drop middle
    keep_head, keep_tail = messages[:1], messages[-6:]
    return keep_head + keep_tail

Good for chat bots where old turns matter little. Risks: silent loss of earlier constraints ("remember to use JSON") — re-inject critical instructions after trimming.

Strategy 2: Sliding Window With Rolling Summary

Summarize evicted history into a compact "memory so far" block:

summary_prompt = [
    {"role": "system", "content": "Compress this conversation into <=150 tokens of facts, decisions, and open questions."},
    {"role": "user", "content": "\n".join(m["content"] for m in evicted_turns)},
]
rolling_summary += ollama.chat(model="llama3.1:8b", messages=summary_prompt)["message"]["content"]

New context = [system] + [rolling_summary] + recent_turns. Costs one extra small call per eviction cycle but preserves continuity indefinitely. Summarize with the same model or a smaller one; use temperature 0.

Strategy 3: Retrieval Instead of Inclusion

Never stuff a whole corpus into the prompt. Index chunks into a vector DB (qdrant-setup, embeddings-vector-db) and retrieve top-k relevant chunks per query (rag-pipeline). Context stays bounded regardless of corpus size — 3–5 chunks × ~400 tokens beats 50k tokens of raw docs on both accuracy and latency.

Decision guide:

Situation	Strategy
Chat, old turns mostly irrelevant	Truncation
Long-running assistant needing continuity	Rolling summary
Query against large/static documents	Retrieval (RAG)
One-off big file analysis	Chunk + map-reduce summarize per chunk
Everything always relevant, small corpus (<window)	Just include it whole

Practical Examples

Example 1: Measuring Real Token Usage

import ollama

r = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": big_log_dump}],
    options={"num_ctx": 8192},
)
print(f"prompt: {r['prompt_eval_count']} tok, "
      f"output: {r['eval_count']} tok, "
      f"prefill {r['prompt_eval_duration']/1e9:.1f}s, "
      f"decode {r['eval_duration']/1e9:.1f}s")

If prompt_eval_count equals exactly num_ctx - margin, your prompt was truncated — shrink it or raise num_ctx.

Example 2: Map-Reduce Summarization of a Huge File

import ollama

CHUNK = 3000  # tokens-worth of chars, conservatively ~12000 chars
chunks = [text[i:i+CHUNK*4] for i in range(0, len(text), CHUNK*4)]

partials = []
for c in chunks:
    r = ollama.chat(model="llama3.1:8b", temperature=0, messages=[
        {"role": "user", "content": f"Summarize key facts in <=100 tokens:\n{c}"}])
    partials.append(r["message"]["content"])

final = ollama.chat(model="llama3.1:8b", temperature=0, messages=[
    {"role": "user", "content":
     "Combine these section summaries into one summary:\n" + "\n---\n".join(partials)}
])["message"]["content"]

Each map step fits comfortably in a small window no matter the source size.

Example 3: Sizing a New Deployment

Before committing a service, dry-run its worst-case prompt:

Build the largest realistic payload (system + max history + max retrieved chunks + question).
Send once with --verbose; read prompt_eval_count.
Set num_ctx ≈ measured × 1.25 headroom + expected output tokens.
Watch nvidia-smi (or RAM) during the call; if near capacity, reduce num_ctx first, quant second (model-quantization).

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Model "forgets" the system prompt in long chats	Oldest tokens silently truncated past num_ctx	Raise num_ctx; re-inject rules each turn; trim history deliberately
Quality degrades gradually in long conversations	Attention dilution ("lost in the middle") even within window	Sliding window + rolling summary
Request slow only on first turn of a session	Prefill of huge prompt + cold model	Cache/prewarm; cap prompt size; stream output
OOM / model falls back to CPU	num_ctx raised without checking KV cache budget	Lower num_ctx before lowering quant; see memory table above
Output cut off mid-JSON	num_predict default limit hit	Set `options={"num_predict": ...}` generously for structured output
Logs blow up token estimates	Timestamps/UUIDs tokenize densely	Pre-filter logs (grep severity) before sending; truncate each line
Same prompt gives different token counts across models	Different tokenizers	Measure per deployed model, not from generic calculators

Next Steps / Ops Actions

Store what doesn't fit in a vector database instead: embeddings-vector-db, qdrant-setup
Assemble retrieved context into well-delimited prompts: prompt-engineering
Trade context headroom for smaller weights: model-quantization

Sources & Related

External references consulted:

https://github.com/ollama/ollama/blob/main/docs/modelfile.md (num_ctx, num_predict)
https://platform.openai.com/tokenizer (visual tokenizer intuition)
https://arxiv.org/abs/2307.03172 ("Lost in the Middle")

Change Log

2026-08-26

Initial creation by KB build session.