fogserv.cloud fogserv.cloud

Prompt Engineering - Practical Prompting for Local Models

Status: Active
Last Updated: 2026-08-26
Category: AI/ML - Phase 2: Local LLMs
Prerequisites: llm-introduction, ollama-cli-basics
Time: 2-3 hours
Tags: prompting, system-prompts, few-shot, evaluation, llm

Summary

A practical, tool-agnostic guide to getting reliable output from local models: role-setting system prompts, few-shot examples, delimiters, explicit output constraints, and lightweight eval loops so improvements are measured rather than vibes.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Context / Why This Matters

After python-llm-integration connects your code to Ollama, the next quality lever isn't infrastructure — it's the prompt. Local 8B-class models are noticeably less forgiving than frontier cloud models: they follow subtle instructions less reliably, drift out of format sooner, and hallucinate more when context is ambiguous. Good prompting recovers most of that gap for free, and a repeatable eval loop tells you whether a prompt tweak actually helped.

Prompt engineering complements, but does not replace, the mechanical constraints covered in context-and-tokens (how much fits) and model-selection-guide (which model can do the task at all).

Implementation / Core Content

Anatomy of a Reliable Prompt

Split every request into four parts, in this order:

  1. System prompt — identity, scope, hard rules. Set once per conversation/session.
  2. Task instruction — one imperative sentence: what to produce.
  3. Context/data — clearly delimited input the model must work on.
  4. Output contract — exact format, length, and what to do when uncertain.
SYSTEM: You are a senior Linux sysadmin assistant for a homelab.
Rules:
- Answer only about Linux, Docker, networking, and backups.
- If unsure, say "I don't know" — never invent command flags.
- Be terse. No pleasantries.

USER TASK: Classify each log line below as severity high/med/low.

LOGS:
<<<LOGS
Aug 25 03:14:01 nas CRON[512]: backup.sh exited with code 1
Aug 25 04:00:00 nas systemd[1]: apt-daily.service succeeded
LOGS>>>

OUTPUT: JSON array of {"line": <first 20 chars>, "severity": "high|med|low"}.
Return [] if no lines qualify.

System Prompts That Work on Small Models

Few-Shot Examples

Show 2–5 input→output pairs before the real input. This is the single most effective technique for format compliance on local models.

Classify the ticket priority. Examples:

Ticket: "Production DB down, site offline"
Priority: P1

Ticket: "Typo on the About page"
Priority: P3

Ticket: "Search sometimes returns stale results"
Priority: P2

Ticket: "{new ticket}"
Priority:

Guidelines:

Delimiters

Delimiters separate "instructions to me" from "data to process". Without them, data containing instruction-like text ("ignore previous...") derails the model.

Use XML-ish tags or triple-brackets consistently:

This pairs with RAG pipelines (rag-pipeline): wrap each retrieved chunk individually and number them so the model can cite which chunk it used.

Output Constraints

For machine-consumed output:

  1. State the exact format, including field names and types.
  2. Enforce mechanically with JSON mode / schema-constrained decoding (see python-llm-integration) — prompting alone is not a guarantee.
  3. Specify the empty/degenerate case: "If nothing applies, return []." Otherwise models invent plausible-looking entries.
  4. Ban preamble: "Output ONLY the JSON. No markdown fences, no explanation." (Small models love wrapping JSON in ```json fences.)

Length constraints: give budgets in sentences or items, not words ("exactly 3 bullet points" is far more reliable than "about 50 words").

Chain-of-thought note: "think step by step" helps reasoning tasks but bloats output and slows parsing. If you use it, ask for a final line delimited as ANSWER: ... and parse only that.

Eval Loops: Stop Guessing

A prompt change is only real if it moves a measured number. Minimum viable eval:

"""eval_prompt.py — score a prompt against labeled cases."""
import json, sys, ollama

CASES = [
    # (input, expected)
    ("Backup failed with exit code 1 at 03:14", "high"),
    ("Certificate renews in 25 days",           "low"),
    ("Disk /var at 91% capacity",               "med"),
    ("RAID array degraded, 1 drive missing",    "high"),
    ("NTP drift of 0.4s detected",              "low"),
]

PROMPT = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_PROMPT

def classify(text):
    r = ollama.chat(model="llama3.1:8b", format="json", temperature=0,
        messages=[{"role": "user", "content":
                   PROMPT.replace("{INPUT}", text)}])
    return json.loads(r["message"]["content"])["severity"].lower()

correct = 0
for text, expected in CASES:
    got = classify(text)
    ok = got == expected
    correct += ok
    print(f"{'PASS' if ok else 'FAIL'}  expected={expected:<4} got={got:<6} {text[:45]}")
print(f"\nScore: {correct}/{len(CASES)}")

Run it before and after every prompt edit. Rules for keeping the eval honest:

A Worked Before/After

Weak prompt: "Summarize this log file." → rambling prose, inconsistent focus, mentions irrelevant DEBUG lines.

Improved prompt: system rules ("only mention ERROR/WARN; max 3 bullets; name the failing unit; say 'no issues' if clean"), delimited log body, output contract ("JSON: {summary: str, failing_units: [str]}"). On our informal suite this moved format-validity from roughly half of runs to effectively all runs, and made the output directly consumable by the triage script in python-llm-integration.

Practical Examples

Example 1: Extraction With Refusal Path

SYSTEM: Extract server hostnames from the text. Only hostnames that
appear verbatim. Output JSON {"hosts": [...]}. Empty array if none.
Never guess or complete partial names.

USER:
<text>
The deploy ran on web01 and db02; staging box stg-web01 was skipped.
Contact alice@internal.example for access.
</text>

EXPECTED: {"hosts": ["web01", "db02", "stg-web01"]}

The email address is a trap — models without the "never guess" rule tend to include alice@internal.example.

Example 2: Self-Check Pass

Add a second cheap call that verifies the first call's output:

check = ollama.chat(model="llama3.1:8b", format="json", messages=[
    {"role": "user", "content":
     f"Every hostname below must appear VERBATIM in the source text.\n"
     f"SOURCE:\n<source>\n{source}\n</source>\n"
     f"CLAIMED: {json.dumps(result)}\n"
     "Reply {\"valid\": true} or {\"valid\": false, \"bad\": [...]}"}
])

Two small-model calls often beat one big prompt for accuracy-per-token.

Example 3: Iterating With the Eval Harness

python eval_prompt.py "$(cat prompt_v1.txt)"   # Score: 3/5
# edit: added one edge-case few-shot example
python eval_prompt.py "$(cat prompt_v2.txt)"   # Score: 5/5
# verify on held-out set once before shipping

Troubleshooting & Common Pitfalls

Problem Cause Fix
Model answers outside its role ("as an AI...") Weak/no system prompt State role + scope rules explicitly in system message
Format breaks after N requests in a chat Instruction fades over long history Repeat output contract in latest user turn; trim history
Model follows text embedded in the data No delimiters Wrap data in tags; instruct to treat contents as inert
Invented facts/flags No permission to be uncertain Add "say I don't know"; lower temperature; provide sources
JSON wrapped in markdown fences Training bias toward fenced code "Output ONLY JSON, no fences"; strip fences defensively in code
Few-shot hurts: outputs copy examples verbatim Examples too templated / real input too similar to one example Vary example phrasing; add diverse classes; reduce to 2-3 examples
Prompt works on GPT-class cloud model, fails locally Local 8B model needs more explicitness More few-shots, shorter sentences, one task per prompt
Eval scores fluctuate between identical runs temperature > 0, non-deterministic backend Set temperature 0 and fixed seed option during evals

Next Steps / Ops Actions

Sources & Related

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse