Documentation as Code - Runbooks That Stay True

Status: Active
Last Updated: 2026-08-14
Category: Infrastructure - Phase 1: Manual Infrastructure
Prerequisites: manual-server-setup
Time: 2 hours
Tags: documentation, runbooks, markdown, version-control, operations, drift

Summary

Treat operational documentation like source code: written in Markdown, stored in Git, reviewed through pull requests, and kept in sync with the systems it describes. Learn runbook structure, doc-as-code workflows, and the discipline that stops your wiki from rotting into fiction.

🎯 What You'll Learn

By the end of this article, you'll be able to:

• ✅ Explain why wikis rot and repos don't
• ✅ Structure a runbook for 3 AM use
• ✅ Write Markdown docs with metadata, verification steps, and rollback sections
• ✅ Keep documentation in the same repo as the code/config it documents
• ✅ Use review workflows to keep docs accurate
• ✅ Automate doc validation where possible
• ✅ Recognize and fix documentation drift

📉 Why Documentation Dies

Every team has experienced this lifecycle:

Week 1:  "We need docs!" → beautiful wiki created
Month 3: Server rebuilt, IP changed — wiki not updated
Year 1:  Nobody trusts the wiki anymore
Year 2:  Tribal knowledge, bus factor = 1

The root cause: documentation separated from change. When docs live in a different place than code and config, every change has two steps — and under pressure, step two gets skipped.

The fix: make documentation part of the change:

Wiki Model	Docs-as-Code Model
Docs live in a separate wiki	Docs live next to code/config
Anyone can edit silently	Changes go through PRs
No history of why	Git blame + commit messages
Staleness invisible	Stale doc = failed CI check
"Update the wiki" = optional	"Docs updated" = part of definition of done

💡 The KB you are reading right now follows this exact model: Markdown files, change logs, ticket references, and a maintenance mandate ("Documentation is Memory").

---

🏗️ Where Docs Live

Three placements, each with a purpose:

repo/
├── README.md                  ← how to run/develop THIS project
├── docs/
│   ├── architecture.md        ← design decisions, diagrams
│   └── runbooks/              ← operational procedures
├── infra/
│   └── ansible/
│       └── roles/webserver/
│           └── README.md      ← role-specific docs travel with the role
└── CHANGELOG.md               ← what changed when

Rules of thumb:

1. Co-locate by default — a role's docs belong in the role directory; they can't be forgotten when the role moves
2. One repo-level docs/ for cross-cutting material (architecture, on-call guides)
3. Runbooks that span systems get their own repo or top-level section — but still in Git

---

📕 Runbook Anatomy

A runbook is written to be executed by a stressed human (or an agent) at an ungodly hour. Optimize for that reader.

Template

# Runbook: <Service> — <Procedure Name>

**Owner**: platform-team  
**Last verified**: 2026-08-14  
**Estimated time**: 10 minutes  
**Risk level**: medium (brief API latency during drain)

## Symptoms
- Grafana panel "API p99" > 2s for 5+ minutes
- Alert: `HighLatency` firing

## Before You Start
- [ ] Confirm access: `ssh ops@api01` works
- [ ] Check current status page / recent deploys:
      git -C /opt/app log --oneline -5

## Procedure
### 1. Verify it's actually the app (not upstream DB)
    curl -sf http://localhost:3000/healthz || echo "APP DOWN"
    # If healthz fails but DB checks pass below, continue.
    pg_isready -h db01.internal

### 2. Restart the app service
    sudo systemctl restart app.service

### 3. Watch recovery
    journalctl -u app -f --since "1 min ago"
    curl -w '%{http_code} %{time_total}\n' -so /dev/null http://localhost:3000/healthz

## Verification
- [ ] `healthz` returns 200 in < 200ms
- [ ] Grafana p99 back under 500ms
- [ ] Alert auto-resolves within 10 min

## Rollback
If restart made things worse:
    sudo systemctl stop app.service
    # previous container image retained:
    docker tag app:previous app:active && sudo systemctl start app.service

## Escalation
- No improvement after 2 restarts → page @platform-oncall
- Suspected data issue → STOP, escalate to @dba immediately

## Post-Incident
- File incident ticket referencing this runbook
- Update this runbook if any step was wrong or missing

Why Each Section Exists

Section	Purpose	Failure mode if missing
Owner	Who maintains/wrote it	Nobody fixes stale content
Last verified	Trust signal	Reader assumes accuracy
Symptoms	Confirms you're in the right runbook	Wrong procedure applied
Before You Start	Cheap pre-flight checks	Discover missing access mid-procedure
Numbered procedure	Copy-pasteable commands	Improvisation under stress
Verification	Defines "done"	Declared fixed while broken
Rollback	Escape hatch	Panic escalation
Escalation	When to stop solo	Heroics at 4 AM

⚠️ The most important field is Last verified. A runbook executed successfully should have its date bumped in the same PR/commit that closes the incident. Unverified docs decay silently.

---

✍️ Writing Rules

1. Commands over prose. sudo systemctl restart app beats "restart the application service."

2. Expected output after commands.

   systemctl is-active nginx
   

   What Happens: Prints active and exits 0. Anything else means the step failed — stop and diagnose rather than proceeding.

3. Explain the why, once. One line per non-obvious command: # drain first so we don't drop in-flight uploads.

4. No tribal shorthand. "Do the usual cert dance" is not documentation. List the actual steps.

5. Idempotent-friendly. Prefer steps safe to re-run from the beginning — responders rarely know exactly where a previous attempt died.

Anti-patterns

❌ "SSH into the box"            → which box? which user?
❌ Screenshots of terminals      → not copy-pasteable, rots visually
❌ "Contact Dave"                → Dave left in March
❌ Steps without expected output → responder can't tell success from failure
❌ Hidden prerequisites          → "Step 7 fails because Step 0 was never written down"

---

🔁 Keeping Docs In Sync With Reality

Sync is a process problem, solved with three mechanisms:

1. Same-PR Rule

Definition of done includes docs:

Change type              → Doc updated in same PR
Add config flag          → README option table
New Ansible role         → Role README + inventory example
Schema migration         → schema-overview.md entry
Incident resolved        → Runbook corrected + last-verified bumped

Reviewers enforce it like tests: "LGTM once the docs block is addressed."

2. Verification Hooks

Make staleness detectable where possible:

# CI check: every runbook must have required sections
#!/usr/bin/env bash
# scripts/lint-runbooks.sh
set -euo pipefail
fail=0
for f in runbooks/*.md; do
  for section in "Symptoms" "Verification" "Rollback"; do
    grep -q "^## $section" "$f" || { echo "MISSING '$section' in $f"; fail=1; }
  done
  grep -q "Last verified" "$f" || { echo "MISSING 'Last verified' in $f"; fail=1; }
done
exit $fail

What Happens: Exits nonzero if any runbook lacks required sections, failing the pipeline. Documentation now has teeth.

3. Drift Checks Against Reality

Wherever truth is machine-readable, compare:

# Example: documented firewall ports must match reality
grep -oP 'allow \K[0-9]+' runbooks/firewall.md | sort -u > /tmp/documented.txt
sudo ufw status numbered | grep -oP '\d+/tcp' | cut -d/ -f1 | sort -u > /tmp/actual.txt
diff /tmp/documented.txt /tmp/actual.txt || echo "⚠️ Firewall docs out of sync!"

The endgame: infrastructure defined as code makes many docs generated. Your Terraform module's variable table comes from variables.tf; your Ansible defaults table from defaults/main.yml. Tools like terraform-docs automate exactly this:

terraform-docs markdown table ./modules/vm > modules/vm/README.md

---

🗂️ Metadata Convention

This KB's standard header doubles as doc metadata — adopt something similar everywhere:

# Title - Subtitle
**Status**: Active | Draft | Deprecated  
**Last Updated**: YYYY-MM-DD  
**Prerequisites**: links  
**Tags**: searchable, keywords

Deprecation rule (from the KB mandate): never delete — mark Status: Deprecated, note why, link to the replacement. Old procedures sometimes need resurrection during incidents.

---

🧪 Exercise: Convert Yesterday's Notes

Take the SETUP-NOTES.md from manual-server-setup and promote it to a real runbook:

1. Add the full header template (owner, last verified, risk)
2. Restructure into Symptoms / Before You Start / Procedure / Verification / Rollback / Escalation
3. Add expected output after every command
4. Put it in Git: git init && git add runbooks/new-server.md && git commit -m "runbook: provision new web server"
5. Add the lint script above to a make lint target and run it

You now own a runbook instead of notes. Next lesson explains why you'll soon throw away the manual steps entirely — and why the runbook remains valuable even then.

---

🛠️ Common Issues

Symptom	Cause	Fix
Docs updated but nobody notices	No review path for docs	Route doc changes through same PR/review as code
Runbooks diverge across repos	Duplication	Single source of truth + cross-links (like this KB)
"Docs" folder is a dumping ground	No structure conventions	Enforce templates via lint script + review
Generated docs committed manually	Forgot regeneration step	Regenerate in CI and fail on diff (git diff --exit-code)
Everyone writes differently	No style guide	Adopt the writing rules above; reference them in CONTRIBUTING

---

🔗 Related

• Previous: manual-server-setup — where these notes come from
• Next: why-infrastructure-as-code — automating what the runbook describes
• gitops-infrastructure — Git as source of truth, taken to its conclusion
• testing-infrastructure — CI checks that keep both code and docs honest
• kb/gitops/gitops — sibling course on GitOps workflows

📚 Sources & Related

• Google SRE Book, Ch. 6 — "Postmortem Culture" and runbook guidance
• terraform-docs documentation
• KB: README store layout — this site's docs-as-code standard