Drift Detection Runbook - Detect, Triage, and Reconcile

Status: Active
Last Updated: 2026-08-26
Category: GitOps - Operations
Prerequisites: gitops, drift-detection
Time: 2 hours
Tags: gitops, drift, runbook, flux, argo, reconciliation, troubleshooting

Summary

An operational runbook for detecting configuration drift between Git and running systems, triaging what caused it, and deciding between reconcile and revert. Covers Flux and Argo CD diff tooling, scripted drift checks, a step-by-step triage flow, and escalation paths.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Run manual and automated diffs against Flux and Argo CD workloads
✅ Script drift checks that fail loudly in CI or cron
✅ Classify drift as benign, intentional-hotfix, or malicious
✅ Choose correctly between reconcile (pull Git to cluster state) and revert (push cluster change back to Git)
✅ Escalate per fogserv.cloud's ticketing-first process

Context / Why This Matters
Detecting Drift
Triage Flow
Reconcile vs Revert
Escalation

Context / Why This Matters

gitops establishes that Git is the source of truth and pull-based reconcilers restore declared state automatically. But reconciliation only covers resources the controller owns. Drift outside controller scope — manual kubectl edit, hand-edited config on hosts, resources created without manifests — silently diverges until something breaks at 3 AM.

This runbook operationalizes the theory in drift-detection: concrete commands, a decision tree for remediation, and escalation rules so any agent or human responds consistently. Every drift incident ends with a retrospective recorded in /kb/, per campsite rule.

Detecting Drift

Flux

# Show resources where live state differs from Git
flux get kustomizations --status-selector ready=false

# Detailed diff for one kustomization
flux diff kustomization apps-prod \
  --path ./clusters/prod/apps

# Force reconcile and observe result
flux reconcile kustomization apps-prod --with-source
flux get kustomizations | grep apps-prod

Flux marks a Kustomization degraded when apply fails or health checks time out. flux diff compares desired vs live without touching anything.

Argo CD

# List applications and their sync status
argocd app list

# Show the actual diff (live vs Git) for one app
argocd app diff my-app

# OutOfSync apps only
argocd app list -o json | jq -r '.[] | select(.status.sync.status=="OutOfSync") | .metadata.name'

Argo CD classifies each resource as Synced or OutOfSync and records diff details including which fields differ. If self-heal is enabled (automated.syncMode + prune + selfHeal), Argo reverts drift automatically within its sync interval — check argocd app history my-app to see whether an unexpected auto-sync fired.

Scripted Drift Sweep (controller-independent)

For hosts, raw manifests, and anything neither controller owns:

#!/usr/bin/env bash
# drift-sweep.sh - compare rendered manifests against live cluster
set -euo pipefail

DIR="${1:-./clusters/prod}"
FAILED=0

for manifest in "$DIR"/*.yaml; do
  name=$(yq '.metadata.name' "$manifest")
  ns=$(yq '.metadata.namespace // "default"' "$manifest")
  kind=$(yq '.kind' "$manifest" | tr 'A-Z' 'a-z')

  # Render live object, strip server-assigned fields, diff
  if ! kubectl get "$kind" "$name" -n "$ns" -o yaml > /tmp/live.yaml 2>/dev/null; then
    echo "MISSING IN CLUSTER: $kind/$name"
    FAILED=1
    continue
  fi
  yq eval 'del(.metadata.managedFields, .metadata.resourceVersion,
        .metadata.uid, .metadata.creationTimestamp,
        .metadata.generation, .status)' /tmp/live.yaml > /tmp/live-clean.yaml
  if ! diff -u "$manifest" /tmp/live-clean.yaml > /tmp/drift.diff; then
    echo "DRIFT: $manifest"
    cat /tmp/drift.diff
    FAILED=1
  fi
done
exit $FAILED

Run it from a scheduled Forgejo Action or a nightly systemd timer (see scheduler-patterns), and alert on nonzero exit via simple-alerts.

Host-Level Drift

For non-Kubernetes servers, compare live config against Ansible-declared state:

ansible-playbook site.yml --check --diff --limit prod-web

Any changed task output is drift. This pairs with ansible-patterns.

Triage Flow

When drift is detected, classify before acting:

1. WHAT changed?
   - Capture the diff immediately (it may auto-heal):
     argocd app diff my-app > /tmp/incident-<ticket>.diff
     kubectl get <resource> -o yaml > /tmp/live-state.yaml

2. WHO changed it?
   - Kubernetes audit logs / managedFields (.metadata.managedFields shows last editor)
   - Shell history on hosts, Forgejo audit trail for pipeline changes
   - Check open incidents: was this an emergency hotfix?

3. WHEN did it change?
   - managedFields timestamps, journald around the window
     journalctl --since "2026-08-25 20:00" --until "2026-08-25 23:00"

4. CLASSIFY:
   [A] Benign      → server-added defaults, status fields, rolling-update noise.
                     Fix: ignore-list the field or add ignoreDifferences.
   [B] Hotfix      → someone fixed prod by hand under pressure.
                     Fix: codify into Git ASAP (see Reconcile vs Revert).
   [C] Unexplained → no owner, no ticket, potentially security-relevant
                     (new port, new RBAC binding, new container image).
                     Fix: treat as incident. Escalate (below).

Rule of thumb: Class C drift on security-relevant objects (RBAC, NetworkPolicy, ingress) is an incident, not housekeeping.

Reconcile vs Revert

Two opposite operations, chosen by which side is wrong:

Situation	Operation	Direction
Live state matches intent; Git is stale	Revert (commit cluster state back to Git)	Cluster → Git
Git matches intent; live state was mutated	Reconcile (let controller reapply Git)	Git → Cluster

Revert (adopt the drift into Git)

# Export the drifted-but-correct live object
kubectl get deployment api -n prod -o yaml > clusters/prod/apps/api.yaml
# Strip runtime fields, then open a PR through normal review
git checkout -b hotfix/adopt-manual-scale-fix clusters/prod/apps/api.yaml
# ... clean, commit, MR to Forgejo per [gitops](gitops) workflow

Never leave a working hotfix uncodified: the next reconcile will erase it.

Reconcile (restore Git's state)

flux reconcile kustomization apps-prod --with-source
# or
argocd app sync my-app

If reconcile fails or loops (controller keeps fighting something), suspect a controller-scope gap or an operator acting on the same object — stop and investigate rather than force-reapplying.

If reconciliation itself caused the problem (bad commit merged), roll the commit back per rollback-strategies and let the reconciler converge to the previous known-good revision — this is exactly the pull-based recovery path described in gitops-principles.

Escalation

P1 (prod broken or suspected compromise): page on-call immediately; open a Forgejo Issue tagged incident; attach the captured diff and timeline. Do not "clean up" evidence before capture.
P2 (unowned drift, no outage): open ticket within one business day; assign owner to either codify or remove the change within the week.
P3 (benign noise): fix the detection (ignoreDifferences, script filters) so alerts stay trustworthy — alert fatigue kills drift programs.

Close every case by updating /kb/problems-solved.md and, for recurring patterns, this article's troubleshooting table.

Practical Examples

Example 1: Nightly drift sweep wired to alerting

# .forgejo/workflows/drift-sweep.yml (excerpt)
schedule:
  - cron: '0 3 * * *'
jobs:
  drift:
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/drift-sweep.sh ./clusters/prod | tee drift-report.txt
      - run: |
          curl -s -X POST "$ALERT_URL" -d @- <<EOF
          {"text": "Drift sweep exit=$?: $(wc -l < drift-report.txt) diff lines"}
          EOF
        if: failure()

Example 2: Adopting an emergency hotfix

Incident: responder scaled api replicas 3→6 by hand during a traffic spike; next sweep flags drift. Correct action is revert: commit replicas: 6 (or better, a properly sized HPA) to Git, merge, reconcile — the manual change becomes durable and auditable instead of being silently reverted later.

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Permanent OutOfSync on fields you never set	Defaults/status injected by admission controllers	Add `ignoreDifferences` (Argo) or post-render patches; never delete status fields from Git
Drift keeps coming back after reconcile	Something else writes the same object (operator, CronJob)	Find the writer via managedFields/audit logs before reconciling again
`kubectl diff` clean but sweep reports drift	Server-side fields not stripped in the comparison	Strip managedFields/resourceVersion/status like the sweep script does
Auto-heal erased an emergency fix	Hotfix never committed	Adopt first, then reconcile — see Revert section
Diff tools show nothing but behavior differs	ConfigMap content hashed into pod spec vs mounted volume semantics	Restart pods after ConfigMap changes; verify with `kubectl exec ... cat`

Next Steps / Ops Actions

Schedule the sweep script and wire failures to simple-alerts.
Prevent drift at admission time with policy checks: policy-as-code.
Review branch hygiene that reduces accidental drift commits: merge-strategies.
After every real incident, record the retrospective in /kb/lessons-learned.md.

Sources & Related

External references consulted:

Change Log

2026-08-26

Initial creation: detection commands (Flux/Argo/scripts), triage classification, reconcile-vs-revert decision table, escalation tiers.