Rollback Procedures - Recovering From Bad Deployments

Status: Active
Last Updated: 2026-08-26
Category: CI/CD - Production
Prerequisites: woodpecker-first-pipeline, deployment-automation, gitops-pipelines
Time: 2-3 hours
Tags: rollback, deployment, docker, git-revert, blue-green, incident-response, woodpecker

Summary

Every deployment will eventually fail — a bad migration, a config typo, a dependency regression. This article covers the rollback playbook for each deployment style used in this stack: Docker Compose image pinning, k3s rollout undo, and Git-revert-driven GitOps rollbacks, plus what not to forget (database migrations and caches don't roll back with your containers).

Context / Why This Matters

deployment-automation makes shipping fast; rollback procedures make fast shipping survivable. The goal is that recovering from a bad release is a single, rehearsed command — not an improvisation at 2 AM while users hit errors. A rollback you've never practiced is a rumor.

Two rules drive everything below:

1. Roll forward or roll back, never "hotfix on prod" — untracked manual changes make the next deploy land on a different system than you think.
2. Anything immutable can be rolled back safely; anything stateful needs its own plan.

Implementation / Core Content

Rule Zero: Know What Changed

Before touching anything, identify the last-known-good version:

# What's running right now?
docker ps --format '{{.Names}}\t{{.Image}}'
kubectl get deployments -o wide          # k3s: shows IMAGE per deployment

# What was deployed before?
docker images --format '{{.Repository}}:{{.Tag}} {{.CreatedAt}}' | head
kubectl rollout history deployment/fogserv-cloud -n fogserv

If you tag images with both latest and the git SHA (ghcr.io/shire/fogserv:abc1234), rollback is trivially: run the previous SHA.

Scenario 1 — Docker Compose: Pin the Previous Image

Compose deployments roll back by re-pointing at the previous image tag:

# docker-compose.yml — tags matter: :latest cannot be rolled back to anything
services:
  app:
    image: ghcr.io/shire/fogserv:v2026.08.25   # versioned, not :latest

# Roll back = edit tag to previous release, then recreate
docker compose pull && docker compose up -d
docker compose logs -f app        # watch startup for migration errors

Keep the last 3-5 images on the host (docker image prune sparingly) so the old version is still local if the registry is unreachable during an incident.

Scenario 2 — k3s: rollout undo

Kubernetes keeps revision history per Deployment automatically:

# See revisions
kubectl rollout history deployment/fogserv-cloud -n fogserv

# One-command rollback to previous revision
kubectl rollout undo deployment/fogserv-cloud -n fogserv

# Or target a specific revision
kubectl rollout history deployment/fogserv-cloud -n fogserv --revision=3
kubectl rollout undo deployment/fogserv-cloud -n fogserv --to-revision=3

# Watch it land
kubectl rollout status deployment/fogserv-cloud -n fogserv

Caveat: --to-revision only works while that RevisionHistoryLimit entry survives (default 10). Don't set revisionHistoryLimit: 0.

Scenario 3 — GitOps: Revert the Commit, Let the Reconciler Do It

In a GitOps flow (gitops-pipelines), production state matches main. The safest rollback is therefore a new commit, not a manual cluster change (manual changes get reverted by the reconciler — that's its job):

git revert <bad-commit-sha>     # not reset — keep history honest
git push                        # pipeline deploys the reverted manifest

This trades speed for correctness: recovery takes one pipeline run (~minutes), and audit history stays intact. For emergencies where minutes matter, do the k3s rollout undo first to stop the bleeding, then immediately push the revert so Git and reality converge again.

The Part Containers Can't Undo: Migrations

Prisma migrations are forward-only in practice. Rolling the app back without handling the schema causes exactly the outage you're trying to end:

Migration type	Rollback risk	Play
Additive (new nullable column/table)	Low	Safe: old code ignores new columns
Column drop / rename / type change	High	Old code breaks against new schema
Data transformation	Highest	Irreversible without a backup

Playbook:

1. Prefer additive migrations (prisma-migrations-guide) so N and N+1 app versions both run against schema N+1.
2. Before risky migrations: prisma migrate diff review + backup (sqlite3 dev.db ".backup ..." / pg_dump).
3. If a bad migration shipped: restore DB from backup and roll back the app together, during a brief maintenance window — see backup-recovery-drill.

Practical Examples

Full incident walkthrough (k3s)

# 14:02 — alerts fire: 500s after deploy of v2026.08.26
kubectl get pods -n fogserv                       # CrashLoopBackOff on new replica
kubectl logs deploy/fogserv-cloud -n fogserv --previous   # read the crash

# 14:04 — stop the bleeding
kubectl rollout undo deployment/fogserv-cloud -n fogserv
kubectl rollout status deployment/fogserv-cloud -n fogserv
curl -fsS https://fogserv.cloud/api/health        # confirm healthy

# 14:10 — converge Git so the reconciler doesn't re-deploy the bad commit
git revert abc1234 && git push

# 14:15 — post-mortem input
kubectl get events -n fogserv --sort-by=.lastTimestamp | tail -20

Woodpecker job that snapshots the current tag before every deploy

steps:
  pre-deploy-snapshot:
    image: alpine:3.20
    commands:
      - echo "Deploying ${CI_COMMIT_SHA:0:8}; previous live tag:"
      - ssh deploy@host "docker inspect --format '{{.Config.Image}}' app || true"

Recording the previous tag in the build log means the rollback target is always written down before you need it.

Common Pitfalls & Troubleshooting

• :latest everywhere — you can't roll back to "the previous latest"; always ship immutable tags (SHA or semver).
• Rolling back the container but not the DB — old code + new schema = subtle failures; check migration compatibility before reverting.
• Manual prod fixes that survive the rollback — they vanish on next deploy and nobody knows why prod behaved differently; encode any emergency change as a commit immediately.
• rollout undo no-op — happens when the bad change came via ConfigMap/Secret rather than pod spec; revert those objects too.
• Cache poisoning — a rolled-back SSR app may serve stale cached pages built by the bad code; clear app caches as part of the procedure.
• Never-practiced runbooks — schedule a quarterly rollback drill alongside the backup-recovery-drill.

Next Steps / Ops Actions

• Add a rollback: section to each service's deploy doc naming: previous-tag source, one command, and DB-compat check.
• Set revisionHistoryLimit: 5 explicitly on k3s Deployments.
• Pair this article with ci-monitoring so failed deploys page you instead of waiting for user reports.

Sources & Related Articles

• Kubernetes docs: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment
• Docker Compose docs: https://docs.docker.com/compose/
• Related: deployment-automation, gitops-pipelines, branch-protection, ../infrastructure/disaster-recovery, ../databases/prisma-migrations-guide

Change Log

2026-08-26

• Initial creation (kb-build-plan Task 1 completion pass)