fogserv.cloud fogserv.cloud

Rollback Strategies - Always Have a Way Back

Status: Active
Last Updated: 2026-08-14
Category: Migrations - Phase 1: Planning & Assessment
Prerequisites: migration-planning, migration-risks
Time: 2 hours
Tags: migration, rollback, parallel-running, blue-green, recovery, plan-b

Summary

A rollback plan you haven't designed, documented, and rehearsed isn't a plan — it's a hope. This guide covers when to trigger a rollback, concrete procedures per service type, parallel-running patterns that make rollbacks cheap, and how to test your rollback before production forces the question.

🎯 What You'll Learn

By the end of this guide, you'll be able to:

Table of Contents

  1. The Golden Rule of Rollbacks
  2. When to Roll Back: Triggers
  3. Rollback Strategy Patterns
  4. Parallel Running
  5. Rollback Procedures by Service
  6. Testing Your Rollback
  7. The Rollback Runbook Template
  8. During an Actual Rollback
  9. After the Rollback
  10. Related Lessons

The Golden Rule of Rollbacks

Never destroy the old system until the new system has proven itself for longer than your longest failure-detection cycle.

Concretely:

Failure type Detection time Minimum old-system retention
Obvious outage Minutes Days
Subtle integration breakage Hours–days 2 weeks
Slow data corruption Weeks 90 days
"We need that old export format" Months Archive forever (it's cheap)

Storage is the cheapest thing you own. Reputation lost from unrecoverable data is not.

The practical implementation: the old service gets frozen, not deleted — read-only, still backed up, VM paused or scaled to minimum — until its decommission ticket clears a waiting period everyone agreed to in advance (see decommissioning-commercial).

When to Roll Back: Triggers

Deciding "should we roll back?" during an incident, under pressure, with sunk-cost feelings, produces bad decisions. Decide beforehand with objective triggers.

Trigger Design

Write these into the migration plan; anyone on the team can invoke them:

AUTOMATIC ROLLBACK if ANY of:
- T+30min: login success rate < 95% on new system
- T+2h: any data-integrity check fails (counts, checksums)
- T+24h: P0/P1 incident attributable to migration without fix in sight
- Users report losing work (any confirmed instance)

ROLLBACK REVIEW (go/no-go meeting) if ANY of:
- Error rate elevated 2x baseline for > 1 hour
- CI pipeline green rate dropped below 80%
- Two or more "annoying but working" complaints from same feature area

NEVER roll back for:
- Cosmetic differences (documented, scheduled for follow-up)
- Performance within 20% of baseline pending tuning

Two properties matter: triggers are measurable (a number, not a vibe) and time-boxed ("within X hours of cutover"). A trigger like "if it feels wrong" always resolves to "we'll push through" at 1 AM.

Who Calls It

One named person owns the go/no-go. Not a committee. If they're unreachable, the second named person decides. Write both names down.

Rollback Strategy Patterns

You have three options when things go wrong. Know which one each migration supports before starting.

Pattern 1: Fail-Back (switch traffic to old system)

Old system still warm and current-ish → point users/applications back.

Requirements:                    Cost: minutes to hours
- Old system frozen, not dead    Risk: LOW
- Data written to new system     Works best when:
  since cutover must be          - Short validation period
  merged back or discarded       - Writes can be replayed/lost acceptably
- Documented switch procedure

Critical detail: what happens to writes made on the new system? Options:

  1. Accept loss — only valid if the window was short and writes were trivial.
  2. Replay forward — export new-system changes and import into old (works for git pushes, issue comments).
  3. Bidirectional sync — expensive, error-prone, last resort.
# Example: failing back git hosting after 1 day on Forgejo
# 1. Freeze pushes on Forgejo
# 2. Mirror everything pushed since cutover back to GitHub
for repo in $(cat repos.txt); do
  cd /tmp && rm -rf $repo
  git clone --mirror "git@git.shire.one:fogserv/$repo.git"
  cd $repo.git
  git push --mirror "git@github.com:fogserv/$repo.git"
done
# 3. Repoint developer remotes / DNS
# 4. Unfreeze GitHub

Pattern 2: Revert-in-Place (undo the change on the new system)

The new system stays, but the specific breaking change is undone — config edit, version downgrade, schema restore.

Use when: the platform is fine; one change broke it.
Example: Nextcloud migration succeeded, but a sharing setting
         breaks client sync → revert that setting, don't abandon Nextcloud.
Requires: knowing exactly WHICH change broke it (change log discipline!),
          and the ability to undo it (config backup taken pre-change).

Pattern 3: Forward Fix

Stay on the new system and fix the problem in place.

Use when: rolling back is MORE dangerous than fixing forward.
Typical cases:
- Data has already been transformed irreversibly on the new system
- Old system capability no longer exists (vendor contract lapsed)
- The bug is understood and a fix is < 1 hour away
Risk: this is where "we'll just push through" hides when it's a bad idea.
Guardrail: forward-fix needs a TIME BOX. Miss the box → fail-back.

Choosing: Decision Table

Situation Strategy
Validation window < 48h, old system untouched Fail-back
One config/version change broke an otherwise-good migration Revert-in-place
> 1 week in, deep data written, root cause known Forward fix (time-boxed)
> 1 week in, deep data written, root cause UNKNOWN Fail-back if possible; else freeze scope + all-hands fix
Data integrity in doubt Fail-back immediately, investigate after

Parallel Running

Parallel running is how you make fail-back cost ~zero. Both systems live; users migrate gradually.

Pattern A: Read/Write Split by User Group

Week 1: Team A (canary) works on NEW system
        Everyone else on OLD system
Week 2: Teams A+B on NEW, rest on OLD
Week 3: All on NEW, OLD frozen read-only

Works well for: file sync, wikis, issue trackers — anything per-user/per-team.

Gotcha: cross-team shared artifacts must exist on both sides during overlap, or collaboration between groups breaks. Decide whether the source of truth per artifact is old or new — write it down.

Pattern B: Write-Behind Mirroring

All writes go to OLD (source of truth); a replication job keeps NEW up to date:

# Every 15 min during transition:
rclone sync commercial-saas: selfhosted-target: \
  --update --log-file /var/log/migration-sync.log

Cutover = flip which side is authoritative. Rollback = flip back. Data divergence limited to one sync interval.

Pattern C: Blue-Green with Proxy Fronting

Both stacks run simultaneously behind Traefik/nginx; a routing switch moves traffic instantly:

# Traefik dynamic config — the entire cutover AND rollback is this one edit
http:
  routers:
    forgejo:
      rule: "Host(`git.shire.one`)"
      service: forgejo-green      # ← change to forgejo-blue to roll back
  services:
    forgejo-green:
      loadBalancer:
        servers: [{ url: "http://10.0.10.20:3000" }]   # new Forgejo
    forgejo-blue:
      loadBalancer:
        servers: [{ url: "http://10.0.10.19:3000" }]   # old system

Rollback latency: seconds. This is the gold standard when both sides speak the same protocol (git SSH/HTTP, S3 API, WebDAV).

Pattern D: Client-Side Dual Commit (CI/webhooks)

For integrations that can't be proxied, register webhooks/deploy keys on BOTH systems during transition and disable the unused side explicitly rather than deleting:

# Inventory integrations before cutover (from planning phase)
gh api repos/fogserv/widget/hooks   # list webhooks on GitHub
# Recreate each on Forgejo; keep list in migration doc with owner + purpose

Rollback Procedures by Service

Git Hosting (GitHub/GitLab → Forgejo)

  1. Announce freeze; block pushes on new host (chmod 000 on repos or admin setting)
  2. Mirror-push all new-host refs back to origin host (script above)
  3. Flip DNS/proxy (Host(...) router) to old host
  4. Restore webhooks/CI on old host (kept registered, just disabled)
  5. Notify developers: remotes unchanged if DNS-fronted; otherwise send git remote set-url one-liner
  6. Verify: push test commit, open test PR, pipeline fires

Container Registry (→ Harbor)

Rollback is usually unnecessary because registries degrade gracefully — but if Harbor misbehaves:

  1. Point CI image-push/pull env vars back to old registry (values kept in secret store)
  2. Re-tag any images that only exist on Harbor: skopeo copy harbor.example/proj/img:tag docker.io/fogserv/img:tag
  3. Resume pipelines; verify a build end-to-end

Object Storage (S3 → MinIO)

  1. Applications' endpoint/bucket/creds env vars revert to AWS values (dotenvx makes this a one-line profile swap)
  2. Objects written to MinIO since cutover: rclone copy minio-new:bucket s3-old:bucket --update
  3. Verify application smoke tests against restored endpoint
  4. Keep MinIO running for retry after root cause

File Sync (Dropbox/Drive → Nextcloud)

Hardest to roll back cleanly because clients hold local state:

  1. Do NOT delete anything server-side while clients are connected mid-flight
  2. Re-enable old sync client per user group; let it re-establish state
  3. Files created in Nextcloud during trial: export manually (they're small in practice)
  4. Communicate clearly: which folder is authoritative RIGHT NOW

DNS

See dedicated lesson: dns-migration — rollback is raising TTLs and repointing records; propagation delay means you decide FAST.

Testing Your Rollback

An untested rollback is theater. Test it in staging:

Game-day script (staging, quarterly or before major migrations):

1. Build staging copies of old + new systems with realistic data
2. Perform the full migration runbook against staging
3. INJECT a failure: e.g., corrupt a database table, break a webhook,
   delete a storage bucket policy
4. Start a timer. Execute the rollback runbook from the document ONLY
   (no memory, no improvising — that's the point).
5. Stop timer. Score:
   - Did we hit the documented steps? Where did the doc lie?
   - How long did detection take?
   - Was data written post-cutover preserved or lost? Expected?
6. Update the runbook with every gap found.
7. Repeat until rollback completes cleanly inside the target window.

Success metric: rollback executes in ≤ the time you promised stakeholders, using only the written runbook, by someone who didn't write it.

The Rollback Runbook Template

Every migration plan carries this section, filled in BEFORE cutover:

## Rollback Runbook: <Migration>

**Decision owner**: Alex (primary), Sam (backup)
**Max fail-back window**: 72h after cutover (while mirror-fresh)

### Triggers (any one suffices)
- [ ] Data-integrity check failure at T+validation
- [ ] Login/auth success < 95% sustained 30 min
- [ ] Confirmed user data loss

### Pre-staged assets (verify ALL exist before cutover)
- [ ] Old system frozen & reachable: http://10.0.10.19 (VM paused, NOT deleted)
- [ ] Post-cutover delta sync job ready: /opt/migration/delta-sync.sh
- [ ] Traefik router file staged: /etc/traefik/dynamic/rollback.yml
- [ ] Integration inventory w/ enable-on-old instructions: migrations/integrations.md
- [ ] Communication drafts (below)

### Steps
1. Invoke freeze on new system:  forgejo-cli admin freeze  (or documented equivalent)
2. Sync delta:                   /opt/migration/delta-sync.sh && verify counts
3. Switch traffic:               cp rollback.yml /etc/traefik/dynamic/ && curl -s localhost:8080/api/rawdata | jq '.routers'
4. Re-enable integrations:       follow integrations.md § "restore on old"
5. Smoke test checklist:         [login ✓] [push ✓] [pipeline ✓] [clone ✓]
6. Send comms draft B (rolled back)
7. Open post-mortem ticket

### Communication drafts
A (success): "...migration complete, here's what changed..."
B (rollback): "...we've temporarily returned to <old> while we resolve <X>.
No action needed from you. Next update at <time>. Work done today is safe."

Note step 7 and draft B are pre-written. Nobody writes good incident comms mid-panic.

During an Actual Rollback

If the day comes:

  1. Declare it early. Half-rolling-back is worse than either option. Once a trigger fires, execute.
  2. Freeze first, move second. Stop writes on the broken side before syncing/moving data.
  3. Narrate in a channel/ticket. Timestamped notes = free post-mortem material.
  4. Don't debug and roll back simultaneously. If you're rolling back, roll back. Debugging happens after stability.
  5. Verify the rollback landed with the same smoke tests you used for cutover — symmetric validation.

After the Rollback

A failed migration with a clean rollback is a successful rehearsal. Most teams' second attempt succeeds.

Common Gotchas

"We'll keep the old account active" ≠ rollback capability

An old Dropbox account you stopped paying for, or a GitHub org where members were removed, cannot absorb a fail-back. Frozen means: paid, accessible, credentials valid.

Certificates expire during parallel running

That blue stack behind Traefik? Its TLS cert renewal must also work, or month-two rollback fails on a certificate error. Monitor cert expiry on BOTH systems.

Secrets drift

While running in parallel, credentials rotate on the active system only. Snapshot secret state into the rollback runbook at freeze time.

The mirror job silently dies

Write-behind mirroring (Pattern B) that failed Tuesday makes Friday's fail-back lose three days of writes. Monitor the sync job itself (exit code, freshness timestamp) — see kb/observability/.

🔗 Related

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse