fogserv.cloud fogserv.cloud

Migration Risks - Understanding What Can Go Wrong

Status: Active
Last Updated: 2026-08-14
Category: Migrations - Phase 1: Planning & Assessment
Prerequisites: migration-planning
Time: 2 hours
Tags: migration, risk, data-loss, downtime, compliance, mitigation

Summary

Deep-dive into the concrete ways migrations fail: data loss scenarios, downtime impact, compliance traps, and skill gaps. For every failure mode you'll get detection methods, prevention strategies, and mitigation playbooks — so the risks in your plan document are specific instead of hand-wavy.

🎯 What You'll Learn

By the end of this guide, you'll be able to:

Table of Contents

  1. The Risk Model
  2. Data Loss Scenarios
  3. Downtime Impact
  4. Compliance Considerations
  5. Skill Gaps
  6. Mitigation Strategies
  7. Risk Register Template
  8. Common Gotchas
  9. Related Lessons

The Risk Model

Every migration risk combines three factors:

Risk severity = Probability × Impact × Detection difficulty

The third factor is the one teams forget. A silent 2% data corruption is far worse than a loud full outage, because the outage gets fixed tonight and the corruption spreads through backups for weeks.

Factor Question to ask
Probability Has this failed for other people? (Search GitHub issues, forum posts.)
Impact Money lost? Trust lost? Work stopped?
Detection How would we know within 1 hour? Within 1 day? Ever?

Score each risk on all three. Prioritize mitigations for high scores in any column — especially detection.

Data Loss Scenarios

Data loss during migration is rarely "the disk caught fire." It's subtle and cumulative. Here are the patterns that actually happen:

Pattern 1: The Truncated Tail

A bulk transfer is interrupted at 97%. The tool exits with an error someone misses in scrollback. Everything looks fine because most files are there.

What it looks like:
- rclone/rsync killed mid-run (SSH disconnect, laptop sleep)
- API rate limits stall a script that doesn't retry
- Timeout defaults truncate large transfers silently

Detection:
  rclone check source: dest: --one-way        # checksum comparison
  rsync -avnc src/ dst/ | tail -20            # dry-run diff listing

Prevention: Always run a verification pass (rclone check, rsync -c) after bulk transfers. Never trust exit codes alone from long-running jobs wrapped in other scripts.

Pattern 2: The Invisible Metadata

Files arrive; their meaning doesn't.

Metadata type Commonly lost during Consequence
File permissions / owners tar-less copies across systems Services can't read files; security holes
Timestamps naive cp without -a Backup rotation logic breaks
Extended attributes cloud sync tools App-specific data vanishes
Symlinks SFTP GUIs Broken references
Sparse files some copy tools Disk usage explodes 
# Preserve everything that matters:
rsync -aAXH src/ dest/     # -a perms/timestamps -A ACLs -X xattrs -H hardlinks

# Verify permissions survived:
find dest/ ! -perm u+rw -ls | head    # anything odd?

Pattern 3: The Uncounted Records

Databases migrate by count but lose meaning:

-- Post-migration sanity checks:
SELECT count(*) FROM users;                    -- matches source?
SELECT max(id) FROM users;                     -- sequences advanced?
SELECT setval('users_id_seq', (SELECT max(id) FROM users));  -- fix sequence

-- Find mojibake (double-encoded UTF-8):
SELECT id FROM articles WHERE title LIKE '%Ã%' OR title LIKE '%Â%';

Pattern 4: The Orphaned Reference

The data moved fine, but pointers to it broke:

This is why the inventory phase (see migration-planning) hunts hidden dependencies before anything moves.

Pattern 5: The Backup That Was Never Tested

You migrated everything including backups — but the restore path was never exercised on the new system. Your backups are Schrödinger's files until proven restorable.

# Monthly ritual, non-negotiable:
# 1. Pick random backup
# 2. Restore into scratch environment
# 3. Run application against restored data
# 4. Diff a few records against production

Data Loss Detection Toolkit

Tool Use case
rclone check src: dst: Object storage checksums, bidirectional
rsync -avnc File trees, dry-run compare
diff -r Small trees, quick sanity
md5deep -r . > manifest then re-check Immutable manifests for big sets
DB row counts + max IDs + spot queries Databases
git fsck Git repos post-move

Generate manifests before migration as evidence:

# Before:
find /data -type f -exec md5sum {} \; > /root/pre-migration-manifest.txt
sha256sum pre-migration-manifest.txt   # keep this hash in the plan doc

# After (on new system), same command, then diff the manifests.

Downtime Impact

Quantifying Instead of Guessing

For each service, fill in the grid:

Service: Git hosting (Forgejo target)

Scenario A: 1 hour outage, Saturday night
  Cost: ~$0. Nobody works weekends. ACCEPTABLE.

Scenario B: 4 hours outage, Tuesday morning
  Cost: 12 devs idle = 48 person-hours ≈ $4,800 + CI blocked.
  Painful but survivable ONCE. NOT acceptable repeatedly.

Scenario C: Silent data corruption found after 2 weeks
  Cost: Trust destroyed. Re-doing work. Possibly unrecoverable commits.
  CATASTROPHIC — this is what we're really defending against.

Downtime Reduction Techniques

Technique 1: Parallel running (preferred where possible) Both old and new services stay live; writes go to both or get replicated; users switch when ready. Costs double infrastructure temporarily. Details in rollback-strategies.

Technique 2: Read-only freeze Instead of full downtime, make the old system read-only during final sync. Users see "temporarily read-only" instead of errors. Works great for wikis, git, issue trackers.

Technique 3: Incremental sync + short cutover Bulk-copy ahead of time, then do incremental deltas, so the final cutover window only handles the last delta:

# T-48h: bulk copy (hours of transfer, no user impact)
rclone sync dropbox-prod: minio-new: --progress

# T-0: final delta (minutes, during announced freeze)
rclone sync dropbox-prod: minio-new:      # only changed objects transfer

# T+10min: flip applications to MinIO endpoints

Technique 4: DNS-based traffic shifting Lower TTLs in advance (see dns-migration), then move traffic incrementally if your setup supports weighted records.

Compliance Considerations

Even small teams have obligations they forget about:

Questions That Change Your Migration Plan

  1. Where must data reside geographically? Self-hosting may help (your rack) or hurt (your basement isn't SOC2).
  2. Retention requirements? If law/policy requires N years of email retention, verify your mail server's archive design before migrating off Google Vault.
  3. Encryption requirements? Data-at-rest encryption obligations follow the data. See kb/security/disk-encryption-luks.md.
  4. Audit logging? Some frameworks require tamper-evident access logs. Does Forgejo/Nextcloud/MinIO config provide equivalent audit trails?
  5. Right-to-deletion? GDPR-style erasure requests need a documented deletion path in the new system too.
  6. Customer contracts? Check DPAs — some name approved subprocessors/locations explicitly.

Practical Rule

Write down which compliance obligations apply (even just "none beyond common sense") in the migration plan. During an incident, "we considered it and documented the decision" beats "nobody thought about it."

Skill Gaps

The risk nobody puts on slides: the migration succeeds, then degrades for months because nobody truly knows the new stack.

Gap Assessment

For each target technology, rate the team:

Capability Level needed Who has it Gap?
Day-to-day operation Basic everyone —
Troubleshooting logs/errors Intermediate ? ⚠️
Restore-from-backup procedure Confident practice ? 🔴
Upgrading safely Intermediate ? ⚠️
Security hardening Intermediate ? ⚠️

Closing Gaps Cheaply

  1. Pair on the staging build — the person who built it walks another through it; record the session as internal documentation.
  2. Game-day exercise: deliberately break staging ("restore last night's backup"), see who fixes it, write down what was hard.
  3. Runbooks before cutover: every routine operation documented as commands-to-paste. If it needs tribal knowledge, it isn't done.
  4. Bus factor ≥ 2 rule from migration-planning applies per technology, not per project.

Mitigation Strategies

Layer defenses — no single mitigation is enough:

Layer 1: Prevent

Layer 2: Detect fast

Layer 3: Recover

Layer 4: Learn

Risk Register Template

Copy into your migration plan:

| ID | Risk                          | P(1-5) | I(1-5) | D(1-5) | Score | Mitigation                          | Owner  |
|----|-------------------------------|--------|--------|--------|-------|-------------------------------------|--------|
| R1 | Truncated file transfer       | 3      | 4      | 4      | 48    | rclone check pass + manifests       | Alex   |
| R2 | Sequence counters not moved   | 2      | 4      | 5      | 40    | Post-import setval script           | Sam    |
| R3 | Webhook endpoints stale       | 4      | 3      | 2      | 24    | Inventory + integration test suite  | Jordan |
| R4 | Nobody can debug Woodpecker   | 3      | 3      | 3      | 27    | Pairing session + game day          | Riley  |
| R5 | Compliance: EU data residency | 2      | 5      | 1      | 10    | Documented in plan; hardware in DE  | Alex   |

P=probability, I=impact, D=detection difficulty. Score = P×I×D. Review weekly until decommission.

Review the register at every checkpoint. Retire risks that passed validation windows; add ones discovered mid-flight.

Common Gotchas

Clock skew between systems

Timestamps compared between old and new hosts can lie if NTP is broken. Verify with timedatectl status before trusting "nothing changed today."

Case-insensitive vs case-sensitive filesystems

Migrating from macOS/Windows shares to Linux: Report.PDF and report.pdf coexist on the source, collide on the destination. Test with find . | tr '[:upper:]' '[:lower:]' | sort | uniq -d style checks.

The 2 AM rule

If a failure mode has no documented first response, assume it happens at 2 AM to whoever is least prepared. Every risk in the register needs a runbook line, not just a name.

Migration tools have their own bugs

Pin versions of rclone/skopeo/etc. used in the runbook, and rehearse with those exact versions. "Latest" changing under you mid-project adds variance you don't need.

🔗 Related

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse