fogserv.cloud fogserv.cloud

Docker Hub to Harbor - Container Registry Migration

Status: Active
Last Updated: 2026-08-14
Category: Migrations - Phase 2: Container Registries
Prerequisites: rollback-strategies, Harbor running (see kb/cloud/harbor-setup), Docker Hub account
Time: 4-6 hours
Tags: migration, docker-hub, harbor, registry, skopeo, containers, cicd

Summary

Step-by-step migration of container images from Docker Hub to a self-hosted Harbor registry. Covers pre-migration checklists, image inventory, bulk transfer with skopeo, tag preservation, CI/CD pipeline updates, verification, and gradual cutover — the recommended first migration because it's low-risk and fully reversible.

🎯 What You'll Learn

By the end of this guide, you'll be able to:

Table of Contents

  1. Why This Migration Is First
  2. Pre-Migration Checklist
  3. Harbor Preparation
  4. Image Inventory
  5. Bulk Transfer with Skopeo
  6. Tag and Digest Preservation
  7. Updating Image References
  8. CI/CD Pipeline Updates
  9. Verification Testing
  10. Gradual Cutover
  11. Troubleshooting
  12. Related Lessons

Why This Migration Is First

Container registry migration is the ideal first self-hosting move:

Property Why it matters
Fully parallel Old and new registries serve simultaneously; zero downtime required
Instant rollback Flip an env var back to Docker Hub
No user retraining docker pull works the same everywhere
Immediate savings Docker Hub rate limits and paid tiers disappear on day one
Practice run Teaches inventory → transfer → verify → cutover discipline used by every later phase

Docker Hub's rate limits (100 pulls/6h anonymous, lower on shared IPs) are themselves a reason to move: your k0s nodes pulling through one NAT can hit limits during routine scaling events.

Pre-Migration Checklist

Work through this list before transferring anything:

Project Layout Mapping

Decide your Harbor project structure up front — it determines every image reference later:

Docker Hub                          Harbor
─────────────────────────────────────────────────────────
fogserv/api                    →    harbor.shire.one/fogserv/api
fogserv/web                    →    harbor.shire.one/fogserv/web
library/nginx (public base)    →    keep pulling from upstream* 
personal-forks/tool            →    harbor.shire.one/tools/tool

* Mirror only what you BUILD. Proxy public bases via Harbor's
  proxy-cache project feature if pull reliability matters.

Recommendation: create a library proxy-cache project in Harbor pointing at Docker Hub for public base images (nginx, postgres, redis). Your nodes then have exactly ONE registry endpoint.

Harbor Preparation

Robot Accounts

CI and clusters should never use human logins:

# In Harbor UI: Projects → fogserv → Robot Accounts → New
# Push robot:   name=ci-push,    permission: push, repository=all
# Pull robot:   name=node-pull,  permission: pull, repository=all
# Save the JSON secret immediately — shown once.

Test Round-Trip Before Bulk Transfer

# Login to both
docker login docker.io                        # your hub account
docker login harbor.shire.one -u 'robot$fogserv+ci-push'

# Tiny round-trip test
docker pull alpine:3.20
docker tag alpine:3.20 harbor.shire.one/fogserv/alpine-test:3.20
docker push harbor.shire.one/fogserv/alpine-test:3.20
docker rmi alpine:3.20 harbor.shire.one/fogserv/alpine-test:3.20
docker pull harbor.shire.one/fogserv/alpine-test:3.20   # proves pull path works

If this round-trip fails, fix Harbor first — do not debug it mid-migration at scale.

Image Inventory

Enumerate everything you own on Docker Hub:

#!/usr/bin/env bash
# hub-inventory.sh — list all repos + tags for a namespace
set -euo pipefail
NAMESPACE="${1:?usage: $0 <namespace>}"
PAGE=1
while : ; do
  REPOS=$(curl -sf "https://hub.docker.com/v2/repositories/${NAMESPACE}/?page_size=100&page=${PAGE}")
  NAMES=$(echo "$REPOS" | jq -r '.results[].name')
  [ -z "$NAMES" ] && break
  for repo in $NAMES; do
    echo "=== ${NAMESPACE}/${repo} ==="
    curl -sf "https://hub.docker.com/v2/repositories/${NAMESPACE}/${repo}/tags/?page_size=100" \
      | jq -r '.results[] | "\(.name)\t\(.full_size)\t\(.last_updated)"'
  done
  PAGE=$((PAGE+1))
done

./hub-inventory.sh fogserv > hub-inventory.txt
wc -l hub-inventory.txt        # total tags
du -sh estimate: sum full_size bytes / 1e9 GB

What Happens: The script walks Docker Hub's v2 API page by page. For each repository it dumps tag name, compressed size, and last-updated date. The output doubles as your cleanup filter — tags untouched for two years probably don't need migrating (see registry-cleanup).

Save the inventory file into version control next to the migration plan. It's also your post-transfer checklist.

Prune Before You Copy

Migrating garbage costs storage forever:

# From the inventory, mark tags to skip:
awk '$2 > 2000000000 {print "SKIP (>2GB):", $0}' hub-inventory.txt
# Drop stale dev tags; keep semver releases + latest
grep -vE '\b(v)?[0-9]+\.[0-9]+\.[0-9]+$|latest$' hub-inventory.txt | head

Typical result: 60–80% of tags are stale build noise. Migrate only what's referenced.

Bulk Transfer with Skopeo

skopeo copies images between registries without a local Docker daemon, preserving manifests, architectures, and signatures metadata. It is the right tool for this job.

Installation

# Debian/Ubuntu
apt install -y skopeo
# Fedora/RHEL
dnf install -y skopeo
# Or containerized:
podman run --rm quay.io/skopeo/stable --version

Authentication for Skopeo

skopeo login docker.io                       # interactive
skopeo login harbor.shire.one \
  -u 'robot$fogserv+ci-push' --password-stdin <<< "$HARBOR_ROBOT_SECRET"
# Credentials land in ~/.config/containers/auth.json

Single Image Copy

skopeo copy \
  --all \                                # copy ALL architectures/manifests
  --dest-tls-verify=true \
  docker://docker.io/fogserv/api:1.4.2 \
  docker://harbor.shire.one/fogserv/api:1.4.2

What Happens: skopeo talks raw OCI registry protocol on both sides — no daemon involved. --all preserves multi-arch manifest lists (amd64/arm64), which plain docker pull && docker push silently flattens to your local architecture. If you skip --all, your arm64 build nodes break three weeks later and you will not remember why.

Bulk Script

#!/usr/bin/env bash
# migrate-images.sh — bulk transfer from inventory
set -euo pipefail
SRC="docker://docker.io"
DST="docker://harbor.shire.one"
NS="fogserv"
LOG=/var/log/harbor-migration.log
FAILED=/var/log/harbor-migration.failures.txt
: > "$FAILED"

transfer() {
  local repo="$1" tag="$2"
  echo "[$(date +%FT%T)] ${repo}:${tag}" >> "$LOG"
  if ! skopeo copy --all --retry-times 3 \
      "${SRC}/${NS}/${repo}:${tag}" "${DST}/${NS}/${repo}:${tag}" >>"$LOG" 2>&1; then
    echo "${repo}:${tag}" >> "$FAILED"
  fi
}

export -f transfer
export SRC DST NS LOG FAILED

while read -r line; do
  repo=$(echo "$line" | cut -d'|' -f1)     # adapt to your inventory format
  tag=$(echo "$line" | cut -d'|' -f2)
  transfer "$repo" "$tag"
done < filtered-tags.txt

echo "DONE. Failures:"; cat "$FAILED"

For large sets, parallelize carefully — Docker Hub rate-limits aggressive pullers:

xargs -P 4 -I{} bash -c 'transfer {}' < tag-list.txt   # 4 concurrent, polite

Resume behavior: skopeo skips layers already present on the destination, so re-running after interruption only transfers missing pieces. This makes the whole process idempotent — safe to loop until $FAILED is empty.

Tag and Digest Preservation

Three things must survive the move:

1. Tags — trivially preserved by the script above

2. Digests change — and that's expected

A digest (sha256:abc123...) hashes the manifest including registry-specific config, so it will differ between Hub and Harbor. What matters is that within Harbor, digest ↔ content mapping matches what was pushed. Verify per-image:

skopeo inspect docker://harbor.shire.one/fogserv/api:1.4.2 --format '{{.Digest}}'
skopeo inspect docker://docker.io/fogserv/api:1.4.2 --format '{{.Digest}}'
# Different values OK. Compare instead:
skopeo inspect --raw .../api:1.4.2 | jq -r '.manifests[].digest' | sort   # per-arch digests should match

Anything pinning images by digest (Kubernetes pods, Dockerfiles with @sha256:) must be repinned against Harbor digests — grep for them now:

grep -rn "@sha256:" deploy/ k8s/ docker-compose*.yml | grep -v harbor.shire.one

3. Multi-arch manifest lists

Verify one known multi-arch image transferred completely:

skopeo inspect --raw docker://harbor.shire.one/fogserv/api:1.4.2 | jq -r '.manifests[] | "\(.platform.architecture)"'
# Expect both: amd64 AND arm64 (whatever you ship). Missing entries = forgot --all somewhere.

Updating Image References

Now find every reference and rewrite it. Keep old/new side-by-side until cutover completes.

#!/usr/bin/env bash
# rewrite-references.sh — dry-run by default, APPLY=1 to write
set -euo pipefail
PATTERN='docker\.io/fogserv/([a-z0-9_-]+)'
REPLACEMENT='harbor.shire.one/fogserv/\1'
FILES=$(grep -rlE "$PATTERN" deploy/ k8s/ compose/ .github/ scripts/ 2>/dev/null || true)

for f in $FILES; do
  echo "--- $f"
  grep -nE "$PATTERN" "$f"
  if [ "${APPLY:-0}" = "1" ]; then
    sed -i -E "s#${PATTERN}#${REPLACEMENT}#g" "$f"
    git add "$f"    # review via git diff before committing
  fi
done

Reference types to sweep:

Location Example Notes
Compose files image: fogserv/api:1.4.2 implicit docker.io prefix!
Kubernetes manifests/Helm values image: fogserv/api:1.4.2 same implicit prefix
Dockerfiles FROM fogserv/base-builder:latest builder images too
CI files IMAGE: docker.io/fogserv/api usually an env var
Watchtower/auto-update configs schedule args easy to forget

The implicit docker.io prefix is the classic miss: image: fogserv/api means Docker Hub. After cutover those refs still work — but they bypass Harbor entirely, so you get rate limits again and think your migration broke something.

CI/CD Pipeline Updates

GitHub Actions example (before → after)

# BEFORE
env:
  REGISTRY: docker.io
  IMAGE_NAME: fogserv/api
jobs:
  build:
    steps:
      - uses: docker/login-action@v3
        with: { username: ${{ secrets.DOCKERHUB_USER }}, password: ${{ secrets.DOCKERHUB_TOKEN }} }

# AFTER
env:
  REGISTRY: harbor.shire.one
  IMAGE_NAME: fogserv/api
jobs:
  build:
    steps:
      - uses: docker/login-action@v3
        with:
          registry: harbor.shire.one
          username: 'robot$fogserv+ci-push'
          password: ${{ secrets.HARBOR_ROBOT_SECRET }}

(Once pipelines live in Woodpecker — see github-actions-to-woodpecker — the same env-var pattern applies.)

Cluster node authentication

k0s/containerd needs the pull robot configured once per node:

# /etc/containerd/config.toml (or via k0s helm charts for cluster-wide)
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.shire.one".auth]
  username = "robot$fogserv+node-pull"
  password = "<secret>"

Then restart containerd and validate from the node itself:

crictl pull harbor.shire.one/fogserv/api:1.4.2

Store robot secrets in your secret manager (dotenvx/Vault/SOPS) — never inline in manifests.

Verification Testing

Before cutting anything over, prove parity:

#!/usr/bin/env bash
# verify-parity.sh — compare tag sets + spot-check image configs
set -euo pipefail
while read -r repo; do
  # 1. Same tag set?
  diff \
    <(skopeo list-tags docker://docker.io/fogserv/"$repo" | jq -r '.Tags[]' | sort) \
    <(skopeo list-tags docker://harbor.shire.one/fogserv/"$repo" | jq -r '.Tags[]' | sort) \
    || { echo "TAG MISMATCH: $repo"; exit 1; }

  # 2. Latest release tag: identical config digest per arch?
  for arch in amd64 arm64; do
    d1=$(skopeo inspect --raw docker://docker.io/fogserv/"$repo":$(cat latest-tag) \
         | jq -r ".manifests[] | select(.platform.architecture==\"$arch\") | .digest")
    d2=$(skopeo inspect --raw docker://harbor.shire.one/fogserv/"$repo":$(cat latest-tag) \
         | jq -r ".manifests[] | select(.platform.architecture==\"$arch\") | .digest")
    [ "$d1" = "$d2" ] || { echo "ARCH DIGEST MISMATCH: $repo/$arch"; exit 1; }
  done
  echo "OK: $repo"
done < migrated-repos.txt

Then functional smoke tests:

  1. Deploy a scratch pod/compose service pinned to a Harbor image → runs, serves traffic
  2. Trigger one real CI pipeline end-to-end → builds, pushes to Harbor, deploys
  3. Pull from a clean machine with only Harbor credentials → works without any Docker Hub auth

Gradual Cutover

Registries allow true blue-green (see rollback-strategies Pattern C):

Day 0:   Transfer complete, verification green. CI pushes to BOTH registries.
Week 1:  Staging + canary nodes pull from Harbor only. Production still Hub.
Week 2:  All nodes pull Harbor. CI still dual-pushes. Watch error rates.
Week 3:  CI single-push to Harbor. Docker Hub frozen (read-only archive).
Day 90:  Decommission decision — see [decommissioning-commercial].

Rollback at any point = flip pull endpoints back to docker.io/.... Because images were never deleted from Hub, rollback cost is minutes.

Dual-push snippet for the transition window:

docker buildx build --push \
  -t docker.io/fogserv/api:${TAG} \
  -t harbor.shire.one/fogserv/api:${TAG} \
  .

Troubleshooting

denied: requested access to the resource is denied on push

Robot account lacks push permission, wrong project name, or you're authenticating as a human whose session expired. Check: skopeo login output, Harbor UI → Project → Robot Accounts.

unexpected media type / missing architectures downstream

You copied without --all at least once. Re-run the copy for affected tags with --all; skopeo fills in missing manifests.

Harbor disk fills during transfer

Registry garbage accumulates from interrupted pushes. Enable/force GC: Harbor UI → trash settings, or CLI docker run --rm goharbor/harbor-gc. Provision 2× expected size beforehand.

CI can authenticate but push gets unauthorized

Harbor projects default to private with strict RBAC. Verify the robot's permission level covers the exact project, and that the project isn't in "read-only" maintenance mode.

Nodes still hitting Docker Hub despite updated manifests

Someone referenced images by digest, or an old DaemonSet/cron still runs the old spec. kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | sort -u | grep -v harbor finds stragglers.

Rate-limit errors DURING migration transfer

Your bulk puller tripped Hub's limits. Reduce -P concurrency, add sleep between repos, or use authenticated pulls (higher limits).

🔗 Related

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse