Phase 6 Artifacts — Workflows & Automation

Status: Draft Last Updated: August 29, 2026 Tags: gitops, deployment, rollback, backup, minio, restic

---

Summary

Phase 6 produces three artifacts: a staging deploy snippet (GitOps), a rollback runbook entry, and a backup automation template using MinIO + systemd timer. These are intended for drop-in use or expansion — not polished infrastructure code.

---

1. Staging Deploy — GitOps Snippet

Context

Forgejo Actions builds the Docker image and pushes to GHCR on pushes to main. Staging is a parallel environment (separate namespace, separate SQLite file, same app) that gates production by running feat/* branches for manual review.

Artifact: .forgejo/workflows/staging.yml

name: Deploy to Staging

on:
  push:
    branches:
      - 'feat/**'
  workflow_dispatch:

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository_owner }}/fogserv-cloud

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Bun
        uses: oven-sh/setup-bun@v2
        with:
          bun-version: latest

      - name: Install dependencies
        run: bun install --frozen-lockfile

      - name: Generate Prisma client
        run: bun run db:generate

      - name: Run type check
        run: bun run type-check

      - name: Build Docker image
        run: |
          VERSION=staging-${{ github.sha }}
          docker build -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${VERSION} .
          docker tag ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${VERSION} \
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:staging-latest

      - name: Push to GHCR
        run: |
          echo ${{ secrets.GITHUB_TOKEN }} | docker login ${{ env.REGISTRY }} --username ${{ github.actor }} --password-stdin
          docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:staging-${{ github.sha }}
          docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:staging-latest

      - name: Deploy to staging node via SSH
        run: |
          STAGING_TAG="staging-${{ github.sha }}"
          ssh ${{ secrets.DEPLOY_SSH_USER }}@${{ secrets.DEPLOY_SSH_HOST }} \
            "kubectl -n fogserv-staging set image deployment/fogserv-staging web=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${STAGING_TAG} && kubectl -n fogserv-staging rollout status deployment/fogserv-staging --timeout=5m"

      - name: Smoke test
        run: |
          sleep 10
          curl -fsS https://staging.fogserv.cloud/api/health \
            || curl -fsSI https://staging.fogserv.cloud | head -3

Notes

• Staging namespace fogserv-staging must be created and PVC provisioned before first deploy.
• DEPLOY_SSH_USER / DEPLOY_SSH_HOST are repo secrets; see kb/fogserv/deployment-runbook.md for the production equivalent.
• Image tag uses sha to ensure every commit gets a unique digest; :staging-latest is the convenience alias.

---

2. Rollback — Runbook Entry

Artifact: kb/cicd/rollback-procedures.md (additions)

Emergency rollback (one command)

# Roll back to the previous ReplicaSet in k3s
kubectl -n fogserv-cloud rollout undo deployment/fogserv-cloud

# Verify the rollback landed
kubectl -n fogserv-cloud rollout status deployment/fogserv-cloud --timeout=3m
curl -fsS https://fogserv.cloud/api/health

Named revision rollback

# List the last 5 revisions
kubectl -n fogserv-cloud rollout history deployment/fogserv-cloud

# Roll back to a specific revision
kubectl -n fogserv-cloud rollout undo deployment/fogserv-cloud --to-revision=3

Data integrity check after rollback

# Confirm SQLite file is intact on the PVC
kubectl -n fogserv-cloud exec deploy/fogserv-cloud -- ls -lh /data/prod.db

# Quick smoke: count posts, users, subscribers
kubectl -n fogserv-cloud exec deploy/fogserv-cloud -- \
  bunx prisma db execute --stdin <<< "SELECT COUNT(*) FROM Post;"

Git revert (prevent pipeline re-deploying bad code)

# After kubectl rollback, prevent the next pipeline run from re-deploying the bad image
git revert <bad-commit-sha>
git push origin main

Staging rollback

kubectl -n fogserv-staging rollout undo deployment/fogserv-staging
# No git revert needed for staging; the bad branch can be deleted instead
git push origin --delete feat/my-bad-branch

Rollback decision tree

1. Site down / 5xx → kubectl rollout undo immediately → then investigate.
2. Wrong content / partial bug → Evaluate severity; if bad enough, rollback.
3. Performance regression → Collect metrics first; rollback only if critical.
4. Never rollback for cosmetic changes; fix forward with a new commit.

---

3. Backup Automation — Template (backup-to-minio + timer)

Context

MinIO is already in scope (see kb/cloud/minio-setup.md). Restic backs up to any S3-compatible backend. This template wires a systemd timer to run restic backup nightly, retaining the last 7 snapshots with weekly and monthly keep policies.

Artifact: scripts/backup-fogserv.sh

#!/usr/bin/env bash
#========================================================
# backup-fogserv.sh — Nightly backup to MinIO via restic
# Run via: systemd timer, cron, or directly for manual backup
# Env: RESTIC_REPOSITORY, RESTIC_PASSWORD (set in /etc/backup/env)
#========================================================

set -euo pipefail

# Load secrets (owned by root, readable only by root)
# Source: /etc/backup/env
if [[ -f /etc/backup/env ]]; then
  set -a
  source /etc/backup/env
  set +a
fi

# Targets
readonly DB_PATH="/data/prod.db"         # PVC-mounted SQLite
readonly CADDY_PATH="/etc/caddy/Caddyfile"
readonly STATIC_PATH="/var/www/fogserv"  # Any static assets
readonly KB_PATH="/home/agentic/fogserv.cloud/kb"
readonly LOG="/var/log/backup-fogserv.log"

log() { echo "[$(date -Iseconds)] $*" | tee -a "$LOG"; }

# Tag with date for retention queries
readonly TAG=$(date +%Y-%m-%d)

log "Starting backup..."

# Backup SQLite DB
if [[ -f "$DB_PATH" ]]; then
  restic backup "$DB_PATH" \
    --host fogserv-cloud-prod \
    --tag "db" --tag "$TAG"
else
  log "WARN: DB not found at $DB_PATH, skipping."
fi

# Backup Caddy TLS and site config
if [[ -f "$CADDY_PATH" ]]; then
  restic backup "$CADDY_PATH" \
    --host fogserv-cloud-prod \
    --tag "config" --tag "$TAG"
fi

# Backup KB docs (for disaster recovery; non-critical)
if [[ -d "$KB_PATH" ]]; then
  restic backup "$KB_PATH" \
    --host fogserv-cloud-prod \
    --tag "kb" --tag "$TAG"
fi

# Retention policy: keep last 7 daily, 4 weekly, 6 monthly
log "Applying retention policy..."
restic forget \
  --keep-last 7 --keep-weekly 4 --keep-monthly 6 \
  --prune

log "Backup complete."

# Check repository size
restic stats || true

Artifact: systemd unit + timer files

/etc/systemd/system/backup-fogserv.service

[Unit]
Description=Backup fogserv.cloud to MinIO via restic
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/backup/env
ExecStart=/home/agentic/backup-fogserv.sh
StandardOutput=journal
StandardError=journal

/etc/systemd/system/backup-fogserv.timer

[Unit]
Description=Run backup-fogserv.service nightly at 03:00 UTC

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
RandomizedDelaysec=30m

[Install]
WantedBy=timers.target

Setup steps

# 1. Install restic
apt install restic || go install github.com/restic/restic@latest

# 2. Create MinIO bucket (once)
mc alias set prod https://minio.internal:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc mb prod/backups-fogserv --ignore-existing

# 3. Initialize restic repo (once)
export RESTIC_PASSWORD="$(openssl rand -base64 32)"
export RESTIC_REPOSITORY="s3:https://minio.internal:9000/backups-fogserv"
restic init

# 4. Save secrets
mkdir -p /etc/backup
cat > /etc/backup/env <<EOF
RESTIC_REPOSITORY=s3:https://minio.internal:9000/backups-fogserv
RESTIC_PASSWORD=<from-step-3>
AWS_ACCESS_KEY_ID=<minio-access-key>
AWS_SECRET_ACCESS_KEY=<minio-secret-key>
EOF
chmod 600 /etc/backup/env

# 5. Install script
cp scripts/backup-fogserv.sh /home/agentic/backup-fogserv.sh
chmod +x /home/agentic/backup-fogserv.sh

# 6. Install systemd unit + timer
cp backup-fogserv.service /etc/systemd/system/
cp backup-fogserv.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now backup-fogserv.timer

# 7. Verify timer is active
systemctl list-timers --no-pager | grep backup-fogserv

Restore procedure

# Restore latest DB snapshot to a temp path (verify before overwriting)
export RESTIC_PASSWORD="<password>"
export RESTIC_REPOSITORY="s3:https://minio.internal:9000/backups-fogserv"
restic restore latest --tag "db" --target /tmp/restore-fogserv
cp /tmp/restore-fogserv/data/prod.db /data/prod.db

# Restore Caddy config
restic restore latest --tag "config" --target /tmp/restore-caddy
cp /tmp/restore-caddy/etc/caddy/Caddyfile /etc/caddy/Caddyfile
systemctl reload caddy

---

Next Steps / Ops Actions

1. Create fogserv-staging k3s namespace and staging PVC before first staging deploy.
2. Add staging.yml workflow to .forgejo/workflows/ and test with a feat/test-deploy branch.
3. Set up MinIO bucket and restic repo credentials on the node.
4. Test the restore procedure end-to-end before relying on backups.
5. Wire the backup timer into uptime monitoring (kb/observability/uptime-kuma-setup.md).

Sources & Related

• kb/fogserv/deployment-runbook.md — existing production deploy docs
• kb/cloud/minio-setup.md — MinIO configuration
• kb/cloud/restic-backups.md — restic patterns
• .forgejo/workflows/ci.yml — existing CI workflow conventions
• kb/cicd/rollback-procedures.md — broader rollback guidance

Change Log

• August 29, 2026 — Created Phase 6 artifacts: staging deploy GitOps workflow, rollback runbook entries (kubectl undo + git revert + staging variant), backup automation script template (restic + MinIO + systemd timer) with setup and restore steps.