fogserv.cloud Deployment Runbook

Status: Active
Last Updated: 2026-08-26
Category: fogserv - Project Reference
Prerequisites: architecture-overview, ../cicd/deployment-automation.md
Time: 30 min read + deploy time
Tags: fogserv, deployment, k3s, caddy, longhorn, pvc, github-actions, secrets, rollback

Summary

How fogserv.cloud goes from a push to main to a running pod: GitHub Actions builds and pushes ghcr.io/fogserv/fogserv-cloud, then deploys over SSH to the node; the k3s manifest (Enterprise/apps/fogserv-cloud.yaml) defines Namespace, Deployment (with PVC-backed SQLite), Service, and Caddy Ingress. Includes go-live sequence, verification steps, and rollback.

Context / Why This Matters

Everything is built but production state lives in three places — Git manifests, on-cluster secrets, and the PVC database. This runbook is the single ordered list for (re)deploying from zero or updating an existing install, so no step (DNS? secret? PVC?) gets forgotten mid-outage. Authoritative sources: repo docs/DEPLOYMENT.md (go-live) and HANDOFF.md.

Implementation / Core Content

The pieces

Artifact	Role
.github/workflows/ci.yml	lint / type-check / build gate on PRs
.github/workflows/deploy-site.yml	build image → GHCR → SSH deploy to node
.github/workflows/deploy-enterprise.yml	cluster bootstrap (k3s, Caddy, Longhorn, Flux, Postgres, Forgejo) with run_bootstrap=true
Enterprise/apps/fogserv-cloud.yaml	Namespace + Deployment + Service + Ingress (caddy class, host fogserv.cloud)
Secret fogserv-cloud-secrets	envFrom into the container (DATABASE_URL, Mailgun keys, signing key)

Required GitHub repo secrets

DEPLOY_SSH_HOST, DEPLOY_SSH_USER, DEPLOY_SSH_KEY — used by the deploy workflow to reach the node.

Go-live sequence (first deploy)

1. Add the three deploy secrets to the GitHub repo.

2. Run Deploy Enterprise infrastructure with run_bootstrap=true → installs k3s, Caddy ingress, Longhorn storage, Flux, Postgres, Forgejo.

3. Create the on-cluster secret:

   kubectl -n fogserv-cloud create secret generic fogserv-cloud-secrets \
     --from-literal=DATABASE_URL=file:/data/prod.db \
     --from-literal=MAILGUN_API_KEY=... \
     --from-literal=MAILGUN_WEBHOOK_SIGNING_KEY=...
   

4. Point DNS A record for fogserv.cloud at the node IP (Caddy needs it for TLS issuance).

5. Push to main (or run Deploy website manually). Smoke test runs automatically.

Updating an existing deployment

git push origin main            # pipeline: build → GHCR → SSH deploy
# watch it land:
kubectl -n fogserv-cloud rollout status deployment/fogserv-cloud
curl -fsS https://fogserv.cloud/api/health || curl -fsSI https://fogserv.cloud | head -3

KB content changes ship with the image (kb-browser) — article edits require this rebuild path too.

Persistence: SQLite on a PVC

Production DB is SQLite at /data/prod.db. It must live on a Longhorn PersistentVolume — a plain emptyDir or container filesystem destroys all users/posts/subscribers on every pod restart. Verify before assuming data survives:

kubectl -n fogserv-cloud get pvc
kubectl -n fogserv-cloud exec deploy/fogserv-cloud -- ls -la /data

If the manifest lacks the PVC mount, add it (standalone PVC mounted at /data, matching DATABASE_URL=file:/data/prod.db) before any real content exists.

Practical Examples

Rollback after a bad release — follow ../cicd/rollback-procedures.md: kubectl -n fogserv-cloud rollout undo deployment/fogserv-cloud, then git revert the bad commit so the next pipeline converges.

Verify TLS — openssl s_client -connect fogserv.cloud:443 -servername fogserv.cloud </dev/null 2>/dev/null | openssl x509 -noout -dates (Caddy auto-renews via ACME).

Common Pitfalls & Troubleshooting

• Pod CrashLoopBackOff right after deploy — usually missing/incorrect secret keys; check kubectl -n fogserv-cloud logs deploy/fogserv-cloud --previous.
• Data gone after restart — PVC not mounted; stop writing, fix manifest, restore from backup if any existed.
• TLS pending/failing — DNS A record not pointing at node yet, or port 80/443 blocked; Caddy ACME needs both.
• Mailgun webhooks silently dropped — fail-closed by design without MAILGUN_WEBHOOK_SIGNING_KEY; set it in the same secret.
• Image not updating — manifest pins :latest; confirm the workflow actually pushed a new digest and the pod pulled it (kubectl describe pod → image ID).

Next Steps / Ops Actions

• Confirm the PVC mount exists in Enterprise/apps/fogserv-cloud.yaml before launch (HANDOFF lists this as highest-priority blocker).
• Rehearse ../cicd/rollback-procedures.md once against staging-like state.
• Wire uptime checks per ../observability/uptime-kuma-setup.md.

Sources & Related Articles

• Repo: docs/DEPLOYMENT.md, HANDOFF.md, Enterprise/README.md
• Related: architecture-overview, kb-browser, ../cicd/gitops-pipelines.md, ../sysadmin/secrets.md

Change Log

2026-08-26

• Initial creation from HANDOFF.md, docs/DEPLOYMENT.md references, and cluster manifests