GitOps Practice Folder

Status: Active
Last Updated: 2026-08-29
Category: GitOps
Prerequisites: ../cicd/cicd-concepts, ../basics/git-fundamentals
Tags: gitops, infrastructure, flux, argo, drift, reconciliation, policy, merge

Summary

This directory holds the operational playbooks that make Git the single source of truth for fogserv.cloud. It covers the governing GitOps workflow, drift detection and triage, merge discipline, and policy-as-code enforcement across CI and admission layers.

Context / Why This Matters

GitOps replaces imperative scripts and tribal knowledge with a declarative, auditable model: the Git repository is the desired state, and operators (Flux or Argo CD) continuously reconcile live cluster state toward it. This produces a complete audit trail, trivially reproducible infrastructure, and fast rollback via git revert.

However, GitOps only delivers these guarantees when the workflow is rigorous: commits must be atomic and ticket-linked, drift must be detected and triaged systematically, merge discipline must keep deployment history clean, and guardrails must prevent regressions from bypassing the pipeline. The articles in this folder codify each of those obligations.

Contents

Article	Status	Last Updated	Purpose
gitops.md	Active	2026-01-30	Governing workflow, Forgejo Actions pipeline, campsite rule
drift-detection-runbook.md	Active	2026-08-26	Detect, classify, and resolve configuration drift
merge-strategies.md	Active	2026-08-26	Atomic commits, squash discipline, release tagging
policy-as-code.md	Active	2026-08-26	Kyverno admission + Conftest CI policy enforcement

Article Map

gitops.md                    (entry point — start here)
    ├── merge-strategies.md   (commit hygiene upstream of merge)
    ├── drift-detection-runbook.md
    │       ├── merge-strategies.md  (reduces accidental drift commits)
    │       └── policy-as-code.md    (prevents drift at admission time)
    └── policy-as-code.md
            └── drift-detection-runbook.md  (policy violations → drift incidents)

Implementation / Core Principles

The GitOps Contract

Every change to fogserv.cloud's running infrastructure follows this contract:

1. Ticket first — no infrastructure change is made without a Forgejo Issue; the ticket number appears in the branch name, commit subject, and MR title.
2. Declarative manifests only — no imperative kubectl apply in normal operations; everything lives in clusters/*/.
3. Pipeline gate — Forgejo Actions lints, renders (kustomize build), and optionally dry-runs before merge; checks must pass.
4. Squash merge to main — the deployment history is the commit history; each MR produces exactly one revertable commit.
5. Reconciler owns drift — Flux or Argo CD pulls changes from main and corrects drift automatically; manual intervention is the exception, never the rule.
6. Campsite rule — leave surrounding code, tests, and docs cleaner than you found them before closing the ticket.

Tooling Summary

Tool	Role	Where
Forgejo + Actions	MR pipeline, render checks, conftest policy tests	CI layer
Flux CD or Argo CD	Pull-based reconciler; drift detection + auto-heal	Cluster layer
Kyverno	Admission policies (Enforce/Audit)	Cluster admission
Conftest / OPA	Rego-based manifest linting	CI layer
Kustomize	Overlay-based environment promotion	CI + reconciler

Enforcement Points

• Branch protection: main requires MR + passing Actions + 1 approval for clusters/prod/**.
• Render gate: kustomize build clusters/prod must exit 0 in CI before merge.
• Policy gate: Conftest runs on rendered output; Kyverno validates admission.
• Drift sweep: nightly Forgejo Action compares live cluster state to Git; alerts on non-zero diff.

Update Rules

• Every addition must include a ticket reference and state whether Forgejo Actions or Gatekeeper policies enforce it.
• Track drifts/resolutions in Next Steps or Change Log so future agents can replay what changed.
• When a new policy is added to Kyverno, start it in Audit mode; flip to Enforce after one week of reviewing violations.
• All four articles follow the KB article format.

Common Pitfalls

Pitfall	Prevention
Hotfix applied by hand, never committed	Forgejo Actions denies non-MR pushes to protected branches; drift sweep catches it within 24 h
Large merge commits polluting deployment history	Repository settings enforce squash-merge only on protected main
Policy too strict on day one, causing outage	Kyverno starts in Audit; Enforce activated only after violation review
Divergent staging/prod overlays	Overlay promotion via Kustomize bases; never hand-edit clusters/prod/ directly
Drift reappears after reconcile	Find the writer (managedFields / audit logs) before reconciling again

Next Steps / Ops Actions

1. First session: read gitops.md end-to-end, then configure Forgejo branch protection and Actions render gate.
2. Second session: schedule the drift-sweep script (see drift-detection-runbook) and wire it to alerting.
3. Third session: enable Kyverno in Audit mode on staging; review violations at next ops sync.
4. Ongoing: every MR touching clusters/prod/ must pass render + conftest checks; enforce via branch protection, not culture.

Sources & Related Articles

External references consulted:

• GitOps Deployment Guide: ArgoCD, Flux, Drift Detection & Multi-Cluster Strategies — comprehensive comparison of Flux vs Argo CD
• Flux CD vs ArgoCD: Which Has Better Drift Detection — drift detection model differences
• GitOps + Git Workflow Complete Guide — branch strategy and PR rules
• Kubernetes Policy Engines 2025: OPA Gatekeeper vs Kyverno vs Polaris — policy engine comparison
• OPA vs Kyverno: A Head-to-Head Guide — selecting the right policy engine
• Weaveworks GitOps Field Guide — foundational principles

Related knowledge-base articles:

• ../cicd/cicd-concepts — CI pipeline model these tools build on
• ../basics/git-fundamentals — Git mechanics underpinning merge discipline
• ./merge-strategies — commit hygiene and squash discipline
• ./drift-detection-runbook — operational runbook for drift incidents
• ./policy-as-code — admission + CI policy enforcement
• ../infrastructure/drift-detection — conceptual foundations
• ../infrastructure/immutable-infrastructure — why tags and SHAs are immutable

Change Log

2026-08-29

• Expanded README from stub (500 bytes) to production-quality article following KB article format.
• Added article map showing cross-references between all four files.
• Added tooling summary, enforcement points, and pitfalls table.
• Added web research sources (Flux vs Argo CD drift detection, Kubernetes policy engines 2025, OPA vs Kyverno guide).
• Marked all four articles as Created with Last Updated dates in the contents table.
• Task 7 complete: all planned gitops articles exist, non-stub, and fully cross-referenced.