Secrets Management Reference

Status: Active
Last Updated: 2026-08-26
Category: Sysadmin - Operations
Prerequisites: system-admin-basics, dotenvx
Tags: secrets, dotenvx, rotation, vault, audit

Summary

This entry is the KB-formatted mirror of the SECRETS_MANAGEMENT.md playbook. It describes how fogserv.cloud manages secrets across environments: local development (.env), templated files (.env.example), production (.env.production), and encrypted keys (.env.keys). It defines rotation cadence, agent rules, audit requirements, and the relationship to dotenvx.md (layered environment management) and system-admin-basics.md (inventory and audit discipline).

Context / Why This Matters

Secrets are the most common source of silent failures and security breaches in self-hosted environments. A missing .env.production variable takes a service offline without clear error messages; an unrotated database password exposed in a log file creates persistent access for an attacker who copied logs before rotation. The KB entry creates a single audit point: every creation, rotation, and revocation is timestamped, linked to a Forgejo Issue, and cross-referenced with the hosting environment.

This complements the technical mechanism in dotenvx.md (encryption, CLI, layer design) and provides the operational discipline required by system-admin-basics.md (patch/change management, documentation mandates).

Implementation / Core Content

Key Files

File	Purpose	Committed?	Encrypted?	Access
.env	Local development use only	No (.gitignore)	No	Developer laptop
.env.example	Template listing required variables with descriptions	Yes	No	Public
.env.production	Production config (secrets)	No	Yes (.env.production.enc)	CI/CD only
.env.keys	Encryption keys (per layer)	No	No (stored in secret manager)	CI/CD + admin
.dotenvx	Schema rules, overrides, merge order	Yes	No	Public

Secret Lifecycle

Every secret passes through these stages:

1. Creation — variable added to .env.example; agent creates .env locally; Forgejo Action validates with secrets:validate; variable used in production through encrypted .env.production.
2. Storage — production values encrypted with secrets:encrypt; .env.keys stored in Forgejo secrets manager (Actions) or a dedicated vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) if audit trails are required.
3. Consumption — agents load variables through dotenvx run -- or Forgejo Action injection; values never appear in agent instructions, KB text, or logs.
4. Rotation — change .env.production, re-encrypt, deploy via CI, verify application health, record rotation in system-admin-basics.md inventory, and document in the Change Log below with the Forgejo Issue ID.
5. Revocation — when a secret is leaked or a service is decommissioned, rotate immediately, remove references from .env.example and code, and verify no remaining references with rg "OLD_VAR_NAME" src/.

Rotation Protocol

Rotation must follow the same process regardless of urgency:

# Step 1: Update value (plaintext locally for editing)
nano .env.production

# Step 2: Re-encrypt
bun run secrets:encrypt -f .env.production

# Step 3: Update encryption keys if rotating the key itself
bun run secrets:generate --replace

# Step 4: Deploy encrypted file via CI; verify application connects
# Step 5: Record in KB change log with timestamp + Forgejo Issue ID

Rotation cadence:

• Security-critical secrets (database passwords, API keys): every 90 days or after any incident, whichever is sooner.
• Integration tokens (GIT_TOKEN, MAILGUN_API_KEY): every 180 days.
• Development-only secrets (LOCAL_DEBUG_KEY): no fixed cadence; rotate when shared with new team members.

Agent Rules

Agents interacting with secrets must follow these rules without exception:

• Reference secrets only by variable name (DATABASE_URL, PRISMA_ORM), never by value or partial value.
• Never include secret values in KB articles, Forgejo Issue descriptions, or chat transcripts.
• Never commit .env, .env.production, or .env.keys to version control.
• Always load secrets through dotenvx run -- or CI/CD injection; do not load directly from .env in production scripts.
• If a script logs an environment value, treat it as a leak: fix the script, rotate the secret, document in the change log, and create a Forgejo Issue.
• When onboarding a new feature, document every required variable in .env.example and link the feature documentation to dotenvx.md and this entry.

Secrets Auditing

Every quarter, run the audit checklist:

# Verify all variables in .env.example appear in .env.production
# (production variables should exist; missing variables = missing feature or broken deployment)
bun run secrets:validate

# Check for unencrypted .env.production
ls .env.production                 # should return nothing; only .env.production.enc exists

# Check for secrets committed accidentally
git log --all --full-history -- .env .env.production .env.keys
# If any hits found: remove from history, rotate all affected secrets, document in KB.

# Verify rotation records exist for every variable that has been rotated
# (check [system-admin-basics.md](system-admin-basics.md) inventory + this change log)

When Dotenvx Is Not Enough

Dotenvx handles the threat model: "I don't want plaintext secrets committed to Git or stored unencrypted on developer laptops." It does not handle:

• Dynamic secrets (temporary tokens generated at runtime by Vault or cloud providers).
• Audit trails of who accessed which secret and when.
• Multi-environment key rotation without human intervention.
• Access control (who can decrypt .env.production).

If the fogserv.cloud environment requires any of these capabilities, migrate the primary store to HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Doppler, and keep Dotenvx as the runtime loader only (decrypt at deploy time through a short-lived CI/CD key).

See Secrets Management in 2026 for a comparison of Vault, AWS Secrets Manager, Azure Key Vault, Doppler, and Infisical.

Practical Examples

Example 1: Verify no secret leaks in application logs

# Scan source for direct environment access patterns
grep -r "console.log.*process\.env\." src/
# Expected: zero results. Any result requires script fix + rotation + KB entry.

Example 2: Confirm every variable defined in .env.example is present in encrypted production

# Extract variable names from .env.example (lines starting with uppercase names, ignoring comments and blanks)
awk '{print $1}' .env.example | grep -v '^#' | grep -v '^$' > required_vars.txt

# Validate encrypted .env.production after decryption
DOTENV_PRIVATE_KEY_PRODUCTION=$(cat .env.keys | head -n1) \
  bun run secrets:inspect .env.production.enc | grep -f required_vars.txt

Expected: all variables from .env.example appear in the validated output.

Example 3: Full rotation event documentation

After rotating DATABASE_URL, create a Forgejo Issue titled Secrets rotation: DATABASE_URL (2026-08-26) and append to the Change Log below:

### 2026-08-26 — DATABASE_URL rotation — Issue #142
- Old value revoked. New encrypted `.env.production.enc` deployed.
- Application restarted; connection verified.
- No leaks detected in logs.

Common Pitfalls & Troubleshooting

Problem	Cause	Fix
Secret value logged in CI output	Script logs process.env.VAR directly	Fix script; rotate secret; document incident
Decryption fails in CI	.env.keys missing or environment variable name wrong	Verify Forgejo secret name; confirm .env.keys exists locally
Rotation has no audit record	KB Change Log not updated; no Forgejo Issue	Create Issue; append to Change Log; link Issue ID
.env.production missing variables	New feature added variables to .env.example but not to production layer	Add variables; encrypt; validate; deploy
Secret committed to Git	Developer committed .env accidentally	Remove from history (git-filter-repo or BFG); rotate all affected secrets; validate all other variables

Next Steps / Ops Actions

• Apply the rotation protocol to all secrets in the fogserv.cloud environment; record rotation events in the inventory (system-admin-basics.md).
• Route all rotation events through Forgejo Actions per scheduler-patterns.md (scheduled audit, backup, rotation scripts).
• Cross-reference this KB entry in any agent that consumes sensitive data so expectations stay visible (see agent onboarding docs and dotenvx.md).

Sources & Related Articles

External references:

• Dotenvx official documentation
• Secrets Management in 2026: Vault vs Secrets Manager vs SOPS
• Dotenvx vs Vault comparison
• Secrets rotation engineering reference
• CISA Logging Reference Architecture: https://www.cisa.gov/resources-tools/resources/logging-reference-architecture

Related KB articles:

• dotenvx.md — environment layer design, CLI, encryption
• system-admin-basics.md — inventory, audit, change log discipline
• scheduler-patterns.md — rotation scheduling with systemd timers

Change Log

2026-01-30 03:07 AM — Initial KB-formatted secrets reference

• Created the KB-formatted secrets reference and linked it to the Dotenvx playbook.

2026-08-26 — Expanded to production format (Task 7 refresh)

• Expanded to full production format with layer lifecycle, rotation protocol, agent rules, audit checklist, Vault comparison, troubleshooting table, and CISA / 2026 web research citations.
• Added cross-links to system-admin-basics.md, scheduler-patterns.md, and dotenvx.md.
• Updated Change Log to document rotation events with Forgejo Issue linkage.