Vault Secrets - Storing, Retrieving, and Generating Credentials

Status: Active
Last Updated: 2026-08-14
Category: Security - Phase 4: Secret & Access Management
Prerequisites: vault-introduction, vault-authentication
Time: 3-4 hours
Tags: vault, kv, dynamic-secrets, database, postgres, rotation, api

Summary

Vault's real power shows when secrets stop being static strings you manage by hand. This lesson covers the KV v2 engine for static secrets (versioning, rollback, soft delete), then graduates to dynamic secrets: asking Vault to mint short-lived PostgreSQL credentials on demand so no human ever knows the production database password. You'll wire a database secret engine, consume it from an application, and build a rotation strategy that finally makes "rotate quarterly" actually happen.

---

🎯 What You'll Learn

• ✅ Use KV v2 correctly: paths, versioning, patch, rollback, destroy
• ✅ Explain static vs dynamic secrets and when each is appropriate
• ✅ Configure Vault's database secrets engine for PostgreSQL
• ✅ Consume dynamic credentials from an app with lease renewal handling
• ✅ Design a rotation strategy using leases instead of calendar discipline
• ✅ Integrate secret retrieval into application code via API or agent templates

---

Static Secrets: The KV v2 Engine

You enabled kv-v2 at secret/ in vault-introduction. Key facts about its behavior:

CLI path:   secret/myapp/config
API path:   secret/data/myapp/config      ← note the data/ segment in raw HTTP
Metadata:   secret/metadata/myapp/config  ← version history lives here

Everyday Commands

export VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=hvs.xxx

# Write (creates version 1)
vault kv put secret/myapp/config \
    db_host=pg.internal db_port=5432 api_key=sk-live-abc123

# Overwrite (creates version 2 — old versions stay readable!)
vault kv put secret/myapp/config api_key=sk-live-def456

# Read specific versions
vault kv get secret/myapp/config
vault kv get -version=1 secret/myapp/config

# Patch one key without clobbering others
vault kv patch secret/myapp/config api_key=sk-live-ghi789

# List what exists under a prefix
vault kv list secret/myapp/

What Happens: every write creates a new immutable version. kv get always returns the latest; -version=N reaches back in time. This is your undo button after a bad overwrite:

# Rollback: create version 3 with version 1's content
vault kv rollback -version=1 secret/myapp/config

Deleting is two-stage, like Git:

vault kv delete secret/myapp/config     # soft delete (latest); recoverable via rollback
vault kv undelete -versions=2 secret/myapp/config
vault kv destroy -versions=1,2 secret/myapp/config   # permanently shred old versions
vault kv metadata delete secret/myapp/config         # remove everything incl. history

⚠️ A secret that was ever written remains in version history until destroyed. If someone pasted a real production password as a test value, destroy those versions immediately — soft delete is not enough.

Organizing Secret Paths

Treat paths like filesystem permissions boundaries (policies in rbac-basics match on these prefixes):

secret/
├── shared/          # team-wide, low sensitivity (webhook URLs)
├── apps/
│   ├── website/     # only website service account can read
│   └── crm/
├── infra/           # terraform/ansible state encryption keys
└── personal/<user>/ # individual tokens, API keys

Rule: one app = one path prefix = one policy. Shared secrets breed shared blast radius.

---

Dynamic Secrets: Credentials That Don't Outlive Their Use

Static DB passwords have three chronic failures: they never rotate, everyone shares one, and revoking one person means changing it everywhere. Vault's database secrets engine fixes all three by creating unique, expiring credentials on demand.

How It Works

App ──"give me postgres creds"──▶ Vault ──CREATE ROLE fog-v-ab12...──▶ PostgreSQL
App ◀─{user, pass, lease=1h}───── Vault
App uses creds for 1h...
        ──lease expires──▶ Vault ──DROP ROLE fog-v-ab12...──▶ PostgreSQL

The database only ever sees Vault as the client creating roles. Applications never learn the admin password; each gets a private role that vanishes on schedule. A credential leaked from logs is worthless within the hour.

Configure It (PostgreSQL Example)

One-time prep on PostgreSQL: give Vault a privileged management account:

-- Run as postgres superuser
CREATE ROLE vault-manager WITH LOGIN SUPERUSER PASSWORD 'V4ult-Mgr-Pass!';

(Use a long random value from Vault itself; scope down from SUPERUSER to CREATEROLE in hardened setups.)

In Vault:

vault secrets enable database

vault write database/config/fogserv-pg \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="website-app" \
    connection_url="postgresql://{{username}}:{{password}}@pg.internal:5432/appdb?sslmode=require" \
    username="vault-manager" \
    password="V4ult-Mgr-Pass!"

vault write database/roles/website-app \
    db_name="fogserv-pg" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
         VALID UNTIL '{{expiration}}'; \
         GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

What Happens: creation_statements is a template Vault executes per credential request — {{expiration}} bakes the expiry into Postgres itself as defense-in-depth, so even if Vault were unreachable at lease-end, the role dies anyway. TTLs cap how long any single credential may live.

Request Credentials

vault read database/creds/website-app

Output:

lease_id        database/creds/website-app/aBc123XyZ
lease_duration  1h
username        v-token-website-ap-Ab12Cd34
password        A1a-B2b-C3c-D4d

Every call returns a different username/password. Test it:

psql "host=pg.internal dbname=appdb sslmode=require" \
     -U v-token-website-ap-Ab12Cd34

Revoke early when needed (e.g., incident response):

vault lease revoke database/creds/website-app/aBc123XyZ   # kills exactly that credential
vault lease revoke -prefix database/creds/website-app     # kill ALL of this role's creds

That last command is your emergency "lock the app out of the database now" switch — something impossible with a shared static password.

Application Consumption Pattern

Apps must handle lease lifecycle: fetch → use → renew while running → refetch on expiry.

// Node/TS sketch
async function getDbCreds() {
  const r = await vault.read('database/creds/website-app');
  return { user: r.data.username, pass: r.data.password,
           leaseId: r.lease_id, ttl: r.lease_duration };
}

// Renew at ~2/3 of TTL while healthy; on renewal failure, fetch fresh creds
setInterval(async () => {
  try { await vault.write(`sys/leases/renew`, { lease_id: leaseId }); }
  catch { ({ user, pass } = await getDbCreds()); /* rebuild pool */ }
}, ttlSecs * 1000 * 0.66);

In practice, use a client library (node-vault, spring-cloud-vault, hvac) or Vault Agent templating so renew/refetch is handled outside business logic.

Other Engines Worth Knowing (Same Lease Model)

Engine	What it mints
AWS	Temporary IAM access keys
PKI	X.509 certs with short validity (pairs with letsencrypt-automation)
SSH	Signed SSH certificates (replaces authorized_keys sprawl)
Transit	Not a secret source — encrypt/decrypt data via API without storing keys

---

Rotation Strategy

Dynamic secrets mostly eliminate rotation anxiety, but static values still exist (third-party API keys, SMTP creds). Make rotation mechanical:

1. Inventory — vault kv list per environment; anything not in Vault is flagged.
2. Dual-write window — write new value as new KV version; consumers re-read on restart.
3. Cutover — deploy services reading latest version.
4. Verify & destroy old versions once all consumers migrated.

For databases already sharing static passwords before Vault adoption: migrate consumers to database/creds/* first, then change the legacy password once, breaking nothing because nobody uses it anymore.

---

Troubleshooting & Common Issues

Symptom	Cause	Fix
404 reading secret/data/x but kv put worked	Mixing KV v1/v2 API paths	v2 needs /data/ segment over raw HTTP
1 error occurred: * unsupported path on kv get	Engine not mounted at that path	vault secrets list; enable kv-v2
Dynamic creds fail login on PG	Role created but grants missing / SSL mismatch	Check PG logs; verify sslmode matches server config
App crashes at lease expiry	No renewal/refetch logic	Add renewal loop or adopt Vault Agent
lease not found on renew	Lease already expired	Fetch new creds — don't treat as fatal error
Can't see version history	Reading data path instead of metadata	GET secret/metadata/<path>?list=true

Inspect lease state directly:

vault list sys/leases/lookup/database/creds/
vault write sys/leases/lookup lease_id="database/creds/website-app/aBc123XyZ"

---

🔗 Related

• Previous: vault-authentication
• Next: rbac-basics — who may read which of these paths
• identity-management — SSO feeding human access to Vault UI
• encrypted-backups — protecting Vault raft backups at rest
• ../infrastructure/secrets-in-iac — Terraform/Ansible reading these secrets
• ../databases/prisma-connections — consuming dynamic DB creds in Prisma apps

---

Change Log

• 2026-08-14 — Initial lesson created as part of KB course build-out (security Phase 4).