Production Backend — Reliability and Security Patterns

Status: Active | Last Updated: 2026-08-29 Category: Backend — Production Prerequisites: API design; observability basics Tags: production, rate-limit, cors, security-headers, logging, graceful-shutdown Estimated Time: 4-5 hours (Self-paced, includes lab time)

Summary

A backend that works locally can fail in production under load, attack, or even routine server restarts. This article covers the patterns that make a backend production-ready: rate limiting, security headers, structured logging, graceful shutdown, and zero-downtime deploys.

What You'll Learn (Core Competencies)

• Implement token-bucket rate limiting per IP and return 429 with Retry-After
• Apply CORS and security headers to prevent XSS, clickjacking, and MIME sniffing
• Emit structured JSON logs with request IDs, user IDs, status codes, and latency
• Handle SIGTERM with graceful shutdown that drains in-flight requests
• Distinguish liveness probes from readiness probes and configure both

Table of Contents

1. Architectural Overview & Core Schema
2. Deep Dive & Implementation
3. Anti-Patterns & Common Pitfalls
4. Independent Challenge
5. Consolidation & Key Invariants
6. Next Steps

---

1. Architectural Overview & Core Schema

Production backends face threats that local development never sees: DDoS attacks, brute force login attempts, traffic spikes, server restarts during deploys, and security vulnerabilities. Each of these has a known mitigation.

Rate limiting protects against DDoS, brute force attacks, and runaway clients. The token bucket algorithm gives each client a budget of tokens; tokens refill at a constant rate; when the bucket is empty, requests are rejected with 429.

CORS and security headers prevent common web attacks. CORS controls which browser origins can call the API. Security headers (HSTS, CSP, X-Frame-Options) prevent XSS, clickjacking, and protocol downgrade.

Structured logging with request IDs makes distributed systems debuggable. Each request gets a unique ID that flows through all log lines, allowing correlation across services.

Graceful shutdown ensures that SIGTERM (from a load balancer, Kubernetes pod termination, or docker stop) drains in-flight requests before exiting. Without it, users see connection errors during every deploy.

Health checks distinguish "process is alive" (liveness) from "ready to serve traffic" (readiness). Kubernetes uses liveness to decide when to restart, and readiness to decide when to add a pod to the load balancer.

2. Deep Dive & Implementation

Rate Limiting

Token bucket per IP/user. Each request consumes one token; tokens refill at a fixed rate.

const buckets = new Map<string, { tokens: number; lastRefill: number }>();

function rateLimit(ip: string, capacity = 100, refillRate = 10): boolean {
  const now = Date.now();
  const bucket = buckets.get(ip) ?? { tokens: capacity, lastRefill: now };

  const elapsed = (now - bucket.lastRefill) / 1000;
  bucket.tokens = Math.min(capacity, bucket.tokens + elapsed * refillRate);
  bucket.lastRefill = now;

  if (bucket.tokens < 1) {
    buckets.set(ip, bucket);
    return false;  // rate limited
  }

  bucket.tokens -= 1;
  buckets.set(ip, bucket);
  return true;
}

// Middleware wrapper
function withRateLimit(handler: Handler, capacity = 100, refillRate = 10): Handler {
  return async (request) => {
    const ip = request.headers.get("X-Forwarded-For") ?? "unknown";
    if (!rateLimit(ip, capacity, refillRate)) {
      return new Response("Too Many Requests", {
        status: 429,
        headers: { "Retry-After": "1" }
      });
    }
    return handler(request);
  };
}

For multi-instance deployments, replace the in-memory Map with Redis so all instances share the same bucket state. Without Redis, an attacker gets N×capacity by hitting N different instances.

CORS and Security Headers

function securityHeaders(origin: string) {
  return new Headers({
    // CORS
    "Access-Control-Allow-Origin": origin,
    "Access-Control-Allow-Methods": "GET, POST, PUT, DELETE",
    "Access-Control-Allow-Headers": "Content-Type, Authorization",
    // Security
    "Strict-Transport-Security": "max-age=31536000; includeSubDomains", // force HTTPS
    "Content-Security-Policy": "default-src 'self'",                     // prevent XSS
    "X-Content-Type-Options": "nosniff",                                 // prevent MIME sniffing
    "X-Frame-Options": "DENY",                                           // prevent clickjacking
    "Referrer-Policy": "strict-origin-when-cross-origin"
  });
}

Apply these headers to every response, not just authenticated ones. CORS preflight (OPTIONS) requests need CORS headers but should not require auth.

Structured Logging

Use JSON logs with pino or similar. Include requestId, userId, status, latency. Avoid console.log in production — it does not support log levels or structured fields.

import pino from "pino";

const log = pino({
  level: process.env.LOG_LEVEL || "info",
  formatters: {
    level: (label) => ({ level: label })
  }
});

// Request-scoped logger middleware
function withRequestLog(handler: Handler): Handler {
  return async (request) => {
    const id = crypto.randomUUID();
    const start = Date.now();

    log.info({ requestId: id, method: request.method, url: request.url }, "request_start");

    const response = await handler(request);
    const duration = Date.now() - start;

    log.info({
      requestId: id,
      method: request.method,
      url: request.url,
      status: response.status,
      durationMs: duration
    }, "request_end");

    response.headers.set("X-Request-Id", id);
    return response;
  };
}

Structured logs are searchable: kubectl logs ... | jq 'select(.status >= 500)' returns all server errors.

Graceful Shutdown

Listen for SIGTERM. Stop accepting connections, drain in-flight requests, close the database pool, then exit.

const server = Bun.serve({ port: 3000, fetch: handler });

async function shutdown(signal: string) {
  log.info({ signal }, "shutdown_start");
  try {
    // Stop accepting new connections
    await server.stop();
    // Close DB pool
    await prisma.$disconnect();
    log.info({ signal }, "shutdown_complete");
    process.exit(0);
  } catch (err) {
    log.error({ err }, "shutdown_error");
    process.exit(1);
  }
  // Force exit after 10s if drain stalls
  setTimeout(() => {
    log.error("shutdown_timeout");
    process.exit(1);
  }, 10_000).unref();
}

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));

The setTimeout().unref() is a safety net: if drain takes longer than 10 seconds, force exit. Without it, a stuck request can prevent the process from ever terminating.

Health Checks

Liveness: the process is alive and responding. Readiness: dependencies (DB, cache) are reachable.

// Liveness — always returns 200 if the process is up
app.get("/healthz", (_, res) => res.status(200).send("ok"));

// Readiness — checks DB connection
app.get("/readyz", async (_, res) => {
  try {
    await prisma.$queryRaw`SELECT 1`;
    res.status(200).send("ready");
  } catch (err) {
    log.error({ err }, "readiness_check_failed");
    res.status(503).send("not ready");
  }
});

Kubernetes uses /healthz to decide when to restart a pod (if liveness fails, kill and restart). It uses /readyz to decide when to add the pod to the load balancer (only ready pods receive traffic).

Zero-Downtime Deploys

Use rolling updates (Kubernetes) or blue-green (Nginx upstream swap). Configure the orchestrator to:

1. Send SIGTERM to the old pod with a 30-second grace period
2. Wait for /readyz to return 503 on the old pod before stopping traffic
3. Start the new pod, wait for /readyz to return 200
4. Direct traffic to the new pod, terminate the old pod

This sequence ensures that requests in flight when the deploy starts are completed on the old pod, while new traffic goes to the new pod.

Guided Checkpoint

Verify the production patterns:

# Start: bun run server.ts

# Rate limiting — burst 100 requests then 429
for i in $(seq 1 105); do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000/api/data
done | sort | uniq -c
# Should show: ~100 × 200, ~5 × 429

# Rate limit response — has Retry-After header
curl -s -I http://localhost:3000/api/data | grep -i retry-after

# Security headers — present on all responses
curl -s -I http://localhost:3000/api/data | grep -E "(Strict-Transport|Content-Security|X-Frame|X-Content-Type)"

# CORS preflight — OPTIONS returns CORS headers
curl -s -X OPTIONS http://localhost:3000/api/data \
  -H "Origin: http://example.com" -I | grep -i access-control

# Structured logs — JSON output with requestId and duration
curl -s http://localhost:3000/api/data > /dev/null
# Check the console for {"level":"info","requestId":"...","status":200,"durationMs":3}

# X-Request-Id header on every response
curl -s -I http://localhost:3000/api/data | grep -i x-request-id

# Liveness — always 200
curl -s -w "\n%{http_code}" http://localhost:3000/healthz

# Readiness — 200 when DB up, 503 when DB down
curl -s -w "\n%{http_code}" http://localhost:3000/readyz

# Graceful shutdown — SIGTERM should drain, then exit
# Run in background: bun run server.ts &
# Send: kill -TERM $!
# Should see "shutdown_start" then "shutdown_complete", exit 0

The rate limit test should show 100 successful requests and 5 rate-limited (429) responses. Security headers must be present on every response, not just API routes.

3. Anti-Patterns & Common Pitfalls

No rate limit on auth endpoints. Login, password reset, and token endpoints must have the tightest rate limits. Without them, brute force attacks can crack passwords. Apply a 5-req/min limit on /auth/login and /auth/register.

Logging sensitive data. Never log raw Authorization headers, session cookies, password fields, or credit card numbers. Use field allowlists or hash sensitive values before logging.

Returning stack traces in API responses. Stack traces leak file paths, library versions, and code structure. In production, log the stack trace but return only { error: "Internal Server Error" }.

Using console.log in production. console.log does not support log levels, structured fields, or log routing. Use pino or winston for production. Disable debug logging via LOG_LEVEL=info.

Confusing liveness and readiness. If /readyz returns 503 because the DB is down, Kubernetes will stop sending traffic but will not restart the pod (because liveness is still 200). When the DB recovers, the pod is ready again. If /healthz returns 503 because the DB is down, Kubernetes kills and restarts the pod — but the DB is still down, so the new pod also fails. This causes restart loops. Liveness must not depend on external systems.

No Retry-After header on 429. Clients need to know when to retry. Without Retry-After, they retry immediately, defeating the rate limit. Always include Retry-After (in seconds).

Graceful shutdown that ignores in-flight requests. A process.exit(0) that runs while requests are in flight closes TCP connections mid-response. Always await server.stop() (drain) before exiting.

Missing security headers on error responses. A 500 Internal Server Error response without X-Content-Type-Options: nosniff can be MIME-sniffed by browsers to execute as JavaScript. Apply security headers in a final middleware, not per route.

Wide-open CORS (Access-Control-Allow-Origin: *) with credentials. Browsers reject this combination. If you need credentials, allow only specific origins, not *.

4. Independent Challenge

Make a backend production-ready by adding the following:

1. Rate limiting middleware that returns 429 with Retry-After after 50 requests in 60 seconds per IP
2. Security headers middleware that adds HSTS, CSP, X-Frame-Options, X-Content-Type-Options to every response
3. CORS middleware that allows http://localhost:3001 for development and reads ALLOWED_ORIGINS env var for production
4. Structured logging with pino that emits JSON with requestId, userId (from auth), method, path, status, durationMs
5. /healthz (liveness) returns 200 always
6. /readyz (readiness) checks the database connection and returns 503 on failure
7. Graceful shutdown on SIGTERM that drains in-flight requests, closes the DB pool, then exits with code 0

Use the patterns from this lesson. Test with curl to verify each pattern. Use the kill signal to test graceful shutdown.

5. Consolidation & Key Invariants

• Return 429 with a Retry-After header for rate-limited requests; clients need to know when to retry
• Apply security headers (HSTS, CSP, X-Frame-Options, X-Content-Type-Options) to every response, including error responses
• Use pino or winston for structured JSON logging; never use console.log in production
• Liveness probes must not depend on external systems (DB, cache); only check process health
• Readiness probes must check external dependencies; they decide load balancer membership
• Listen for SIGTERM and SIGINT; drain in-flight requests before closing the DB pool and exiting
• Use Redis-backed rate limiting in multi-instance deployments; an in-memory Map is per-instance
• Never log raw Authorization headers, cookies, or password fields — use allowlists or hash before logging
• Return generic error messages in API responses (Internal Server Error); log full stack traces server-side

6. Next Steps

• Add distributed tracing with OpenTelemetry for cross-service request tracking.
• Implement circuit breakers for downstream service calls to prevent cascade failures.
• Set up log aggregation (ELK stack, Loki, Datadog) for centralized log analysis.
• Configure autoscaling on CPU usage or requests-per-second metrics.
• Read kb/backend/middleware-patterns.md for composing rate limiting, CORS, and logging middleware.
• Read kb/backend/authentication-patterns.md for auth-specific rate limiting patterns.

---

Change Log

• 2026-08-29: Initial scaffold
• 2026-08-29: Expanded logging, shutdown, health, deploys, code examples
• 2026-08-29: Migrated to 6-section canonical template (2026-08-29)