Load Balancing - Distributing Traffic Across the Fleet

Status: Active
Last Updated: 2026-08-26
Category: Networking - Phase 2: Services
Prerequisites: reverse-proxy-basics, tcp-ip-fundamentals, nginx-configuration
Time: 3 hours
Tags: load-balancing, haproxy, keepalived, upstream, health-checks, l4, l7

Summary

Understand Layer 4 vs Layer 7 load balancing and implement both in the homelab: nginx/HAProxy upstream pools with active health checks, and keepalived VRRP for a floating VIP so a proxy host can fail without DNS changes.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Choose between L4 (TCP) and L7 (HTTP) balancing
✅ Configure upstream pools with weights, health checks, and keepalive
✅ Run HAProxy with a stats page
✅ Provide a highly available entry point with keepalived + VRRP

Context / Why This Matters
Implementation / Core Content
Practical Examples
Troubleshooting & Common Pitfalls
Next Steps / Ops Actions

Context / Why This Matters

Most fogserv services are single-replica containers behind one reverse proxy, which is fine — until you run something stateless with two or more replicas, or you want the proxy itself to survive a host reboot/maintenance. This article covers both: distributing traffic across backends, and removing the proxy as a single point of failure. For container-native distribution, also read docker-networking.

Implementation / Core Content

L4 vs L7 at a glance

	L4 (TCP/UDP)	L7 (HTTP/HTTPS)
Decides on	IP + port	URL, headers, cookies
TLS termination	Passthrough (or terminate)	Usually terminates
Typical use	Databases, game servers, MQTT	Web apps, APIs
Cost	Very cheap	Slightly higher (parsing)

Nginx upstream pool (L7)

upstream forgejo_pool {
    least_conn;
    server 192.168.20.10:3000 weight=2 max_fails=3 fail_timeout=15s;
    server 192.168.20.11:3000 max_fails=3 fail_timeout=15s;
    server 192.168.20.12:3000 backup;
    keepalive 32;
}

server {
    server_name git.fogserv.cloud;
    location / {
        proxy_pass http://forgejo_pool;
        proxy_next_upstream error timeout http_502;
        proxy_set_header Connection "";
    }
}

Nginx open source has passive health checks only (max_fails/fail_timeout): a backend is skipped after failures and retried after the timeout.

HAProxy with active health checks (preferred for real balancing)

sudo apt install haproxy

/etc/haproxy/haproxy.cfg:

frontend web
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/fogserv.pem alpn h2,http/1.1
    default_backend app_pool

backend app_pool
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 5s fall 3 rise 2
    server app1 192.168.20.10:8080 check
    server app2 192.168.20.11:8080 check

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    # Restrict 8404 to management subnet via nftables!

sudo haproxy -c -f /etc/haproxy/haproxy.cfg   # validate
sudo systemctl reload haproxy
curl -u admin:secret http://lb1.fogserv.cloud:8404/stats

Algorithms: roundrobin (default), leastconn (long-lived connections), source (client-IP stickiness). For sticky sessions use a cookie: cookie SRV insert indirect nocache plus cookie SRV on each server line.

keepalived floating VIP

Give the pair lb1 (192.168.20.5) and lb2 (192.168.20.6) a shared VIP 192.168.20.4 that always lives on whichever node is master. Point internal DNS at the VIP once and never care which proxy is up.

Both nodes, /etc/keepalived/keepalived.conf on lb1 (priority 150; lb2 uses 100):

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight -30
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP            # both BACKUP; priority decides master
    interface ens18
    virtual_router_id 51    # must match on both nodes
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ChangeMe51
    }
    virtual_ipaddress {
        192.168.20.4/24
    }
    track_script {
        chk_haproxy
    }
}

sudo apt install keepalived
# Allow VRRP through nftables:
sudo nft add rule inet filter input ip protocol vrrp accept

Practical Examples

Prove failover works

# From a client, continuously hit the VIP
while true; do curl -s -o /dev/null -w '%{http_code}\n' http://192.168.20.4/healthz; sleep 1; done
# On lb1: sudo systemctl stop haproxy
# Expect at most ~3-4 failed seconds while lb2 takes the VIP.

Drain one backend for maintenance

HAProxy: set the server in maintenance mode via the runtime API:

echo "disable server app_pool/app1" | sudo socat stdio /run/haproxy/admin.sock
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | grep app_pool

Simulate an unhealthy backend

Stop the app on app1 and watch the HAProxy stats page flip it to DOWN within ~15s (inter 5s × fall 3); traffic continues on app2 with no client-visible errors.

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Both nodes claim the VIP (split-brain)	VRRP adverts blocked, VRID mismatch	Open protocol vrrp in nftables; verify identical `virtual_router_id`
VIP doesn't move on failure	No health tracking	Add `track_script` checking haproxy/nginx
Session lost on every request	Round-robin without stickiness	Use `least_conn` + cookie insertion or `balance source`
Backend flaps UP/DOWN	Health check too aggressive	Increase `fall`/`rise`, lengthen `inter`
502 after adding backend	Backend answers on wrong port/interface	`curl` it directly from the LB host first
Stats page exposed publicly	Port 8404 not firewalled	Allow 8404 only from mgmt subnet in nftables

Next Steps / Ops Actions

Segment LB traffic from management: network-segmentation.
Monitor backend health from Prometheus: prometheus-basics and alert on DOWN servers with alertmanager-config.
Terminate TLS consistently per tls-configuration.

Sources & Related

External references consulted:

Change Log

2026-08-26

Initial creation by KB build session.