fogserv.cloud fogserv.cloud

Log Management - journald, Rotation, and Central Shipping

Status: Active
Last Updated: 2026-08-26
Category: Sysadmin - Operations
Prerequisites: system-admin-basics, loki-logging
Time: 2 hours
Tags: logs, journald, logrotate, rsyslog, retention, forensics

Summary

Covers how Linux systems produce, store, rotate, and ship logs: configuring journald, controlling disk usage with rotation and retention policies, options for shipping logs to a central store, and the quick commands you need during an incident.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Context / Why This Matters
  2. journald Configuration
  3. Rotation with logrotate
  4. Central Shipping Options
  5. Retention Policy
  6. Forensics Quick Reference

Context / Why This Matters

Logs are the first evidence source in every incident and the last thing anyone configures properly. On a default systemd host, /var/log/journal may not even exist — meaning every reboot destroys your history exactly when you need it to explain a crash. Unbounded logs also fill disks quietly, which takes down services that have nothing to do with logging.

This article pairs with metrics-vs-logs (when to log vs. when to measure) and loki-logging (where shipped logs go). Read system-admin-basics first for the inventory and audit discipline these policies feed into.

Implementation / Core Content

1. journald Configuration

Config lives in /etc/systemd/journald.conf (or drop-ins in /etc/systemd/journald.conf.d/*.conf). The single most important setting is persistence:

# /etc/systemd/journald.conf.d/99-fogserv.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
MaxFileSec=30day
ForwardToSyslog=no

Apply and verify:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --verify          # integrity check
du -sh /var/log/journal      # current footprint

Early-boot caveat: journald always starts in volatile mode, even with Storage=persistent, until systemd-journal-flush.service completes. A crash during that window loses messages regardless of configuration. Treat the first ~30 seconds of boot as best-effort and pair journald with console= kernel logging or an out-of-band shipper if early-boot forensics matter.

Key settings explained:

Setting Meaning
Storage=persistent Logs survive reboot (stored under /var/log/journal)
SystemMaxUse=2G Hard cap on disk usage; oldest journals vacuumed first
SystemKeepFree=1G Never consume the last 1G of free space
MaxFileSec=30day Time-based retention on top of size-based
Seal=yes (default) Cryptographic sealing — detect tampering after the fact

For tamper-evidence on security-relevant hosts, enable forward-secure sealing:

sudo journalctl --setup-keys    # generates FSS key pair; keep secret key off-host

2. Rotation with logrotate

journald rotates itself; plain-text application logs do not. Anything an app writes directly to /var/log/<app>.log needs logrotate. Configs live in /etc/logrotate.d/:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    maxsize 100M
    compress
    delaycompress
    missingok
    notifempty
    dateext
    copytruncate
}

3. Central Shipping Options

One host's logs are anecdotes; a fleet's logs are a dataset. Pick one shipper per host:

Tool Fit Notes
rsyslog Already installed everywhere RFC5424 forwarding over TCP/TLS; lowest effort
Grafana Alloy Grafana/Loki stack Successor to Promtail; Promtail reached EOL 2026-03-02 and is now in maintenance mode. New deployments should use Alloy
Promtail Legacy Loki stacks Still works but no security fixes — migrate to Alloy on the next host rebuild
Vector High-throughput, transforms in-flight TOML/VRL config, very efficient; vendor-agnostic
journald + systemd-journal-remote Pure-systemd shops Push/pull structured journal files

Promtail EOL (2026-03-02): If any host still ships via Promtail, plan a migration to Grafana Alloy — the configuration shape is similar, and Alloy unifies metrics, logs, and traces in one agent. See Grafana's Alloy documentation for the conversion.

Minimal Loki example (matches our loki-logging stack):

# /etc/alloy/config.alloy (Promtail-compatible scrape of journald)
local.file_match "logs" {
  path_targets = [{ __path__ = "/var/log/myapp/*.log", job = "myapp" }]
}
loki.source.file "app" {
  targets    = local.file_match.logs.targets
  forward_to = [loki.write.default.receiver]
}
loki.write "default" {
  endpoint { url = "http://loki.internal:3100/loki/api/v1/push" }
}

rsyslog fallback for hosts without agents:

# /etc/rsyslog.d/60-forward.conf
*.* action(type="omfwd" target="syslog.internal" port="514"
           protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format")

Rule of thumb: ship structured copies centrally, but never rely on the pipeline as the only copy. Local journald retention stays in place as a buffer.

4. Retention Policy

Define retention per log class, not globally. A starting policy:

Class Examples Local Central Rationale
Security/auth auth.log, sudo, sshd 30 days 1 year Incident forensics window
Application service stdout 14 days 90 days Debugging recent regressions
System/kernel journalctl -k 30 days 90 days Crash correlation
Access/audit nginx access, auditd 7 days 180 days Volume-heavy, cheap to re-ingest

Document the chosen numbers in this KB entry and enforce them mechanically (SystemMaxUse, logrotate rotate N, Loki's retention_period). An unenforced policy is a wish.

5. Forensics Quick Reference

# Everything since boot, errors only
journalctl -p err -b

# A unit's logs since yesterday, following
journalctl -u myapp.service --since yesterday -f

# Kernel ring + OOM kills
journalctl -k --grep "Out of memory"

# All activity by UID 1000 today
journalctl _UID=1000 --since today

# SSH login attempts in the last hour
journalctl -u ssh --since "-1h" | grep -i accept

# Disk space used by journals, then vacuum manually
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=14d

# Export a time window for offline analysis / ticket attachment
journalctl --since "2026-08-26 08:00" --until "2026-08-26 09:00" > incident.log

# Which boot had the crash?
journalctl --list-boots

Practical Examples

Example 1: Diagnose why a service died overnight

journalctl --list-boots                      # find the relevant boot
journalctl -u api.service -b -1 --since "23:00" --until "06:00"
# Follow with:
journalctl -k -b -1 --grep oom               # was it killed by the kernel?

Expected pattern: app logs stop at T, kernel OOM line at T+0s → memory limit too low.

Example 2: Prove logrotate works before trusting it

sudo logrotate --debug /etc/logrotate.d/myapp   # dry run, prints plan
sudo logrotate -f /etc/logrotate.d/myapp        # force one rotation now
ls -lh /var/log/myapp/                          # confirm .1 + dateext file exists

Troubleshooting & Common Pitfalls

Problem Cause Fix
Journal empty after reboot Storage=auto and /var/log/journal missing Create dir, set Storage=persistent, restart journald
Disk full from logs No SystemMaxUse; app logs unrotated Set caps; add logrotate rule; vacuum once with --vacuum-size
Rotated log keeps growing in .1 App holds open fd; no HUP/reopen Use copytruncate or add postrotate SIGHUP
Gaps in shipped logs Shipper crashed silently Monitor the shipper itself; alert on ingestion lag in Loki
Can't read journal as non-root User not in adm/systemd-journal group sudo usermod -aG systemd-journal $USER

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

2026-08-26 (Task 7 refresh)

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse