fogserv.cloud fogserv.cloud

Scheduler Patterns - cron, systemd Timers, and Safe Job Design

Status: Active
Last Updated: 2026-08-26
Category: Sysadmin - Operations
Prerequisites: system-admin-basics, log-management
Time: 1-2 hours
Tags: cron, systemd timers, flock, scheduling, backups, automation

Summary

How to schedule recurring work on Linux reliably: choosing between cron and systemd timers, preventing overlapping runs with flock, handling missed jobs, and applying the patterns to backups and maintenance windows.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Context / Why This Matters
  2. cron vs systemd Timers
  3. Locking with flock
  4. Missed-Job Handling
  5. Scheduling Backups and Maintenance

Context / Why This Matters

Almost every operational task worth doing twice is worth scheduling: patch audits from system-admin-basics, log vacuums from log-management, database dumps. But naive scheduled jobs fail in three predictable ways: they overlap themselves when a run is slow, they silently skip runs while the machine was down, and they produce no output because cron discards stdout into a void you never read.

This article gives the standard defenses for all three failure modes.

Implementation / Core Content

1. cron vs systemd Timers

Aspect cron systemd timer
Syntax crontab, 5 fields + command .timer + .service unit pair
Missed-run catch-up No (anacron partially) Yes: Persistent=true
Dependencies None Full (After=, Wants=, network wait)
Logging Mail to local mbox (usually unread) journald — query with journalctl -u myjob.service
Randomized delay Manual sleep hacks RandomizedDelaySec= built in
Per-user jobs Native (crontab -e) User units + loginctl enable-linger

Decision rule: use systemd timers for anything system-level or anything you need to observe via log-management; keep cron for simple per-user tasks.

A systemd timer pair:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Nightly database backup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target

# /etc/systemd/system/db-backup.service
[Unit]
Description=Database backup job
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/db-backup.sh

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now db-backup.timer
systemctl list-timers --all          # next run times, last result
systemctl cat db-backup.timer        # confirm what's loaded

cron equivalents and calendar expressions:

# crontab -e : 02:30 daily, log to file explicitly
30 2 * * * /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

# Test systemd OnCalendar syntax before committing
systemd-analyze calendar "*-*-* 02:30:00"
systemd-analyze calendar "Mon..Fri *-*-* 06,18:00:00"

Useful OnCalendar forms:

Expression Meaning
daily 00:00 every day
*-*-* 02:30:00 02:30 daily
Mon *-*-* 04:00:00 Mondays 04:00
*-*-01 03:00:00 First of month 03:00
*:*:00/15 Every 15 minutes

2. Locking with flock

If a run can outlast its interval (backup slows down, API hangs), the next trigger starts a second copy: double writes, corrupted outputs, resource exhaustion. The fix is one line:

# In a crontab:
*/15 * * * * flock -n /tmp/report.lock /usr/local/bin/report.sh >> /var/log/report.log 2>&1

flock -n takes an exclusive lock on the lockfile or exits immediately if held — so the second invocation just skips. For systemd services, put it in the ExecStart:

[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /run/db-backup.lock /usr/local/bin/db-backup.sh

Variants:

3. Missed-Job Handling

Three scenarios:

  1. Machine was off/asleep → cron simply doesn't fire. Fix with Persistent=true (systemd) — the timer fires immediately after boot if its last scheduled run was missed. For classic cron, anacron covers daily/weekly/monthly granularity.
  2. Job failed mid-run → the fix belongs in the job itself: retries with backoff, and idempotent design (see failure-recovery-patterns). A scheduler retries nothing.
  3. You don't know either happened → observability gap. Check:
systemctl list-timers --all                  # LAST column = last trigger time
journalctl -u db-backup.service --since yesterday   # what did it do/log
grep CRON /var/log/syslog                    # cron-side evidence (Debian)

Rule: every scheduled job writes a heartbeat — touch a timestamp file on success and alert if it goes stale (see simple-alerts):

#!/bin/bash
# tail of db-backup.sh
/usr/local/bin/db-dump.sh && touch /var/lib/fogserv/heartbeat/db-backup.ok \
  || { echo "db-backup FAILED" | logger -t db-backup -p user.err; exit 1; }

4. Scheduling Backups and Maintenance

Backups

# Stagger across hosts to avoid hammering shared storage:
# host A 01:00, host B 01:20, host C 01:40
OnCalendar=*-*-* 1:20:00

Maintenance windows

# Unattended upgrades: only in the early morning, with randomized start
OnCalendar=*-*-* 04:00:00
RandomizedDelaySec=30m

Practical Examples

Example 1: Fully-specified nightly backup timer

sudo tee /etc/systemd/system/pg-backup.service >/dev/null <<'EOF'
[Unit]
Description=Postgres nightly backup
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /run/pg-backup.lock /usr/local/bin/pg-backup.sh
RuntimeMaxSec=4h
EOF
sudo tee /etc/systemd/system/pg-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Run pg-backup nightly at 02:30

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now pg-backup.timer
systemctl list-timers pg-backup.timer

Expected: NEXT shows tomorrow ~02:30 (+ jitter), LAST shows - until first run.

Example 2: Cron job that must never overlap

crontab -e
# every 5 min, skip if previous run still going, always log
*/5 * * * * flock -n /run/sync.lock /opt/bin/sync.sh >> ~/.sync.log 2>&1

Test overlap behavior manually: flock -n /run/sync.lock sleep 300 &, then re-run the cron command — it should exit instantly.

Troubleshooting & Common Pitfalls

Problem Cause Fix
Cron job never ran Host down at trigger time Move to systemd timer with Persistent=true
Ran but output vanished cron mailed it to an empty mailbox Append >> log 2>&1 or switch to journald via timers
Multiple copies running Slow run + short interval Wrap in flock -n
Timer never fires Service/timer name mismatch, or not enabled systemctl list-timers --all; check daemon-reload after edits
Job works manually, fails under cron Different env/PATH, relative paths Use absolute paths; set PATH explicitly in script
Failed to start ... Unit is masked Leftover mask from hardening systemctl unmask <unit>
Backfill storm after long downtime Persistent=true fires many missed slots? No — it fires once Understand: persistent fires one catch-up run, not N

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse