fogserv.cloud fogserv.cloud

Backup Automation - Systemd Timers, Notifications, and Restore Testing

Status: Active
Last Updated: 2026-08-26
Category: Cloud - Backup Operations
Prerequisites: backup-to-object-storage, system-admin-basics
Time: 2-3 hours
Tags: automation, systemd-timers, notifications, uptime-kuma, restore-testing

Summary

Turning individual backup scripts into an unattended, observable system: systemd timer units for scheduling, lock handling to prevent overlapping runs, failure notification hooks into the fogserv.cloud monitoring stack (Uptime Kuma / simple alerts), and a calendarized verify-and-restore testing cadence. The principle throughout: a backup that has never been restored, and whose failure nobody notices, does not exist.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Why systemd Timers over Cron
  2. Timer & Service Units
  3. Locking & Overlap Prevention
  4. Notification Hooks
  5. Verification Cadence
  6. The Backup Registry

Context / Why This Matters

backup-to-object-storage produced solid scripts for restic against MinIO — but a script sitting in /opt/restic-scripts/ runs only when someone remembers it, fails silently, and is never proven restorable. This article closes those three gaps with scheduling (storage-backup-strategies' RPO made mechanical), notification, and rehearsal.

Implementation / Core Content

Why systemd Timers over Cron

For backup workloads, systemd timers beat crontab on every dimension that matters:

Timer & Service Units

One service+timer pair per backup job:

# /etc/systemd/system/restic-nextcloud.service
[Unit]
Description=Restic backup: Nextcloud files + DB to MinIO
Wants=network-online.target
After=network-online.target docker.service

[Service]
Type=oneshot
EnvironmentFile=/etc/restic-env
ExecStartPre=/opt/restic-scripts/dump-nextcloud-db.sh
ExecStart=/usr/bin/restic backup /srv/nextcloud/data \
    --tag nextcloud --tag auto
ExecStartPost=/usr/bin/restic forget --keep-daily 7 --keep-weekly 4 \
    --tag auto --group-by paths,tags
User=root
Nice=10
IOSchedulingClass=idle
# Locking: refuse to start if another instance holds the lock
RuntimeDirectory=restic-lock

# /etc/systemd/system/restic-nextcloud.timer
[Unit]
Description=Nightly Nextcloud restic backup

[Timer]
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=10m       # don't hammer MinIO at exactly 02:00 from every host
Persistent=true              # catch up after downtime
Unit=restic-nextcloud.service

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now restic-nextcloud.timer
systemctl list-timers 'restic-*'        # confirm next trigger time

Weekly maintenance (prune + check) gets its own pair with OnCalendar=Sun *-*-* 04:00:00 running restic forget --prune and restic check --read-data-subset=10%.

Useful operational commands:

sudo systemctl start restic-nextcloud.service   # run now (test!)
journalctl -u restic-nextcloud.service -n 100
systemctl show restic-nextcloud.timer -p LastTriggerUSec -p NextElapseUSecRealtime

Locking & Overlap Prevention

Overlapping restic runs cause lock errors at best, repo churn at worst. Two layers:

  1. restic's own lock: exclusive by default; a second run fails with "repository already locked". Fine, but the failure looks like a backup failure.
  2. systemd-level serialization for jobs sharing a repo or heavy I/O:
# In each backup .service sharing infrastructure:
[Service]
...
ExecStartPre=/usr/bin/flock -n /run/backup-infra.lock true

Better: give all backup units the same slice so they queue instead of colliding:

systemd-run --scope --slice=backup.slice systemctl start restic-immich.service

Or simplest robust pattern — a wrapper used as ExecStart:

#!/usr/bin/env bash
exec 9>/run/locks/restic-global.lockflock || exit 75
flock -n 9 || { echo "another backup is running; skipping"; exit 0; }
exec "$@"

Exit-code semantics matter for alerting: distinguish "ran fine" (0), "skipped due to overlap" (0 with message), "real failure" (non-zero).

Notification Hooks

Failures must reach humans within minutes of RPO breach risk. fogserv.cloud uses two complementary paths.

Path 1: push heartbeat to Uptime Kuma (uptime-kuma-setup). Create a Push monitor per job ("nextcloud-backup", expected period 26 h). Last line of every backup unit:

ExecStartPost=/bin/sh -c 'curl -fsS -m 10 --retry 2 \
  "https://kuma.fogserv.cloud/api/push/<token>?status=up&msg=$(date +%s)&ping="' \
  || logger -t restic "WARNING: heartbeat push failed"

If the nightly job dies entirely, the monitor goes "down" after its grace period → existing alert routing fires (see ../observability/simple-alerts.md). Heartbeat beats log-scraping because it catches silence, not just errors.

Path 2: explicit failure hook inside the script for immediate detail:

#!/usr/bin/env bash
# /opt/restic-scripts/notify.sh <service> <exit-code>
SERVICE=$1; RC=$2
if [ "$RC" -ne 0 ]; then
  journalctl -u "$SERVICE" -n 40 --no-pager \
    | curl -fsS -m 10 -H 'Content-Type: application/json' \
        -d "$(jq -nc --arg t "BACKUP FAIL: $SERVICE (rc=$RC)" \
                   --arg c "$(cat)" '{text: ($t + "\n" + $c)}')" \
        "$ALERT_WEBHOOK_URL"
fi

Wire via OnFailure= in the service unit — systemd's built-in failure hook:

[Unit]
OnFailure=notify-failure@%n.service

# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Report failure of %i

[Service]
Type=oneshot
EnvironmentFile=/etc/alert-env
ExecStart=/opt/restic-scripts/notify.sh %i 1

OnFailure fires on any non-zero exit, timeout, or signal — one mechanism covering everything, no per-script try/catch needed.

Verification Cadence

Three escalating levels, all calendarized:

Level What Command Frequency
Structural repo metadata intact restic check after every prune (weekly)
Data sampling read 10% of blobs restic check --read-data-subset=10% weekly
Full read every blob verified restic check --read-data quarterly
Restore drill actually restore + diff see below monthly (1 random service)

Restore drill template (rotate through services: nextcloud, immich, jellyfin config, syncthing mirrors):

#!/usr/bin/env bash
set -euo pipefail
source "/etc/restic-${1}.env"
DEST=/tmp/drill-$1-$(date +%F)
time restic restore latest --target "$DEST"
echo "Drill $1: $(du -sh "$DEST" | cut -f1) restored, RTO above."
rm -rf "$DEST"

Record in the ops journal: date, service, duration, anomalies. The measured number is your demonstrated RTO; compare against targets from storage-backup-strategies. A drill without a recorded duration proves nothing about recovery time.

Automate the reminder rather than the discipline: a monthly timer that runs one drill and pushes the result as a heartbeat keeps the cadence honest even when humans forget.

The Backup Registry

Keep a single file (/opt/restic-scripts/registry.tsv) listing every protected dataset — this is what you check during audits and incidents:

service     rpo_target  schedule      repo                        last_drill
nextcloud   24h         daily 02:15   restic-nextcloud@minio      2026-08-12 OK
immich      24h         daily 03:00   restic-immich@minio         2026-07-30 OK
jellyfin    7d          weekly Sun    restic-jellyfin@minio       2026-08-05 OK
syncthing   24h         daily 02:45   restic-syncthing@minio      never  ← TODO

Review it monthly alongside timer status:

systemctl list-timers 'restic-*' --all | awk '{print $1, $NF}'

Any row where last_drill is older than a quarter, or missing entirely, is an action item.

Practical Examples

Example 1: Complete rollout for a new service

# Given: working script /opt/restic-scripts/jellyfin.sh and env /etc/restic-jellyfin.env
cp templates/restic-job.service{,.tmpl}  # sed service name, ExecStart path
systemctl link /etc/systemd/system/restic-jellyfin.{service,timer}
systemctl enable --now restic-jellyfin.timer
systemctl start restic-jellyfin.service && journalctl -u restic-jellyfin -f  # first run supervised
curl -fsS "https://kuma.fogserv.cloud/api/push/<token>"   # confirm heartbeat registered

Example 2: Simulate failure to prove alerting works

sudo systemctl stop minio-server            # controlled outage window!
sudo systemctl start restic-nextcloud.service
systemctl status restic-nextcloud.service   # expect failure
# Expect: OnFailure hook fired → webhook message received;
#         Uptime Kuma goes down after grace period.
sudo systemctl start minio-server           # end outage window

An alert pipeline never tested with a real failure is a hypothesis, not a control.

Example 3: Catch-up behavior after downtime

sudo shutdown -h now    # Friday evening maintenance
# boot Monday morning
systemctl list-timers | grep restic
# Persistent=true shows the missed job ran shortly after boot — verify in journalctl.

Troubleshooting & Common Pitfalls

Problem Cause Fix
Timer enabled but never triggers Forgot daemon-reload; timer not started systemctl enable --now; check list-timers
Missed runs while server was off Cron semantics; Persistent unset Set Persistent=true in timer
repository is already locked alerts Overlapping runs (stale lock or slow prune) Stale lock: restic unlock; structural: serialize via flock/slice
Alerts silent during failures Only success-heartbeats implemented; no OnFailure Add OnFailure= hook + push monitors
Backup green but repo dead No checks/drills ever run Weekly --read-data-subset, quarterly full read + drills
Jobs stampede MinIO at same minute Identical OnCalendar across hosts RandomizedDelaySec per unit
Journal floods disk Verbose restic output at level info Keep logs; set LogRateLimitIntervalSec in unit if extreme

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse