fogserv.cloud fogserv.cloud

Alertmanager Config - Routing, Grouping, Silences, and Escalation

Status: Active
Last Updated: 2026-08-26
Category: Observability - Alerting
Prerequisites: simple-alerts, prometheus-basics
Time: 2-3 hours
Tags: alertmanager, routing, silences, inhibition, telegram, webhook, escalation

Summary

Alertmanager takes the firing alerts Prometheus rules produce and decides who hears about them, how fast, and how often. This article covers the routing tree, grouping and repeat intervals, silences vs inhibition, receiver setup for email/webhook/Telegram, and escalation patterns that keep 2 a.m. pages meaningful — building directly on the rules written in simple-alerts.md.

What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Context / Why This Matters
  2. Implementation / Core Content
  3. Practical Examples
  4. Common Pitfalls & Troubleshooting
  5. Next Steps / Ops Actions

Context / Why This Matters

simple-alerts.md got alert rules firing in Prometheus. Out of the box those go nowhere useful: without Alertmanager you get no deduplication, no grouping, no "stop yelling during planned maintenance," and no way to route a disk warning somewhere different from an API outage. Alertmanager is the state machine between "condition is true" and "a human knows."

The core mental model: rules decide what matters; Alertmanager decides who cares. Keep severity/routing metadata on alerts as labels at rule-definition time — routing then becomes configuration, not rewrites.

Implementation / Core Content

Wiring Prometheus to Alertmanager

In prometheus.yml:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - /etc/prometheus/rules/*.yml

And every rule from simple-alerts.md should carry routing labels:

- alert: HostDiskWillFillIn14Days
  expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[7d], 14*24*3600) < 0
  for: 30m
  labels:
    severity: warning
    area: infra          # ← used by the router
  annotations:
    summary: "Disk on {{ $labels.instance }} fills within ~14 days"

The routing tree

Alerts enter at the root receiver, then traverse child routes depth-first; first matching child wins (children are checked before continuing siblings only via continue: true). Unmatched alerts fall back to the parent's receiver.

Full worked config (alertmanager.yml):

global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "alerts@fogserv.cloud"
  smtp_auth_username: "alerts@fogserv.cloud"
  smtp_auth_password_file: /etc/alertmanager/secrets/smtp-password

templates:
  - "/etc/alertmanager/templates/*.tmpl"

route:
  receiver: email-default          # fallback: nothing matched anywhere
  group_by: ["alertname", "instance"]
  group_wait: 30s                  # collect siblings before first notify
  group_interval: 5m               # min gap between notifies of same group
  repeat_interval: 4h              # re-notify while still firing

  routes:
    # 1. Page-worthy problems → immediate, everywhere
    - matchers: [severity="critical"]
      receiver: telegram-oncall
      group_wait: 10s
      repeat_interval: 1h
      continue: true               # ALSO send to default for the record

    # 2. Infra warnings → email, relaxed cadence
    - matchers: [severity="warning", area="infra"]
      receiver: email-infra
      repeat_interval: 12h

    # 3. App-level warnings → webhook into your ticket bot
    - matchers: [severity="warning", area="apps"]
      receiver: webhook-tickets
      repeat_interval: 8h

    # 4. Nightly noisy-but-harmless batch jobs: only page during work hours
    - matchers: [job="batch-jobs"]
      receiver: email-default
      active_time_intervals: [work-hours]
      repeat_interval: 24h

time_intervals:
  - name: work-hours
    time_intervals:
      - weekdays: ["monday:friday"]
        times:
          - start: "08:00"
            end: "18:00"
          - location: "Europe/Berlin"

receivers:
  - name: email-default
    email_configs:
      - to: "homelab-alerts@fogserv.cloud"

  - name: email-infra
    email_configs:
      - to: "infra@fogserv.cloud"

  - name: telegram-oncall
    telegram_configs:
      - bot_token_file: /etc/alertmanager/secrets/tg-bot-token
        chat_id: -1001234567890
        send_resolved: true

  - name: webhook-tickets
    webhook_configs:
      - url: "http://task-bot.internal:8080/alertmanager"
        send_resolved: true

Interval semantics worth internalizing (they cause most confusion):

Setting Meaning Typical value
group_wait Delay before the FIRST notification of a new group, to batch friends arriving 30s
group_interval Minimum delay between notifications about the SAME group 5m
repeat_interval How often to re-send while the group stays firing 4h (critical: 1h)

Grouping strategy

group_by controls which alerts collapse into one notification. Sensible choices:

Silences vs inhibition

They solve different problems:

inhibit_rules:
  # If a host is down, suppress all other alerts from that host
  - source_matchers: [alertname="HostDown"]
    target_matchers: [severity=~"warning|info"]
    equal: ["instance"]

  # Critical suppresses warning of the same alertname+instance
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: ["alertname", "instance"]

Rule of thumb: if you'd create the same silence every week, encode it as an inhibition rule instead.

Receivers

Email — shown above under global.smtp_*. Test with a deliberately-firing rule.

Webhook — POSTs this JSON shape (handle status: firing/resolved):

{
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "HostDiskWillFillIn14Days", "severity": "warning", "instance": "docker-01:9100" },
      "annotations": { "summary": "Disk on docker-01 fills within ~14 days" },
      "startsAt": "2026-08-26T09:30:00Z",
      "endsAt": "0001-01-01T00:00:00Z"
    }
  ],
  "groupLabels": { "alertname": "HostDiskWillFillIn14Days" }
}

Minimal handler (Node/Express):

app.post("/alertmanager", (req, res) => {
  for (const a of req.body.alerts ?? []) {
    const { alertname, instance } = a.labels;
    if (a.status === "firing") createTicket(`${alertname} on ${instance}`, a.annotations.summary);
    else closeTicket(`${alertname} on ${instance}`);
  }
  res.sendStatus(200);
});

Telegram — create a bot with @BotFather, add it to a group, get the chat id via getUpdates, then use telegram_configs as shown. send_resolved: true so recovery messages arrive too.

Escalation patterns without extra software

True paging tools (PagerDuty/Grafana OnCall) are overkill here; these patterns cover most homelab needs:

# Pattern 1: re-notify faster the longer something burns (two routes, same target,
# second gated on nothing but repeat_interval difference won't escalate by age —
# so pair Alertmanager with a webhook consumer that counts notifications):
routes:
  - matchers: [severity="critical"]
    receiver: telegram-oncall
    repeat_interval: 30m
    continue: true
  - matchers: [severity="critical"]
    receiver: webhook-escalator   # escalator re-sends to SMS after 3 unacked

# Pattern 2: business-hours vs night split
route:
  routes:
    - matchers: [severity="warning"]
      receiver: email-warnings
      active_time_intervals: [work-hours]
    - matchers: [severity="warning"]
      receiver: telegram-oncall     # same class of alert pages at night
      inactive_time_intervals: [work-hours]

Validate every change before reload — Alertmanager refuses bad configs at startup otherwise:

amtool check-config /etc/alertmanager/alertmanager.yml
curl -X POST http://alertmanager:9093/-/reload
amtool alert query alertmanager=http://alertmanager:9093   # see live state from CLI

Practical Examples

Example 1: Stop the restart-storm spam

A flapping container fires ContainerCrashLooping every few minutes. Fix at three levels:

  1. Alert side (simple-alerts.md): require persistence — for: 15m
  2. Route side: give that alertname its own route with repeat_interval: 2h
  3. Incident side: silence the instance while you actually fix it
amtool silence add instance="docker-01:9100" alertname="ContainerCrashLooping" \
  --duration 2h --comment "fixing image pull, JIRA-123" \
  --alertmanager.url http://localhost:9093

Example 2: Maintenance window without lost history

Before upgrading the Proxmox host:

amtool silence add cluster="proxmox" --duration 90m \
  --comment "kernel upgrade 2026-08-26" --author "ops@fogserv.cloud"

Alerts still fire and appear in the UI (marked suppressed); nobody gets pinged; resolved-after-maintenance states are clean.

Example 3: Verify the whole chain end-to-end

Fire a synthetic alert straight into Alertmanager, bypassing Prometheus:

curl -XPOST http://localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[
  {"labels":{"alertname":"TestCritical","severity":"critical","area":"infra","instance":"test:1"},
   "annotations":{"summary":"synthetic test alert"}}
]'

Expected: Telegram message within ~10s (group_wait), email too (because of continue: true), and both again resolved once you clear it. If any leg fails, check curl localhost:9093/api/v2/status and Alertmanager logs.

Common Pitfalls & Troubleshooting

Problem Cause Fix
Alerts fire in Prometheus UI, nobody notified No alerting section in prometheus.yml, or wrong AM address Add alerting: block; verify /api/v2/status shows the Prometheus peer
Notification storm during an incident group_by missing key labels, or group_wait too long letting groups fork Group by ["alertname","instance"]; keep group_wait modest
Same alert re-pings every few minutes repeat_interval inherited too short from a parent route Set explicit repeat per route; remember child routes don't inherit values they override
Nothing arrives during off hours unexpectedly An active_time_intervals gate left on a broad route Audit routes with time gates; amtool config show the effective tree
Silences pile up forever Forgotten ad-hoc silences Prefer --duration over open-ended; review weekly via UI filter
Critical alert also needed in the archive Default routing stops at first match continue: true on the critical route (as configured above)
Telegram messages never arrive Bot not in the target chat, or wrong chat_id sign Add bot to group, fetch real id via getUpdates (group ids are negative)

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse