fogserv.cloud fogserv.cloud

Simple Alerts - Get Notified When Things Break

Status: Active
Last Updated: 2026-08-14
Category: Observability - Phase 1: Basic Monitoring
Prerequisites: uptime-kuma-setup, netdata-basics
Time: 2 hours
Tags: alerting, notifications, email, slack, discord, telegram, webhooks, alert-fatigue, oncall

Summary

Monitoring you don't hear about is decoration. This lesson wires real notification channels — email, Discord/Slack webhooks, and Telegram — into Uptime Kuma and Netdata, then teaches the harder skill: designing an alert strategy that stays trustworthy. You'll build severity tiers, write your first runbooks, and apply concrete anti-fatigue rules so every notification deserves attention.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Alert Design Before Alert Plumbing
  2. Email Alerts via SMTP
  3. Discord and Slack Webhooks
  4. Telegram Notifications
  5. Severity Tiers
  6. Fighting Alert Fatigue
  7. Minimum-Viable Runbooks
  8. Testing Your Alert Paths
  9. Troubleshooting
  10. Key Takeaways
  11. Next Steps

Alert Design Before Alert Plumbing

Plugging in a webhook takes five minutes. Deciding what deserves to interrupt you is the actual work — do it first.

The Three Questions Every Alert Must Answer

  1. Is something actually wrong? (not just unusual)
  2. Does someone need to act now? (vs. tomorrow's ticket)
  3. What should they do? (the runbook exists)

An alert that fails any question becomes noise, and noise has a compounding cost: every irrelevant ping lowers your response speed to the next real one.

Symptom-First Checklist

Revisit the principle from why-monitor with concrete examples:

❌ Cause alert (noisy) ✅ Symptom alert (actionable)
CPU > 90% Checkout endpoint error rate > 5% for 5 min
Memory > 80% Service restarted 3× in 1 h (OOM loop)
Disk I/O high Backup push monitor missed
Nginx worker count high Homepage returns non-200 for 3 checks

Keep cause metrics on dashboards; page humans only for symptoms.

Email Alerts via SMTP

The universal channel — works everywhere, survives phone loss, searchable. Start here even if you add chat channels later.

Pick an SMTP Source

Option Notes
Your existing mailbox + app password Fastest; Gmail/Fastmail need app-specific passwords
Transactional service (Mailgun, SES free tier) Best deliverability; matches fogserv.cloud's emailService pattern
Self-hosted relay Full control, deliverability is your problem

Configure in Uptime Kuma

Settings → Notifications → Setup Notification:

Field Example
Notification type SMTP
Host smtp.fastmail.com
Port 465 (implicit TLS) or 587 (STARTTLS)
Secure ✅ match port choice
Username / Password account + app password
From monitor@yourdomain.com
To ops@yourdomain.com

Click Test, confirm arrival, then save with "Apply on all existing monitors" checked.

Configure in Netdata

Edit /opt/netdata/etc/netdata/health_alarm_notify.conf (path per netdata-basics):

SEND_EMAIL="YES"
EMAIL_SENDER="monitor@yourdomain.com"
EMAIL_SMTP_SERVER="smtp.fastmail.com"
EMAIL_SMTP_PORT="465"
EMAIL_SMTP_USER="monitor@yourdomain.com"
EMAIL_SMTP_PASSWD="<app-password>"
DEFAULT_RECIPIENT_EMAIL="ops@yourdomain.com"

Then restart and test:

sudo systemctl restart netdata
sudo su -s /bin/bash netdata -c \
  '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh test'

Expect three messages (warning/critical/clear). No mail? See Troubleshooting.

🔒 Store SMTP credentials with dotenvx or your secrets manager (kb/sysadmin/secrets) — never commit them alongside compose files.

Discord and Slack Webhooks

Chat channels shine for team visibility: history is shared, reactions acknowledge ownership, and recovery messages close the loop publicly.

Discord

  1. Server Settings → Integrations → Webhooks → New Webhook
  2. Choose channel (#infra-alerts), copy URL
  3. In Kuma/Netdata paste the URL as the Discord notification target

Recommended extras:

Slack

Slack uses Incoming Webhooks apps instead:

  1. api.slack.com/apps → Create New App → Incoming Webhooks → Activate
  2. Add Webhook to Workspace → pick #infra-alerts
  3. Copy the https://hooks.slack.com/services/... URL

Both Kuma and Netdata treat Slack and Discord identically once the URL is pasted — the payload format differs but the tools handle it.

Channel Hygiene Rules

Telegram Notifications

For true pocket-pages without paying PagerDuty, Telegram bots are the self-hoster standard — instant delivery, read receipts, quiet-hours control per device.

Create the Bot (one time)

1. Chat with @BotFather → /newbot
2. Name it "fogops-bot", get the bot TOKEN (keep secret!)
3. Add the bot to your group (or DM it directly)
4. Visit https://api.telegram.org/bot<TOKEN>/getUpdates
   → find "chat":{"id": -1001234567890 }   ← the CHAT ID

Wire It Up

Uptime Kuma: Notification type Telegram → paste token + chat ID → Test.

Netdata health_alarm_notify.conf:

SEND_TELEGRAM="YES"
TELEGRAM_BOT_TOKEN="<TOKEN>"
DEFAULT_RECIPIENT_TELEGRAM="-1001234567890"

Why Keep Telegram Alongside Chat

Severity Tiers

One channel at one volume trains bad habits. Split everything into three tiers:

Tier Meaning Channel Latency expectation
🔴 Page Users impacted or about to be Telegram (+ call/SMS escalation) Respond ≤15 min
🟡 Notify Wrong but not user-facing yet; disk at 80% Discord/email Same-day look
⚪ Log Recorded for trends, no human action Dashboard only Never interrupts

Applying Tiers in Practice

In Uptime Kuma, create separate notifications per tier ("Telegram-Pager", "Discord-Notify") and attach selectively per monitor:

Homepage HTTP check        → Telegram-Pager + Discord-Notify
Backup push monitor        → Telegram-Pager (backups are silent death)
Staging site check         → Discord-Notify only
Certificate expiry         → Notify at 30 days, escalate to Page at 7
Internal NAS ping          → Log/Notify (redundant paths exist)

In Netdata, tier by alarm severity using role-based recipients:

role_recipients_email[sysadmin]="ops@yourdomain.com"          # warnings
role_recipients_telegram[sysadmin]="-1001234567890"           # criticals page
role_recipients_discord[weblog]="https://hooks.slack.com/..." # info to chat

Escalation Without Paid Tooling

True auto-escalation arrives with Alertmanager in Phase 5 (alertmanager-setup). Until then, fake it acceptably:

Fighting Alert Fatigue

Fatigue is the terminal disease of monitoring systems: ignored pagers have negative value because they burn trust needed for future true alarms. Prevent it structurally.

Rule 1: Every Alert Has an Owner and a Runbook

No runbook → downgrade to Notify until written. This single rule kills 80% of noise because writing "what would I even do?" exposes half your alerts as unactionable.

Rule 2: Aggregate Flappy Things

A service restarting every few minutes shouldn't produce 40 messages. Alert on the pattern: "restart count > 3 in 30 min." In Kuma, raise retries; in Prometheus (Phase 2+) use increase() over windows rather than instantaneous triggers.

Rule 3: Deduplicate Related Failures

When the database dies, every dependent service screams simultaneously. Prefer alerting on the root dependency (DB health) and letting dependents show degraded-on-dashboards. Alertmanager's inhibition handles this properly later (alertmanager-setup); manually, order monitors so the DB check is the loudest.

Rule 4: Maintenance Windows

Before planned work, silence notifications for affected monitors (Kuma supports per-monitor disable + scheduled maintenance; Netdata supports silencing alarm types). Post-maintenance, re-enable and verify green. Unsilenced planned work manufactures false pages.

Rule 5: Monthly Alert Review

Calendar reminder: for each alert that fired last month ask —

fired N times → required action N times?
  yes, all      → keep
  sometimes     → tighten threshold / add hysteresis
  never         → demote tier or delete
never fired    → still relevant? test it or retire it

Thirty minutes monthly preserves years of pager trust.

Minimum-Viable Runbooks

A runbook is not documentation theater — it's the difference between a 5-minute fix and a 45-minute panicked archaeology session at 3 AM.

Template (steal this)

## [ALERT-NAME] <one-line what broke>

**Fires when**: <exact condition, e.g. "homepage non-200 ×3 @60s">
**User impact**: <what users see>
**Severity**: Page / Notify

### Diagnose (fastest first)
1. `curl -I https://site` from outside — confirm scope (all vs one node)
2. `docker ps` — anything restarting/exited?
3. `df -h && free -m` — resource exhaustion?
4. Netdata dashboard → what changed in the last hour?

### Fix
- App down: `docker compose up -d` in ~/apps/<app>, watch logs 2 min
- Disk full: clear docker build cache `docker system prune -f`, then investigate growth
- OOM: identify consumer via Netdata containers view, restart it, open issue

### If that fails
- Reboot the host (last resort, ~2 min downtime): document why afterwards
- Escalate: post in #infra-alerts with steps tried

### After resolution
- Verify alert cleared automatically
- Append 3 lines to the incident log: cause / fix / prevention idea

Linking Runbooks to Alerts

Testing Your Alert Paths

Untested alerting fails exactly when it matters. Build a quarterly (monthly is better) drill:

#!/bin/bash
# alert-drill.sh — deliberately trigger each tier, confirm delivery
set -x
# 1. Page tier: stop a monitored container
docker stop nginx-proxy && sleep 240 && docker start nginx-proxy

# 2. Notify tier: fill a scratch disk past warning threshold
dd if=/dev/zero of=/mnt/scratch/filler bs=1M count=9000
rm /mnt/scratch/filler

# 3. Recovery: confirm CLEAR/recovery messages arrived too

Checklist after each drill:

Log drills in the KB like any other operational change — GitOps is Law (kb/workflows).

Troubleshooting

Test notification never arrives (email)

Discord/Slack webhook posts nothing

Telegram bot silent

Alerts arrive but hours late

Everything alerts at once during a network blip

That's deduplication failure (Rule 3). Short term: raise retries. Long term: this is the strongest argument for graduating to Prometheus + Alertmanager inhibition in Phase 2/5.

Key Takeaways

  1. Design before plumbing: symptom alerts only, three questions per alert, owner + runbook mandatory.
  2. Email is the backbone, Discord/Slack give shared visibility, Telegram gives real paging with DND-bypass.
  3. Three tiers — Page / Notify / Log — with different channels and response expectations each.
  4. Fatigue is structural: aggregate flaps, dedupe cascades, use maintenance windows, review monthly.
  5. Every page links to a runbook; unrunbooked alerts get demoted.
  6. Drill the paths quarterly — an unfired alert path is a rumor, not a capability.

🔗 Related

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse