fogserv.cloud fogserv.cloud

Why Monitor - The Importance of Observability

Status: Active
Last Updated: 2026-08-14
Category: Observability - Phase 1: Basic Monitoring
Prerequisites: None
Time: 30 minutes
Tags: monitoring, observability, mttr, mttd, incident-response, concepts, fundamentals

Summary

The conceptual foundation for the entire observability course. This article explains why monitoring is not optional infrastructure plumbing but the difference between minutes-long outages you catch instantly and days-long outages your users discover first. It introduces MTTR vs MTTD, the concept of unknown unknowns, and a practical framework for deciding when (and what) to monitor.

🎯 What You'll Learn

By the end of this article, you'll understand:

Table of Contents

  1. What Can Go Wrong
  2. MTTD vs MTTR
  3. Unknown Unknowns
  4. When to Monitor (and When Not To)
  5. The Observability Ladder
  6. Building the Monitoring Mindset
  7. Common Objections
  8. Key Takeaways
  9. Next Steps

What Can Go Wrong

Unmonitored infrastructure fails silently. Here are the classic failure modes that every operator eventually meets — usually at 2 AM.

The Slow Leak

A memory leak grows by 2% per day. Day 1: nothing. Day 20: nothing. Day 45: the server starts swapping. Day 47: the OOM killer randomly executes your database mid-write.

Day 1   RAM: 34%    ✓ healthy
Day 15  RAM: 62%    ✓ "still fine"
Day 35  RAM: 96%    ✗ nobody is looking
Day 46  RAM: swap thrashing, disk I/O saturated
Day 47  OOM killer terminates postgres during peak traffic

With monitoring, day 15 triggers an alert ("memory trending above baseline"). Without it, you discover the problem from user complaints — or from data corruption.

The Silent Death

Services don't always crash loudly. A cron job that stops running produces no error anywhere. A TLS certificate expires quietly. A backup job writes to a full disk and fails — while the application itself stays perfectly "up."

These failures are invisible to anyone who only checks "is the website responding?" They're only visible if something is watching for them specifically.

The Cascade

Small failures compound:

  1. Disk hits 95% full → logs can't rotate
  2. Log rotation fails → app can't write logs → app hangs
  3. App hangs → health checks time out → load balancer marks all backends down
  4. Total outage, root cause was one partition that filled over three weeks

Each individual step was observable. Nobody observed any of them.

The Cost Asymmetry

Approach Detection Typical cost
User reports outage Minutes to days Reputation loss, churn, SLA penalties
Weekly manual check Up to 7 days Wasted human hours, missed incidents between checks
Automated monitoring Seconds to minutes Setup effort once, small resource overhead

Monitoring is cheap relative to a single prevented incident.

MTTD vs MTTR

Two numbers define how an incident hurts:

Total outage pain = MTTD + diagnosis + repair = MTTD + MTTR

Why MTTD Is the Multiplier

MTTR is bounded by skill and tooling. MTTD is unbounded — it can literally be infinite (nobody ever notices). Every minute of undetected downtime is pure loss, and worse: users find out before you do, which converts a technical hiccup into a trust problem.

Monitoring's primary job is to drive MTTD toward zero:

No monitoring:      failure ────────(days?)────────→ first human notices
Uptime checks:      failure ──(60s)──→ alert fires
Full observability: failure ──(10s)──→ alert + dashboard + probable cause

Diagnosis Time Is Part of MTTR Too

Detection alone isn't enough. When the alert fires, you must answer: what broke, where, and since when? This is why the course doesn't stop at uptime checks:

Better observability compresses the diagnosis portion of MTTR, often more than the fix itself.

Unknown Unknowns

Donald Rumsfeld's matrix applies directly to operations:

Known Unknown
Knowns "The API gets slow under load" "Which query causes it"
Unknowns "We didn't know the disk was filling" "We didn't know that feature even existed, let alone that it crashes"

Monitoring Shrinks the Grid

You cannot enumerate everything that will break. But you can instrument enough signals — CPU, memory, disk, network, error rates, latencies — that when something breaks, the anomaly shows up whether or not you predicted it. That is the entire argument for general-purpose metrics collection over hand-written check scripts: scripts only answer questions you thought to ask; metrics answer questions you haven't asked yet.

When to Monitor (and When Not To)

Monitoring everything is its own failure mode (see Pitfall 1 in the README). Use this decision framework.

Always Monitor

Conditionally Monitor

Deliberately Don't Monitor (or Alert On)

The One-Question Test

For each candidate metric or check, ask:

"If this number goes weird tonight, will someone need to do something?"

This test prevents alert spam before it starts.

The Observability Ladder

Observability is layered. Each rung answers a different question and builds on the previous:

Rung 6: Dashboards   "What's the current state at a glance?"
Rung 5: Alerts       "Tell me when it breaks — before users do"
Rung 4: Traces       "WHY is the request slow/failing across services?"
Rung 3: Logs         "WHAT happened inside the app? Show me the events"
Rung 2: Metrics      "HOW is the system trending? CPU, RAM, latency..."
Rung 1: Uptime       "IS it alive right now?"

Climb in This Order

Trying to start with distributed tracing is like buying a telescope before owning glasses. Each layer is useful immediately and motivates the next:

  1. Uptime checks (uptime-kuma-setup) take an hour and already beat 80% of self-hosted setups.
  2. Real-time metrics (netdata-basics) give instant visibility into why something is unhealthy.
  3. Alerting (simple-alerts) closes the loop so you don't have to watch dashboards.
  4. Prometheus + Grafana (Phase 2–3) provide history, trends, and capacity planning.
  5. Centralized logging (Phase 4) provides the forensic detail metrics lack.
  6. Tracing and SLOs (Phase 5) scale the practice to complex systems and real reliability engineering.

Where This Course Goes

Phase Rungs covered Tools
1: Basic Monitoring 1–2, intro to 5 Uptime Kuma, Netdata
2: Metrics Collection 2 Prometheus, Node Exporter, exporters
3: Visualization 2, 6 Grafana, PromQL
4: Logging 3 Loki, Promtail, LogQL
5: Advanced & Production 4, 5, 6 Jaeger, Alertmanager, SLOs

Building the Monitoring Mindset

Tools change; principles don't. These five habits separate operators who sleep through the night from those who get paged.

1. Alert on Symptoms, Not Causes

CPU alerts fire constantly for harmless reasons and train you to ignore pages. Symptom alerts are rare and always meaningful.

2. Every Alert Needs a Runbook Link

An alert without instructions is trivia, not signal. Minimum viable runbook:

## Alert: HighMemoryUsage on web-prod-01
**Impact**: Risk of OOM kill on the app server.
**First steps**:
1. Check top memory consumers: `docker stats --no-stream`
2. If postgres RSS > 80%: restart with `systemctl restart postgresql` (safe, brief blip)
3. Escalate to #ops if memory returns above 85% within 1h.

3. Baselines Before Thresholds

You can't pick a threshold until you know what "normal" looks like. Run new monitors in observe-only mode for 1–2 weeks, look at the graphs, then set thresholds at clearly-above-normal levels.

4. Test Your Monitoring Like You Test Backups

An alert path that has never fired is a rumor. Deliberately break things:

# Simulate high CPU — confirm the alert fires end-to-end
stress-ng --cpu 4 --timeout 120s
# Kill a monitored service — confirm recovery alert too
docker stop nginx && docker start nginx

Schedule this monthly. Monitoring rots silently otherwise.

5. Review Alerts Monthly

For each alert that fired last month ask: did it require action? Alerts that didn't get deleted or downgraded. Alert quality decays without maintenance — this is the single highest-leverage habit in this list.

Common Objections

"My setup is tiny, nothing to monitor"

Tiny setups fail identically to big ones — disks fill, certs expire, OOM kills happen. In some ways they're riskier: no redundancy, no on-call rotation, no second pair of eyes. Uptime Kuma on the same box takes 10 minutes.

"I'll notice when something breaks"

You'll notice user-visible breaks, eventually. You won't notice the backup failing today, the certificate expiring in 12 days, or memory creeping toward OOM over six weeks. Those become tomorrow's outage.

"Monitoring uses resources"

Budget reality on a small VPS:

Component RAM Notes
Uptime Kuma ~100 MB Or host externally
Netdata ~150 MB Streaming mode, not full DB
Prometheus (10 targets) ~500 MB–1 GB Tune retention
Grafana ~250 MB

Under 2 GB total for a complete metrics stack — less than most people spend on one unused container.

"False alarms are annoying so I turned alerts off"

False alarms mean the thresholds are wrong, not that monitoring is wrong. Fix them with baselines (habit #3) and monthly reviews (habit #5). An ignored pager is worse than no pager — it burns trust in future true alarms.

Key Takeaways

  1. Undetected failure is the expensive kind. MTTD dominates outage cost; monitoring exists to crush it.
  2. Monitor what matters: user symptoms, finite resources, data integrity, money paths.
  3. Use the one-question test: would a weird value require action tonight? If not, dashboard it or drop it.
  4. Climb the ladder in order: uptime → metrics → logs → traces → alerts → dashboards.
  5. Monitoring is a practice, not a setup: baselines, tested alert paths, and monthly review keep it alive.
  6. Start today: one uptime check on your most important URL beats a perfect plan.

🔗 Related

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse