Observability - Monitoring, Metrics & Logs
Status: Active
Last Updated: 2026-08-26
Category: Monitoring & Observability
Prerequisites: kb/basics/, kb/containers/docker-basics
Tags: monitoring, metrics, logs, observability, prometheus, grafana, loki, netdata, self-hosted
Summary
Complete observability stack learning path using self-hosted open-source tools. From simple uptime monitoring through complete Prometheus + Grafana + Loki stacks for production infrastructure. Updated for Prometheus 2.53+, Grafana 11+, Loki 3.0+ with current best practices.
🎯 Learning Philosophy
See Everything, Understand Everything:
Uptime → Metrics → Logs → Traces → Alerts → Dashboards
(Alive?) (Health) (What) (Why) (Know) (Visualize)
This directory teaches observability assuming running services but no prior monitoring experience. Progressive implementation from basic checks through comprehensive production observability.
📚 Learning Path
Prerequisites: Running services to monitor
↓
┌────────────────────────────────────────┐
│ PHASE 1: Basic Monitoring │
│ ├─ Why monitor? │
│ ├─ Uptime checks (Uptime Kuma) │
│ ├─ Simple metrics (Netdata) │
│ ├─ Alert basics │
│ └─ Monitoring mindset │
└────────────────────────────────────────┘
↓
┌────────────────────────────────────────┐
│ PHASE 2: Metrics Collection │
│ ├─ Prometheus fundamentals │
│ ├─ Node Exporter (system metrics) │
│ ├─ Container metrics (cAdvisor) │
│ ├─ Application metrics │
│ └─ Service discovery │
└────────────────────────────────────────┘
↓
┌────────────────────────────────────────┐
│ PHASE 3: Visualization │
│ ├─ Grafana setup │
│ ├─ Dashboard creation │
│ ├─ Pre-built dashboards │
│ ├─ Custom queries (PromQL) │
│ └─ Variables and templating │
└────────────────────────────────────────┘
↓
┌────────────────────────────────────────┐
│ PHASE 4: Logging │
│ ├─ Centralized logging why │
│ ├─ Loki setup │
│ ├─ Promtail for log shipping │
│ ├─ LogQL queries │
│ └─ Log aggregation patterns │
└────────────────────────────────────────┘
↓
┌────────────────────────────────────────┐
│ PHASE 5: Advanced & Production │
│ ├─ Distributed tracing (Jaeger) │
│ ├─ Alertmanager configuration │
│ ├─ On-call workflows │
│ ├─ SLOs and SLIs │
│ └─ Observability-driven development │
└────────────────────────────────────────┘
📖 Articles in This Directory
✅ Recently Created (2026-08-26 headless-browser research session)
- prometheus-basics — Pull-model metrics: Prometheus + Node Exporter + cAdvisor Compose stack, PromQL starter kit. ✅ CREATED 2026-08-26
- grafana-dashboards — Grafana install, datasources, pre-built dashboards, custom panels/variables, alerting, provisioning-as-code. ✅ CREATED 2026-08-26
🟢 Phase 1: Basic Monitoring (Start Here)
Simple is Beautiful:
why-monitor - The importance of observability
- What can go wrong
- MTTR vs MTTD
- Unknown unknowns
- When to monitor
- Prerequisites: None
- Time: 30 minutes
- Resources:
[█░░░░░░░░░]10% - Conceptual
uptime-kuma-setup - Beautiful uptime monitoring
- Uptime Kuma installation
- HTTP checks
- TCP checks
- Docker health monitoring
- Status pages
- Prerequisites: Docker basics
- Time: 1-2 hours
- Resources:
[███░░░░░░░]30%
netdata-basics - Real-time system monitoring
- Netdata installation
- System metrics dashboard
- Container monitoring
- Zero-config setup
- Alert configuration
- Prerequisites: Linux basics
- Time: 1-2 hours
- Resources:
[███░░░░░░░]30%
simple-alerts - Get notified when things break
- Email alerts
- Slack/Discord webhooks
- Telegram notifications
- Alert fatigue prevention
- Prerequisites: Monitoring setup
- Time: 2 hours
- Resources:
[██░░░░░░░░]20%
🟡 Phase 2: Metrics Collection (The Foundation)
prometheus-introduction - Metrics database
- What is Prometheus
- Pull vs Push metrics
- Time series data
- PromQL basics
- Architecture overview
- Prerequisites: Monitoring concepts
- Time: 2 hours
- Resources:
[███░░░░░░░]30%
prometheus-installation - Setting up Prometheus
- Docker compose deployment
- Configuration file
- Scrape configs
- Storage retention
- Data persistence
- Prerequisites: Docker basics
- Time: 2-3 hours
- Resources:
[█████░░░░░]50% - 2GB RAM
node-exporter - System metrics collection
- Node Exporter setup
- CPU, memory, disk metrics
- Network statistics
- Custom text file metrics
- Prerequisites: Prometheus running
- Time: 1-2 hours
- Resources:
[██░░░░░░░░]20%
exporters-catalog - Exporter ecosystem
- PostgreSQL exporter
- Redis exporter
- Nginx exporter
- Blackbox exporter
- Custom exporters
- Prerequisites: Prometheus basics
- Time: 3-4 hours
- Resources:
[████░░░░░░]40%
service-discovery - Auto-discover targets
- File-based discovery
- DNS service discovery
- Docker service discovery
- Kubernetes service discovery
- Prerequisites: Prometheus proficiency
- Time: 2-3 hours
- Resources:
[█████░░░░░]50%
🟠 Phase 3: Visualization (Making Sense of Data)
grafana-setup - Beautiful dashboards
- Grafana installation
- First login and config
- Adding Prometheus datasource
- User management
- Prerequisites: Prometheus running
- Time: 1-2 hours
- Resources:
[████░░░░░░]40% - 1GB RAM
first-dashboard - Creating dashboards
- Dashboard basics
- Panel types
- Query builder
- Visualization options
- Dashboard variables
- Prerequisites: Grafana installed
- Time: 2-3 hours
- Resources:
[████░░░░░░]40%
promql-queries - Prometheus query language
- PromQL syntax
- Selectors and matchers
- Functions and operators
- Rate calculations
- Aggregations
- Prerequisites: Prometheus basics
- Time: 4-5 hours
- Resources:
[██████░░░░]60%
dashboard-library - Pre-built dashboards
- Grafana dashboard marketplace
- Node Exporter dashboard
- Docker monitoring
- k8s monitoring
- Customizing imports
- Prerequisites: Grafana basics
- Time: 2 hours
- Resources:
[███░░░░░░░]30%
dashboard-best-practices - Design principles
- Dashboard hierarchy
- Signal vs noise
- Color conventions
- Annotations
- Sharing dashboards
- Prerequisites: Dashboard experience
- Time: 2 hours
- Resources:
[████░░░░░░]40%
🔴 Phase 4: Logging (Centralized Log Management)
loki-introduction - Like Prometheus, for logs
- What is Loki
- Log aggregation architecture
- Labels vs full-text search
- Cost-effective logging
- Prerequisites: Prometheus understanding
- Time: 1-2 hours
- Resources:
[███░░░░░░░]30%
loki-setup - Installing Loki stack
- Loki installation
- Storage configuration
- Retention policies
- High availability
- Prerequisites: Docker basics
- Time: 2-3 hours
- Resources:
[██████░░░░]60% - 2-4GB RAM
promtail-setup - Shipping logs to Loki
- Promtail agent installation
- Log file discovery
- Label extraction
- Pipeline stages
- Docker/k8s integration
- Prerequisites: Loki running
- Time: 2-3 hours
- Resources:
[███░░░░░░░]30%
logql-queries - Querying logs
- LogQL syntax
- Stream selectors
- Log pipeline
- Metric queries from logs
- Real-world examples
- Prerequisites: Loki + Promtail
- Time: 3-4 hours
- Resources:
[█████░░░░░]50%
log-aggregation-patterns - Production logging
- Structured logging
- JSON logs
- Correlation IDs
- Log levels
- Performance considerations
- Prerequisites: Logging experience
- Time: 3 hours
- Resources:
[██████░░░░]60%
⚫ Phase 5: Advanced & Production (Complete Observability)
alertmanager-setup - Alert routing
- Alertmanager installation
- Alert rules in Prometheus
- Routing trees
- Silences and inhibitions
- Integration with ticketing
- Prerequisites: Prometheus proficiency
- Time: 3-4 hours
- Resources:
[█████░░░░░]50%
on-call-workflows - Incident response
- On-call best practices
- Alert escalation
- PagerDuty alternatives (open-source)
- Runbooks and playbooks
- Post-mortems
- Prerequisites: Production experience
- Time: 2-3 hours
- Resources:
[████░░░░░░]40% - Conceptual
distributed-tracing - Jaeger setup
- What is distributed tracing
- Jaeger installation
- Instrumenting applications
- Trace analysis
- Prerequisites: Microservices
- Time: 4-5 hours
- Resources:
[███████░░░]70%
slos-and-slis - Service level objectives
- SLIs, SLOs, SLAs explained
- Defining objectives
- Error budgets
- Measuring reliability
- Prerequisites: Production monitoring
- Time: 2-3 hours
- Resources:
[█████░░░░░]50% - Conceptual
observability-driven-development - Build observability in
- Instrumentation from day 1
- Metrics in code
- Structured logging
- Tracing spans
- Testing observability
- Prerequisites: Development experience
- Time: 3-4 hours
- Resources:
[██████░░░░]60%
🔗 What Comes Next?
After mastering observability:
For Infrastructure:
- kb/infrastructure/monitoring-automation - IaC for monitoring
For Containers:
- kb/containers/k0s-monitoring - k8s observability
For CI/CD:
- kb/cicd/monitoring-pipelines - Pipeline metrics
For Security:
- kb/security/security-monitoring - Security metrics
📊 Resource Requirements
Uptime Kuma:
- Minimal: 256MB RAM, 1 CPU
[██░░░░░░░░]20%
Netdata:
- Per Host: 100-200MB RAM
[██░░░░░░░░]20%
Prometheus (v2.53+ current stable as of 2025):
- Small (10 targets): 1GB RAM, 1 CPU
[███░░░░░░░]30% - Medium (50 targets): 2GB RAM, 2 CPU
[█████░░░░░]50% - Large (200+ targets): 4GB+ RAM, 4 CPU
[████████░░]80% - Note: Prometheus 2.45 reached EOL July 2024. Use 2.53+ for active support.
Grafana (v11+ current stable as of 2025):
- Minimal: 512MB RAM, 1 CPU
[███░░░░░░░]30% - Comfortable: 1GB RAM, 2 CPU
[████░░░░░░]40% - New in v11: Data exploration without SQL, improved dashboard templating, better alerting integration.
Loki Stack (v3.0+ current stable as of 2025):
- Small: 2GB RAM, 2 CPU
[█████░░░░░]50% - Medium: 4GB RAM, 4 CPU
[███████░░░]70% - Large: 8GB+ RAM, 6+ CPU
[█████████░]90% - Breaking in v3.0: Removed experimental LogQL variants() queries. Multi-variant queries now supported via operator.
Complete Stack (Prom + Grafana + Loki):
- Homelab: 4-6GB RAM
[██████░░░░]60% - Small Production: 8-12GB RAM
[████████░░]80% - Medium Production: 16-32GB RAM
[█████████░]90%
Learning Time Investment:
- Basic Monitoring: 1 week
[███░░░░░░░]30% - Prometheus & Grafana: 2-3 weeks
[█████░░░░░]50% - Logging: 2 weeks
[████░░░░░░]40% - Advanced: 1-2 months
[███████░░░]70% - Production Mastery: 6-12 months
[██████████]100%
🛠️ Recommended Tool Stack
Monitoring Stack (The Big 3):
- Prometheus
[██████████]Required - Metrics (v2.53+) - Grafana
[██████████]Required - Dashboards (v11+) - Loki
[█████████░]90% - Logs (v3.0+)
Supporting Tools:
- Uptime Kuma
[████████░░]80% - Uptime checks - Netdata
[████████░░]80% - Real-time metrics - Alertmanager
[████████░░]80% - Alert routing - Jaeger
[██████░░░░]60% - Distributed tracing - Tempo
[██████░░░░]60% - Grafana-native tracing alternative
Exporters (Essential):
- Node Exporter
[██████████]Required - System metrics - cAdvisor
[████████░░]80% - Container metrics - Blackbox Exporter
[███████░░░]70% - Endpoint checks
Alternatives:
- Victoria Metrics
[███████░░░]70% - Prometheus alternative (single-node, better compression) - Mimir
[██████░░░░]60% - Scalable Prometheus (Grafana Labs) - Thanos
[██████░░░░]60% - Prometheus HA/long-term storage - Elasticsearch
[█████░░░░░]50% - vs Loki (heavier)
💡 Pro Tips for Observability
- Start Simple: Uptime first, metrics second, logs third
- Monitor What Matters: Not everything needs monitoring
- Alert on Symptoms: Not causes - user impact first
- Reduce Alert Fatigue: If it doesn't need action, don't alert
- Use Labels Wisely: Cardinality explosion kills Prometheus
- Dashboard Hierarchy: Overview → Service → Details
- Document Dashboards: What each panel means
- Test Alerts: Trigger them intentionally
- Practice Runbooks: Step-by-step response procedures
- Continuous Improvement: Review alerts monthly
🔄 Common Observability Pitfalls
Pitfall 1: Monitoring Everything
- High cardinality metrics, storage explosion
- Fix: Monitor what matters, sample appropriately
Pitfall 2: Alert Spam
- Too many alerts, ignoring all
- Fix: Alert on user impact, aggregate noisy alerts
Pitfall 3: No Baselines
- Don't know what "normal" looks like
- Fix: Establish baselines, understand patterns
Pitfall 4: Dashboards Without Context
- Graphs without explanation
- Fix: Add descriptions, thresholds, runbook links
Pitfall 5: Ignoring Logs
- Only metrics, no context when debugging
- Fix: Centralized logging, correlate with metrics
Pitfall 6: No Retention Policy
- Storing everything forever
- Fix: Define retention, aggregate old data
Pitfall 7: Single Point of Failure
- Monitoring goes down with the system
- Fix: External monitoring, HA setup
Pitfall 8: Not Testing Monitoring
- Broken alerts discovered during incident
- Fix: Regular testing, chaos engineering
🔗 Related KB Sections
- kb/containers/ - Container monitoring
- kb/infrastructure/ - IaC for monitoring
- kb/cicd/ - Pipeline monitoring
- kb/security/ - Security monitoring
- kb/sysadmin/ - System performance
📝 Change Log
2026-01-30
- Created observability directory structure
- Defined complete learning path from uptime to distributed tracing
- Established Prometheus + Grafana + Loki as core stack
- Listed all planned articles with time estimates
- Added resource requirements for monitoring tools
- Emphasized self-hosted, cost-effective approach
- Organized by learning phases (1-5)
- Added tool recommendations and alternatives
- Cross-referenced related KB sections
- Included production observability patterns
2026-08-15
- Updated for current versions: Prometheus 2.53+, Grafana 11+, Loki 3.0+
- Added version-specific notes and breaking changes
- Added Tempo as Grafana-native tracing alternative
- Added Victoria Metrics, Mimir, Thanos as Prometheus alternatives
- Updated resource requirements with current version notes
Change Log
- 2026-08-26: Added 2 new articles created via headless-browser web research session; marked planned items as created.