fogserv.cloud fogserv.cloud

Prometheus Basics - Pull-Based Metrics Collection

Status: Active
Last Updated: 2026-08-26
Category: Observability - Phase 2: Metrics Collection
Prerequisites: netdata-basics, uptime-kuma-setup
Time: 2-3 hours
Tags: prometheus, metrics, promql, node-exporter, cadvisor, docker-compose, time-series, monitoring

Summary

Prometheus is the de facto standard for time-series metrics collection: a single Go binary that pulls (scrapes) metrics from HTTP endpoints every 15-60 seconds, stores them in a local time-series database, and answers queries with PromQL. This article installs Prometheus in Docker Compose alongside Node Exporter (host metrics) and cAdvisor (container metrics), walks through prometheus.yml scrape configuration line by line, and teaches the PromQL you'll actually use daily — rate(), up, memory queries — plus retention tuning.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. The Pull Model
  2. Architecture Overview
  3. Docker Compose Installation
  4. Understanding prometheus.yml
  5. Node Exporter - Host Metrics
  6. cAdvisor - Container Metrics
  7. PromQL Basics
  8. Retention Configuration
  9. Troubleshooting & Common Pitfalls
  10. Sources & Related

The Pull Model

Every monitoring system must answer: who initiates — the server collecting, or the thing being measured?

Model How it works Consequence
Push Agents send metrics to the server Server can't tell "down" from "quiet"; agents need config/credentials
Pull (Prometheus) Server fetches /metrics HTTP endpoints Dead-simple health check: no response = target down; targets expose plain HTTP

In Prometheus's pull model, anything that serves a plain-text metrics page at /metrics is a valid target:

$ curl http://localhost:9100/metrics | head -4
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 123456.78
node_cpu_seconds_total{cpu="0",mode="user"} 9876.54

No SDKs, no daemons required — any app that can print text on an HTTP port is monitorable. This is why thousands of exporters exist.

📌 When pushes make sense: short-lived batch jobs that die before a scrape can happen use the Pushgateway; event-driven systems use remote write. Those are exceptions — 95% of setups are pure pull.

Architecture Overview

┌────────────────────────────── Docker host ──────────────────────────────┐
│                                                                          │
│  ┌────────────┐  scrapes /metrics   ┌──────────────────────────┐         │
│  │ Prometheus │ ───────────────────►│ Node Exporter  :9100     │ host    │
│  │    :9090   │ ───────────────────►│ cAdvisor       :8080     │ contnr  │
│  │            │ ───────────────────►│ Netdata        :19999    │ (opt.)  │
│  └─────┬──────┘                     └──────────────────────────┘         │
│        │ PromQL                                                          │
│  ┌─────▼──────┐                                                          │
│  │ Grafana    │  ← Phase 3 ([grafana-dashboards](grafana-dashboards))    │
│  └────────────┘                                                          │
└──────────────────────────────────────────────────────────────────────────┘

Each component has one job: exporters expose, Prometheus collect and store (and query), Grafana visualize. Alerting comes later via Alertmanager or Grafana alerting — see simple-alerts.

Docker Compose Installation

Create a project directory with three files:

mkdir -p ~/apps/prometheus && cd ~/apps/prometheus

docker-compose.yml

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "127.0.0.1:9090:9090"          # localhost only — proxy/tunnel for UI
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=90d
      - --web.enable-lifecycle          # allows POST /-/reload without restart
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./data:/prometheus              # TSDB survives rebuilds
    depends_on:
      - node-exporter
      - cadvisor

  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    pid: host                           # see all host processes
    ports:
      - "127.0.0.1:9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro                    # filesystem stats need root mountpoint
    command:
      - --path.procfs=/host/proc
      - --path.sysfs=/host/sys
      - --path.rootfs=/rootfs

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    devices:
      - /dev/kmsg                       # kernel log access for OOM events
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "127.0.0.1:8080:8080"

Notes:

prometheus.yml (minimal starter)

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

Targets use service names, not localhost — containers resolve each other through the Compose network (kb/containers/docker-networking).

Start and verify

mkdir -p data && sudo chown 65534:65534 data   # prometheus runs as nobody
docker compose up -d

# Validate config inside the container (catches YAML errors early):
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

# Confirm targets are UP:
curl -s http://127.0.0.1:9090/api/v1/targets | grep -o '"health":"[a-z]*"'
# expect: {"health":"up"} x3

Open http://<server>:9090 → Status ▸ Targets: all three should show green UP. The built-in graph page (Graph tab) lets you test queries before Grafana ever enters the picture.

Understanding prometheus.yml

Every scrape job follows this shape:

scrape_configs:
  - job_name: "my-app"                # becomes the `job` label on every sample
    scrape_interval: 30s              # optional override of global 15s
    metrics_path: /metrics            # default; change for nonstandard apps
    scheme: http                      # https if the exporter has TLS
    static_configs:
      - targets:
          - "app1.example.com:9100"
          - "app2.example.com:9100"
        labels:                       # extra labels attach to all samples here
          env: prod
          region: eu-west

Key rules learned the hard way:

  1. YAML is indentation-sensitive — run promtool check config after every edit.
  2. Reload without downtime: curl -X POST http://localhost:9090/-/reload (needs --web.enable-lifecycle). Config errors leave the old config running.
  3. Labels are your query dimensions. Everything a label distinguishes is stored as a separate time series — don't put unbounded values (user IDs, request paths) into labels, or cardinality explodes your disk.
  4. job + instance are automatic labels: instance defaults to host:port of the target.

Node Exporter - Host Metrics

Node Exporter exposes ~2,000 host-level metrics: CPU time per core/mode, memory breakdown, disk I/O, filesystem usage, network interfaces, load average, systemd units. It's the same data Netdata charts in real time — but now it's queryable history.

Useful sanity checks once scraped:

# CPU utilization (all cores averaged) — see PromQL section for derivation
100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])))

# Root filesystem % used
100 * (1 - node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} 
        / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay"})

# Available memory in GB
node_memory_MemAvailable_bytes / 1024^3

⚠️ Run one Node Exporter per host (bare metal install or container with the mounts shown above). A single instance cannot see other machines' /proc.

For bare-metal installs, systemd handles it:

# quick manual run for testing
./node_exporter --web.listen-address=127.0.0.1:9100
# then add the host to prometheus.yml targets: ["host1.example.com:9100"]

cAdvisor - Container Metrics

cAdvisor (Google's Container Advisor) auto-discovers every Docker/containerd container and exports per-container CPU, memory, network, and filesystem stats under the container_* metric family. No per-container registration needed — new containers appear as targets' series automatically within one scrape cycle.

The most useful queries:

# Memory used per container, sorted
topk(10, container_memory_usage_bytes{name!=""} )

# Container memory vs its limit (% of cgroup limit)
100 * container_memory_usage_bytes{name!=""}
   / on(container) container_spec_memory_limit_bytes

# CPU usage normalized to one core (cgroup accounting is cumulative-nanoseconds)
rate(container_cpu_usage_seconds_total{name!=""}[5m])

# Network receive rate per container
rate(container_network_receive_bytes_total{name!=""}[5m])

Filtering tricks worth memorizing:

PromQL Basics

PromQL reads like SQL filtered through vector algebra. Four concepts cover 90% of daily use:

1. Instant vectors — "what is X right now"

up                                        # 1 = scraping OK, 0 = target down
node_load1                                # 1-minute load average
node_memory_MemAvailable_bytes            # bytes available now

Every result carries its labels: up{job="node", instance="node-exporter:9100"} = 1.

2. Range selectors + rate() — "how fast is X changing"

Counters only go up (bytes sent since boot), so raw values are useless — take the per-second derivative over a window:

rate(node_network_receive_bytes_total[5m])        # bytes/sec inbound
rate(http_requests_total[5m])                     # requests/sec
irate(...[5m])                                    # last-two-points slope, spikier
increase(node_cpu_seconds_total{mode="user"}[1h]) # total increase over window

Rule of thumb: window ≥ 4× scrape interval (5m for 15 s scrapes) so rate() can absorb missed scrapes gracefully.

3. Aggregation — reshape with sum/avg/by

sum(rate(container_cpu_usage_seconds_total{name!=""}[5m])) by (name)

avg by (mode) (
  rate(node_cpu_seconds_total[5m])
) * 100

4. Arithmetic between series — ratios and percentages

# Disk usage % — match series on device/fstree labels
100 * (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes)

Copy-paste starter kit

# Is anything down?
up == 0

# CPU % busy (host-wide)
100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])))

# RAM used %
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Disk write throughput per device
rate(node_disk_written_bytes_total[5m])

# Top 5 memory-hungry containers
topk(5, container_memory_usage_bytes{name!=""})

# Network throughput (bits/sec) summed across interfaces
sum(rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m])) * 8

# Predict root-fs full date (days until disk fills at current trend,
# minimum 1h of history needed)
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 7*24*3600) < 0

Test everything on the Prometheus Graph tab first; panels fail loudly there but quietly in dashboards.

Retention Configuration

Default retention is 15 days — fine for demos, wasteful for capacity planning. Set it explicitly via flags (already in the Compose file above):

command:
  - --storage.tsdb.retention.time=90d    # age-based (preferred)
  # --storage.tsdb.retention.size=50GB   # size-based cap (alternative/complement)

Sizing guidance:

Resolution Approximate cost
~2,000 active series @ 15 s ~2–5 GB per month

Rough math: count your series (curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries'), assume a few KB per series-day, and pick a number your disk tolerates. For years-long history, add Thanos/Mimir/VictoriaMetrics remote storage — beyond Phase 2 scope.

Changing retention requires a container recreate (docker compose up -d after editing), not just a reload — storage flags are startup-only.

Troubleshooting & Common Pitfalls

Target shows DOWN

Charts flatline or show huge spikes after gaps

Empty cAdvisor results

Disk filling faster than expected

Config edits do nothing

You restarted the browser, not Prometheus. Either docker compose restart prometheus or better: curl -X POST http://localhost:9090/-/reload after validating with promtool.

Permission denied on ./data

The container writes as uid 65534 (nobody): sudo chown -R 65534:65534 ./data. Symptom: crash loop with "open /prometheus/lock: permission denied".

Sources & Related

Web sources consulted during research (2026-08-26):

Related KB lessons:

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse