fogserv.cloud fogserv.cloud

Node Exporter Setup - Host Metrics for Every Machine

Status: Active
Last Updated: 2026-08-26
Category: Observability - Metrics Collection
Prerequisites: prometheus-basics, docker-installation
Time: 1-2 hours (first host), ~15 min per additional host
Tags: node-exporter, prometheus, systemd, docker, metrics, linux

Summary

node_exporter exposes CPU, memory, disk, network, and filesystem metrics from any Linux host on port 9100. This article covers installing it as a systemd service and as a Docker container, adding scrape targets to Prometheus, the metrics that actually matter, and recording rules that pre-compute the expensive expressions your dashboards use.

What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Context / Why This Matters
  2. Implementation / Core Content
  3. Practical Examples
  4. Common Pitfalls & Troubleshooting
  5. Next Steps / Ops Actions

Context / Why This Matters

prometheus-basics installed Prometheus itself; this article gives it something useful to scrape. Every host you manage — bare metal, VM, or the Docker host — should report its own vitals. Without node_exporter, your first clue that a disk filled up is usually an application failing mysteriously. With it, you get days of trend line and can alert at 80% instead of discovering it at 100%.

This is deliberately narrow: installation and host-level metrics only. Application instrumentation and alert routing are covered by simple-alerts.md and alertmanager-config.md.

Implementation / Core Content

Method 1: systemd install (recommended for bare metal and VMs)

Download and install the binary:

# Grab the latest release tarball (check https://github.com/prometheus/node_exporter/releases)
NODE_EXPORTER_VERSION="1.8.2"
cd /tmp
wget "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
tar xvfz node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz

# Dedicated unprivileged user + binary in PATH
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter
sudo mv node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Systemd unit at /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes \
  --web.listen-address=0.0.0.0:9100

Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=-/proc

[Install]
WantedBy=multi-user.target

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://localhost:9100/metrics | head -5

Method 2: Docker container (for container-only hosts)

The exporter must see the host's namespaces, not the container's — hence the pid/net mounts and --path.rootfs:

docker run -d \
  --name node-exporter \
  --restart unless-stopped \
  --pid=host \
  --network host \
  -v /:/host:ro,rslave \
  quay.io/prometheus/node-exporter:v1.8.2 \
  --path.rootfs=/host

Or as part of a compose file:

services:
  node-exporter:
    image: quay.io/prometheus/node-exporter:v1.8.2
    container_name: node-exporter
    restart: unless-stopped
    pid: host
    network_mode: host
    volumes:
      - /:/host:ro,rslave
    command:
      - "--path.rootfs=/host"

Pick one method per host, not both — two exporters on port 9100 means one silently fails.

Scrape configuration

Append to prometheus.yml (see prometheus-basics.md for the full config layout):

scrape_configs:
  - job_name: "nodes"
    static_configs:
      - targets:
          - "192.168.1.10:9100"   # proxmox-01
          - "192.168.1.11:9100"   # docker-01
          - "192.168.1.12:9100"   # backup-nas
    # relabel so dashboards show hostname, not ip:port
    relabel_configs:
      - source_labels: [__address__]
        regex: "([^:]+):.*"
        target_label: instance

For more than a handful of hosts, switch to file-based service discovery so adding a machine doesn't touch prometheus.yml:

  - job_name: "nodes"
    file_sd_configs:
      - files: ["/etc/prometheus/targets/nodes.yml"]
        refresh_interval: 30s

/etc/prometheus/targets/nodes.yml:

- targets:
    - "192.168.1.10:9100"
    - "192.168.1.11:9100"
  labels:
    env: prod

Then reload: curl -X POST http://prometheus:9090/-/reload. Verify under Status → Targets in the web UI; each target should be UP.

Key metrics worth knowing

Metric Meaning Watch for
node_cpu_seconds_total Cumulative CPU time per core/mode Sustained rate > 0.8 cores/core
node_memory_MemAvailable_bytes Usable memory incl. reclaimable cache Below ~10% of total
node_filesystem_avail_bytes filtered to real filesystems Free disk per mount (exclude tmpfs/overlay via label filter) Below 15–20% of size
node_disk_io_time_seconds_total Fraction of time device was busy Sustained near 1.0 = saturated
node_network_receive_bytes_total RX bytes per interface Baseline drift, saturation
node_load1 / node_load5 Load averages Compare against core count
node_systemd_unit_state Systemd unit states (needs flag) state="failed" = 1

The single most-used derived expression — usable CPU % across all modes except idle:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Recording rules

Dashboards that compute rates over many hosts re-evaluate those expressions constantly. Pre-compute them once per interval with a rules file, e.g. /etc/prometheus/rules/node-rules.yml, referenced from prometheus.yml via rule_files: ["/etc/prometheus/rules/*.yml"]:

groups:
  - name: node-recording
    interval: 30s
    rules:
      - record: instance:node_cpu_utilisation:percent
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

      - record: instance:node_memory_utilisation:percent
        expr: 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      - record: instance:node_filesystem_free_percent:ratio
        expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}

      - record: instance:node_disk_io_utilisation:percent
        expr: rate(node_disk_io_time_seconds_total[5m]) * 100

      - record: instance:node_network_receive_mbps:rate5m
        expr: rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m]) * 8 / 1e6

      - record: instance:node_systemd_units_failed:count
        expr: count by (instance) (node_systemd_unit_state{state="failed"} == 1)

Recording rule naming convention <scope>:<what>:<unit> keeps autocomplete sane in Grafana. Dashboards then query instance:node_memory_utilisation:percent directly — cheaper, and consistent across every panel (grafana-dashboards.md).

Reload rules without restart: curl -X POST http://prometheus:9090/-/reload, then check Status → Rules.

Practical Examples

Example 1: Full setup on a new Debian/Ubuntu host

# 1. Install (systemd path above)
# 2. Verify locally
curl -s localhost:9100/metrics | grep node_load1
# node_load1 0.14

# 3. Open firewall only to the Prometheus host
sudo ufw allow from 192.168.1.5 to any port 9100 proto tcp

# 4. Add to /etc/prometheus/targets/nodes.yml on the Prometheus host
# 5. Reload Prometheus and check http://prometheus:9090/targets → state UP

Example 2: Disk usage dashboard query using recorded series

# All hosts' root filesystems, percent used, sorted worst-first
sort_desc(100 * (1 - instance:node_filesystem_free_percent:ratio{mountpoint="/"}))

# Predict days until full at current 7-day growth rate
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[7d], 14*24*3600) < 0

That second query is the basis of a much better alert than a static 80% threshold — it warns based on when the disk fills, not how full it is now.

Example 3: Find which process group ate the RAM

With --collector.processes enabled:

rate(node_processes_state_bytes{state="resident"}[5m])

Combined with per-container metrics (cAdvisor job from prometheus-basics.md), this narrows a memory-pressure incident to the offending workload within minutes.

Common Pitfalls & Troubleshooting

Problem Cause Fix
Target DOWN, connection refused Exporter not running or wrong port systemctl status node_exporter; ss -tlnp | grep 9100
Filesystem metrics show overlay/tmpfs noise Default collectors include container mounts Filter out tmpfs and overlay fstypes in queries, or use --collector.filesystem.mount-points-exclude
Metrics show container's CPU, not host's (Docker mode) Missing pid: host or /host mount Recreate with both flags and --path.rootfs=/host
node_systemd_unit_state absent systemd collector disabled by default Add --collector.systemd to ExecStart/command
Prometheus shows old target list after edit Static configs need reload/restart curl -X POST http://prometheus:9090/-/reload (start with --web.enable-lifecycle)
Two exporters fight over port 9100 Both systemd and docker installed Remove one: docker rm -f node-exporter or systemctl disable --now node_exporter
Load alert fires constantly on 1-core box Alert threshold written for 8-core servers Normalize: node_load5 / count without (cpu) (node_cpu_seconds_total{mode="idle"})

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse