fogserv.cloud fogserv.cloud

Grafana Dashboards - Visualizing Your Metrics

Status: Active
Last Updated: 2026-08-26
Category: Observability - Phase 3: Visualization
Prerequisites: prometheus-basics, netdata-basics
Time: 2-3 hours
Tags: grafana, dashboards, prometheus, provisioning, panels, templating, alerting, docker

Summary

Grafana is the visualization layer of a metrics stack: point it at your Prometheus from prometheus-basics, and every query becomes a panel, every panel a dashboard. This guide covers installing Grafana in Docker, adding the Prometheus datasource, importing battle-tested community dashboards (Node Exporter Full & friends), building custom panels with PromQL, using template variables to make one dashboard serve many hosts, configuring Grafana-managed alert rules with notification contact points, and — most importantly — provisioning all of it as code so a rebuilt server reassembles itself from files in Git.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Why Grafana
  2. Docker Installation
  3. Adding the Prometheus Datasource
  4. Pre-Built Dashboards
  5. Building Custom Panels
  6. Variables & Templating
  7. Alerting in Grafana
  8. Provisioning as Code
  9. Troubleshooting & Common Pitfalls
  10. Sources & Related

Why Grafana

Prometheus ships a functional graph page, but it's a lab bench: no persistent dashboards, no variables, no sharing, no alert UI. Grafana adds the presentation layer:

Capability Prometheus built-in Grafana
Ad-hoc PromQL graphs ✅ ✅
Saved, organized dashboards ❌ ✅
Template variables (host dropdowns) ❌ ✅
Mixed datasources on one board ❌ ✅
Alert rules + routing + silences via Alertmanager ✅ unified
Public links / snapshots / RBAC ❌ ✅

Grafana doesn't store metrics — it queries Prometheus on every page load. That means deleting a Grafana dashboard loses only the dashboard, never data.

Docker Installation

services:
  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}   # from .env
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.example.com
    volumes:
      - ./data:/var/lib/grafana          # sqlite DB, plugins, dashboards state
      - ./provisioning:/etc/grafana/provisioning   # as-code configs (below)

echo "GRAFANA_ADMIN_PASSWORD=change-me-long-random" >> .env
mkdir -p data provisioning/datasources provisioning/dashboards
sudo chown -R 472:472 data        # grafana runs as uid 472
docker compose up -d
curl -I http://127.0.0.1:3000     # HTTP 200

Log in at http://<server>:3000 with the credentials above, then change the password immediately if you ever plan to expose this beyond localhost.

🔒 Same rule as Netdata/Prometheus: bind to 127.0.0.1, reach it through an SSH tunnel or an authenticated reverse proxy (kb/security/ssh-security-hardening).

Adding the Prometheus Datasource

Via the UI

  1. Connections ▸ Data sources ▸ Add data source ▸ Prometheus
  2. URL: http://prometheus:9090 — service name on the shared Compose network, not localhost
  3. Leave auth empty; set Scrape interval matching your Prometheus (15s)
  4. Save & test → green "Successfully queried the Prometheus API"

If Save & test fails here, 90% of later panel errors trace back to this one URL.

Verify with a quick query

Open Explore, pick the datasource, type up, press Run. You should see three series (prometheus itself, node-exporter, cadvisor) all with Value = 1. This confirms the whole pipeline: Grafana → Prometheus → exporters.

Pre-Built Dashboards

Before building anything yourself, check grafana.com/grafana/dashboards — thousands of community dashboards import in seconds.

Dashboard ▸ New ▸ Import, enter an ID:

ID Dashboard Needs
1860 Node Exporter Full node-exporter job
14282 / 193 Docker/cAdvisor containers cAdvisor
315 Prometheus overview prometheus self-scrape
12019 Node Exporter (simpler) node-exporter

On import:

  1. Choose a name/UID.
  2. Select your Prometheus datasource in the dropdown (community dashboards assume one exists).
  3. If the dashboard uses variables like $job or $node, confirm they resolve — Node Exporter Full auto-populates them from label values.

Node Exporter Full alone replaces most of what you'd hand-build: CPU per core, memory breakdown, disk I/O and utilization, filesystem forecasts, network throughput, systemd units. Start there; build customs only for gaps.

📌 Imported dashboards are read-only JSON copies. Editing them locally is fine — they won't receive upstream updates afterward. Re-import to refresh and diff.

Building Custom Panels

Edit ▸ Add visualization ▸ select Prometheus datasource. Every panel has three parts: query, legend, and visualization type.

Panel types that matter

Type Use for Good example queries
Time series anything trending rate(node_network_receive_bytes_total[5m])
Stat single current number count(up == 0) → "targets down"
Gauge % against known max RAM used %
Table inventory/rankings topk(10, container_memory_usage_bytes{name!=""})

Example: "RAM Used %" gauge

100 * (1 - node_memory_MemAvailable_bytes{instance="$host"} 
        / node_memory_MemTotal_bytes{instance="$host"})

Example: top memory consumers table

topk(10,
  sum by (name) (container_memory_usage_bytes{name!=""})
)

Set Instant query mode (not range) so tables show current values instead of one row per timestamp.

Panel hygiene

  1. Name panels after questions, not metrics: "Are we about to run out of disk?" not "node_filesystem_avail".
  2. Put rate() windows in variables ($__rate_interval) so they adapt to zoom level.
  3. One idea per panel — a wall of 12-line PromQL in one panel helps nobody at 3 AM.

Variables & Templating

Variables turn one dashboard into N. The classic pattern: a host dropdown feeding every query.

Dashboard settings ▸ Variables ▸ New:

Field Value
Name host
Type Query
Query (datasource: Prometheus) label_values(up{job="node"}, instance)
Multi-value ✅
Include All option ✅

Then reference $host anywhere a label value goes:

rate(node_cpu_seconds_total{instance=~"$host",mode="idle"}[$__rate_interval])

Useful variable flavors:

# list containers from cAdvisor data
label_values(container_last_seen{name!=""}, name)

# static list (env selector)
Type: Custom → prod , staging , lab

# nested: services filtered by chosen $host
label_values(process_cpu_seconds_total{instance="$host"}, group)

$__rate_interval is a magic variable that picks a rate window ≥ 4× scrape interval at the current zoom — use it everywhere instead of hardcoding [5m].

Chained variables (query A's result filters query B) are how you get prod→region→host drill-downs without duplicating dashboards per environment.

Alerting in Grafana

Grafana's unified alerting evaluates any panel-style query on a schedule and routes notifications. Flow: Alerting ▸ Alert rules ▸ New alert rule.

Rule anatomy (example: host down)

  1. Query A: up{job="node"} (last value)
  2. Expression: Reduce ▸ Last, then Threshold ▸ below 1
  3. Evaluation: every 1m, for 3m (pending period avoids flap on one bad scrape)
  4. Labels: severity=critical, team=infra
  5. Annotations: summary Host {{ $labels.instance }} stopped responding to scrapes

Good starter rules, expressed as thresholds over queries:

Disk almost full:
  100*(1 - node_filesystem_avail_bytes{mountpoint="/"}
       /node_filesystem_size_bytes{mountpoint="/"}) > 85

Memory pressure:
  100*(1 - node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes) > 90

Container restart loop (cAdvisor):
  changes(container_start_time_seconds{name!=""}[30m]) > 3

Any target down:
  up == 0

Contact points & policies

  1. Contact points: add Discord/Slack/email integration (webhook URL), then Test it before trusting it — same doctrine as netdata-basics' alarm test and simple-alerts.
  2. Notification policies: route by label — e.g. severity=critical → pager channel; everything else → daily-digest channel. Default policy catches the rest.
  3. Silences exist for maintenance windows; use them instead of disabling rules.

Anti-fatigue checklist (mirrors simple-alerts):

Provisioning as Code

Everything configured above lives in Grafana's SQLite DB by default — lost on rebuild, unversionable, unreproducible. Provisioning fixes this: Grafana loads YAML/JSON from /etc/grafana/provisioning at startup (the Compose file already mounts it). Per Grafana's docs, config files support env-var interpolation ($ENV_VAR_NAME), and each startup reconciles files into the DB.

Directory layout

provisioning/
├── datasources/
│   └── prometheus.yaml
├── dashboards/
│   └── provider.yaml
└── dashboards-json/
    ├── host-overview.json
    └── container-overview.json

provisioning/datasources/prometheus.yaml

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    uid: prom-main            # stable UID — dashboards reference this
    access: proxy             # server-side requests, not browser
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s       # matches scrape_interval

Datasources listed here are created/updated on every start — no clicking, no drift between environments.

provisioning/dashboards/provider.yaml

apiVersion: 1

providers:
  - name: default
    orgId: 1
    folder: Infrastructure
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/provisioning/dashboards-json
      foldersFromFilesStructure: false

Dashboard JSON

Export any dashboard (Share ▸ Export ▸ Save to file) and drop the JSON into dashboards-json/. Two edits make it portable:

  1. Replace "datasource": {"type":"prometheus","uid":"OLD-ID"} with "uid": "prom-main" everywhere.
  2. Keep variables intact — they export automatically.

Restart once (docker compose restart grafana) and both datasource and dashboards appear without touching the UI. From here the workflow is Git-native:

git add provisioning/ && git commit -m "grafana: add container overview board"

Rebuild the host → git clone → docker compose up -d → identical Grafana. That's the whole point of Phase 3 done right. (Kubernetes deployments use the same YAML shape via ConfigMaps — see containers/k0s-configmaps-secrets.)

Troubleshooting & Common Pitfalls

"Datasource not found" / panels empty after import

Imported dashboards carry the original author's datasource UID. Re-point each panel (or edit the JSON) to your prom-main UID — the error message names the missing UID verbatim.

Save & test fails against Prometheus

Panels show "No data" but Explore works

Variable regex mismatch: $host values come from instance labels — if your job renamed targets, the dropdown is stale. Check Dashboard settings ▸ Variables ▸ run the preview query.

Permission denied writing ./data

Grafana runs as uid 472: sudo chown -R 472:472 data. Symptom: crash loop with "migration failed" or lock errors.

Provisioned dashboard won't update

Providers cache by mtime; updateIntervalSeconds defaults can lag. Force with docker compose restart grafana, and remember: UI edits to provisioned dashboards are overwritten by files on next load — edit the JSON, not the UI.

Admin password reset

Provisioning env vars apply on first DB init only. Later resets: docker exec -it grafana grafana cli admin reset-admin-password <newpass>.

Too many series / slow dashboards

A dashboard rendering 50 heavy rate() panels will hammer Prometheus. Mitigate: smaller time ranges, fewer panels per board, recording rules in Prometheus for expensive recurring queries (out of scope here — see Prometheus docs' Recording Rules section).

Sources & Related

Web sources consulted during research (2026-08-26):

Related KB lessons:

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse