fogserv.cloud fogserv.cloud

Monitoring Networks - Watching the Wires

Status: Active
Last Updated: 2026-08-26
Category: Networking - Production Patterns
Prerequisites: tcp-ip-fundamentals, network-segmentation, why-monitor
Time: 2-3 hours
Tags: networking, monitoring, snmp, prometheus, blackbox-exporter, smokeping, latency, packet-loss, vlan

Summary

Host monitoring tells you a server is sick; network monitoring tells you where it hurts. This article covers the four practical layers of network observability for a self-hosted stack — availability checks, latency/packet-loss trending, interface and bandwidth metrics (SNMP), and flow visibility — wired into the existing Prometheus/Grafana/Uptime Kuma setup from the observability section.

Context / Why This Matters

When "the site is slow," the question is always: is it DNS, the path, the proxy, or the app? Without network telemetry you guess. With it, a Grafana panel answers in seconds: DNS resolution jumped 400 ms from the probe location, or WAN packet loss hit 5% at 14:02, or the uplink saturated at 940 Mbps.

This builds on network-segmentation: once VLANs exist, each segment boundary becomes a place where things can silently break — and a place worth watching.

Implementation / Core Content

Layer 1 — Availability: Blackbox Probes

Prometheus' blackbox_exporter probes ICMP/TCP/HTTP from your monitoring host:

# docker-compose.yml addition
services:
  blackbox:
    image: prom/blackbox-exporter:v0.25.0
    restart: unless-stopped
    cap_add: [NET_RAW]
    ports: ["9115:9115"]

# prometheus.yml — probe targets through the blackbox exporter
scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params: { module: [http_2xx] }
    static_configs:
      - targets:
          - https://fogserv.cloud
          - http://192.168.0.104:3000
          - https://git.shire.one
    relabel_configs:
      - { source_labels: [__address__], target_label: __param_target }
      - { source_labels: [__param_target], target_label: instance }
      - { target_label: __address__, replacement: blackbox:9115 }

  - job_name: blackbox-icmp
    metrics_path: /probe
    params: { module: [icmp] }
    static_configs:
      - targets: ["192.168.0.1", "192.168.0.104", "1.1.1.1"]
    relabel_configs: &icmp_rel
      - { source_labels: [__address__], target_label: __param_target }
      - { source_labels: [__param_target], target_label: instance }
      - { target_label: __address__, replacement: blackbox:9115 }

Key queries:

probe_success == 0                                  # anything down right now
histogram_quantile(0.95, rate(probe_duration_seconds_bucket[5m]))   # p95 latency
probe_dns_lookup_time_seconds > 0.1                 # slow DNS, not slow app

For simple needs, uptime-kuma-setup already covers HTTP/TCP/ping checks with notifications; use blackbox when you want the data in PromQL next to everything else.

Layer 2 — Latency & Loss Trends: smokeping-style History

Point-in-time pings miss patterns ("every evening at 19:00 the WAN gets lossy"). Keep continuous ping history per hop:

Grafana panel: avg_over_time(probe_duration_seconds[10m]) per target, plus 1 - avg_over_time(probe_success[10m]) as packet loss %.

Layer 3 — Interface & Bandwidth Metrics: SNMP

Routers, switches, and APs speak SNMP even when they can't run exporters:

# Enable SNMP on the device (community v2c on a management VLAN only),
# then test from the monitor host:
snmpwalk -v2c -c public 192.168.0.1 IF-MIB::ifDescr

Scrape via SNMP exporter:

  - job_name: snmp-router
    metrics_path: /snmp
    params: { module: [if_mib] }
    static_configs:
      - targets: ["192.168.0.1"]
    relabel_configs:
      - { source_labels: [__address__], target_label: __param_target }
      - { source_labels: [__param_target], target_label: instance }
      - { target_label: __address__, replacement: snmp-exporter:9116 }

Useful panels: rate(ifHCInOctets{ifDescr="eth0"}[5m]) * 8 (bits/sec in), errors/discards counters (rate(ifInErrors[5m]) > 0 = real cable/driver problems).

Layer 4 — "Who's Talking": Flow & Quick Captures

Alerting Rules That Earn Their Keep

See alertmanager-config and simple-alerts for routing; network-specific ones:

- alert: HostUnreachableFromMonitor
  expr: probe_success{job="blackbox-icmp"} == 0
  for: 2m
  labels: { severity: critical }

- alert: WanPacketLoss
  expr: 1 - avg_over_time(probe_success{instance="1.1.1.1"}[5m]) > 0.02
  for: 10m
  labels: { severity: warning }

- alert: InterfaceErrors
  expr: rate(ifInErrors[10m]) > 0
  for: 15m
  labels: { severity: warning }        # flaky cable/SFP, not an outage yet

Probe from more than one vantage point (monitor host + one remote/VPS peer) so you can distinguish "server down" from "my ISP is down."

Practical Examples

Localizing "the site is slow"

  1. Blackbox panel shows https://fogserv.cloud p95 rose from 80 ms → 900 ms at 14:00.
  2. Same probe against LAN IP 192.168.0.104:3000 stayed flat → not the app.
  3. Probe to 1.1.1.1 shows 3% loss starting 13:58 → WAN/ISP event, nothing to fix locally; note it and move on.
  4. If instead LAN probe degraded too: check node CPU/disk first (node-exporter-setup), then switch port error counters via SNMP.

Verifying a new VLAN after segmentation

After carving out an IoT VLAN (dhcp-dns-split-horizon), add its gateway to the icmp probe list and one client IP to TCP probes. Two days of green graphs prove the firewall rules didn't break mDNS/DNS before anyone complains.

Common Pitfalls & Troubleshooting

Next Steps / Ops Actions

Sources & Related Articles

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse