fogserv.cloud fogserv.cloud

Metrics vs Logs vs Traces - Choosing the Right Signal for the Job

Status: Active
Last Updated: 2026-08-26
Category: Observability - Fundamentals
Prerequisites: why-monitor, prometheus-basics
Time: 1 hour
Tags: observability, metrics, logs, traces, cardinality, architecture

Summary

Metrics, logs, and traces are three different answers to three different questions. This article explains when each signal type is the right tool, how cardinality drives cost, and which fogserv.cloud component handles each signal so you stop feeding logs into Prometheus or trying to graph your way out of a debugging session.

What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Context / Why This Matters
  2. Implementation / Core Content
  3. Practical Examples
  4. Common Pitfalls & Troubleshooting
  5. Next Steps / Ops Actions

Context / Why This Matters

why-monitor established that we monitor; prometheus-basics covered how we collect one specific kind of telemetry. The gap most homelab operators fall into is treating "monitoring" as one thing. Then they either scrape per-request log lines as metrics (Prometheus explodes), or grep gigabytes of logs to answer "is the CPU high right now?" (wrong tool, slow answer).

Each signal type answers a different class of question:

Signal Question Shape
Metrics "How much / how many / is it healthy?" Numbers over time, cheap to store, cheap to query
Logs "What exactly happened at 03:14?" Discrete events with full detail, expensive to store
Traces "Where did this request spend its time?" Causal chains across services

Getting the mapping right early keeps your stack small and your bills (or disk usage) flat.

Implementation / Core Content

Metrics: pre-aggregated numbers over time

A metric is a number sampled on a schedule — node_cpu_seconds_total, HTTP request counts, queue depths. The aggregation happens before storage: you keep counters/histograms/gauges, not individual events.

Strengths:

Weaknesses:

Cardinality: the metric that kills Prometheus clusters

Cardinality = the number of unique time series created by your label combinations. Every distinct label value multiplies series count:

http_requests_total{service="api"}                          → 1 series
http_requests_total{service="api", status="200"|"500"}      → 2 series
... add user_id with 10,000 users                           → 20,000 series

Rules of thumb:

Before adding a label, ask: "Will I ever want to aggregate along this dimension?" If you only need it for debugging an individual event, it belongs in logs or traces instead.

Logs: the ground truth of events

Log lines are the only place where the complete record survives: the actual SQL statement, the exact payload, the stack trace. Use them when you know roughly when something happened and need to know what happened.

Cost profile is inverted from metrics: storage scales linearly with traffic volume, and querying means scanning text (indexed by time + labels in Loki's case, not full-text by default).

Discipline that pays off:

Traces: causality across services

A trace follows one request through everything it touched: gateway → auth service → database → cache. Each step is a span with timing. Traces answer latency questions ("p99 doubled — which hop regressed?") and are the only signal that shows work happening in parallel vs sequentially.

The catch: instrumentation is opt-in. Metrics come free from node_exporter; traces require code changes or an auto-instrumentation agent. That's why traces come last in the maturity ladder.

Maturity ladder for a small fleet

Stage 1  Uptime checks          → Uptime Kuma        (is it reachable?)
Stage 2  Node/container metrics → Prometheus         (is it healthy? trending?)
Stage 3  Dashboards             → Grafana            (visualize stage 2)
Stage 4  Alerts                 → Alertmanager       (tell me when stage 2 breaks)
Stage 5  Logs                   → Loki               (why did it break?)
Stage 6  Traces                 → Jaeger             (where in the request path?)

Most homelabs should be at Stage 4 within a weekend and can defer Stages 5–6 until they run more than two or three interdependent services. Netdata (netdata-basics.md) overlaps Stage 2 with zero-config per-node dashboards — fine to run alongside Prometheus, but pick one as the alerting source of truth.

Mapping fogserv.cloud signals to tools

Signal Producer Backend Article
Host CPU/RAM/disk/network node_exporter Prometheus node-exporter-setup
Docker container metrics cAdvisor / docker engine Prometheus prometheus-basics
Service reachability (black-box) Uptime Kuma probes Uptime Kuma (+ optional Push to Prometheus) uptime-kuma-setup
Container/app stdout+stderr Promtail/Alloy Loki loki-logging
Request flows between services OpenTelemetry SDK Jaeger jaeger-tracing
Threshold notifications Prometheus rules Alertmanager alertmanager-config
Visual layer for all of the above Grafana datasources Grafana grafana-dashboards

Cost/storage tradeoffs in practice

For a 3-node fleet with ~40 containers, realistic retention sizing:

The general principle: aggressiveness of retention should be inverse to how much you'd miss it. Keep alerts' underlying metrics long enough to compare against last month; keep debug logs just long enough to cover your incident-response window.

Practical Examples

Example 1: Same outage, three signals

API latency spikes at 14:00. How each signal contributes:

# METRICS (Prometheus): detect and quantify
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
# → confirms p95 went from 120ms to 2.4s starting 13:58

# LOGS (Loki): explain
{job="docker", container="api"} |= "error" | json | level="ERROR"
# → "connection refused postgres-primary:5432" repeated since 13:57

# TRACES (Jaeger): locate
Filter service=api, duration > 1s
# → spans show db.connect() retries consuming 2.1s per request

Three queries, three complementary halves of the story — detection, explanation, localization.

Example 2: Spotting a cardinality bomb before it lands

Check your highest-cardinality series before and after adding labels:

topk(10, count by (__name__)({__name__=~".+"}))

If http_requests_total jumped from 200 to 200k series after a deploy, find the recently added label:

count(count by (user_id) (http_requests_total)) # if huge, remove user_id

Fix: move the identifier into the log line (where Loki indexes it cheaply) rather than the metric label.

Common Pitfalls & Troubleshooting

Problem Cause Fix
Prometheus RAM climbs steadily after a deploy High-cardinality label (request ID, URL path with IDs) added to a metric Drop the label via metric_relabel_configs or fix the code; check with topk(10, count by (__name__)({...}))
"Grep the logs" takes minutes during incidents Using logs where metrics would answer the health question Add the aggregate metric first, keep logs for the drill-down step
Alerts fire but nobody knows why Alerting on symptoms without correlated log context Include a Grafana Explore link to the relevant LogQL query in the alert annotation
Traces show nothing useful Sampling too low, or only one instrumented hop in a multi-service path Raise sampling during debugging; instrument both ends of the slow boundary
Loki fills the disk DEBUG-level application logging left enabled in production Set log level at the app, or use Promtail pipeline stages to drop DEBUG before shipping
Duplicate dashboards in Netdata and Grafana disagree Different scrape intervals/sources Pick Prometheus as canonical for alerts; treat Netdata as per-node live view

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse