Jaeger Tracing - Distributed Tracing with OpenTelemetry

Status: Active
Last Updated: 2026-08-26
Category: Observability - Tracing
Prerequisites: metrics-vs-logs, loki-logging, docker-compose-intro
Time: 3-4 hours
Tags: jaeger, tracing, opentelemetry, otlp, spans, docker

Summary

Jaeger collects and visualizes distributed traces: the end-to-end journey of a single request across services. This article explains traces and spans, deploys Jaeger all-in-one in Docker, instruments a small application with OpenTelemetry, and shows how to jump from a slow span to the exact log lines via trace ID correlation.

What You'll Learn

By the end of this article, you'll be able to:

Explain traces, spans, context propagation, and sampling
Run Jaeger all-in-one with OTLP ingestion
Auto-instrument an app (Node.js or Python) with the OTel SDK
Find latency bottlenecks in the Jaeger UI
Correlate traces with Loki logs using injected trace IDs

Context / Why This Matters
Implementation / Core Content
Practical Examples
Common Pitfalls & Troubleshooting
Next Steps / Ops Actions

Context / Why This Matters

Metrics tell you p95 got worse; logs tell you what one service printed; neither tells you where in the request path time went when a request touches five services. That's the gap tracing fills — see the signal-choice rationale in metrics-vs-logs.md. Adopt it once your stack has real inter-service request paths (API → database, API → worker queue). If everything is a single container hitting one database, finish loki-logging.md first.

The modern standard here is OpenTelemetry (OTel): vendor-neutral SDKs + the OTLP wire protocol. Instrument once against OTel; Jaeger, Grafana Tempo, or anything else can consume it later. Jaeger itself no longer accepts its old Thrift protocols by default — OTLP is the way in.

Implementation / Core Content

Core concepts

TRACE  = one request's full journey, identified by a 128-bit trace_id
SPAN   = one timed operation within it (name, start, duration, status)
         ├── span: http.request  (api-gateway)          12ms
         │    ├── span: auth.verify (auth-service)       3ms
         │    └── span: db.query (api)                  8ms   ← the bottleneck
Parent/child links form a tree; spans carry attributes (key/values).
CONTEXT PROPAGATION = trace_id travels between services in HTTP headers
         (traceparent: 00-<trace_id>-<span_id>-01) so children attach correctly.
SAMPLING = you record only a fraction of traces; head sampling decides at
         request start, tail sampling after seeing the whole trace.

Deploying Jaeger all-in-one

All-in-one bundles collector, query, UI, and in-memory storage — perfect for a homelab; traces are lost on restart unless you mount Badger storage:

services:
  jaeger:
    image: jaegertracing/all-in-one:1.60
    container_name: jaeger
    restart: unless-stopped
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      # optional: persist traces to disk instead of memory
      - SPAN_STORAGE_TYPE=badger
      - BADGER_EPHEMERAL=false
      - BADGER_DIRECTORY_VALUE=/badger/data
      - BADGER_DIRECTORY_KEY=/badger/key
    ports:
      - "127.0.0.1:16686:16686"   # UI
      - "127.0.0.1:4317:4317"     # OTLP gRPC
      - "127.0.0.1:4318:4318"     # OTLP HTTP
    volumes:
      - badger-data:/badger
volumes:
  badger-data:

Verify: open http://<host>:16686, pick any service (none yet — instrument next), and confirm the collector answers curl localhost:4318 (any response means it's listening).

Instrumenting a Node.js service

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
            @opentelemetry/exporter-trace-otlp-http

tracing.js — loaded before anything else (node --import ./tracing.js server.js):

const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

const sdk = new NodeSDK({
  serviceName: process.env.OTEL_SERVICE_NAME || "api",
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT, // http://jaeger:4318/v1/traces
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Run it:

OTEL_SERVICE_NAME=api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318/v1/traces \
node --require ./tracing.js server.js

Auto-instrumentation covers express/http/pg/redis/etc. with zero code changes — spans appear for every inbound request and outbound DB call automatically. Add manual spans only around custom business operations:

const tracer = require("./tracer"); // trace.getTracer("api")
async function processOrder(order) {
  return tracer.startActiveSpan("order.process", async (span) => {
    span.setAttribute("order.id", order.id);
    try {
      const result = await chargeAndShip(order);
      span.setStatus({ code: 1 }); // OK
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: 2, message: err.message }); // ERROR
      throw err;
    } finally {
      span.end();
    }
  });
}

Instrumenting a Python service

pip install opentelemetry-distro opentelemetry-exporter-otlp \
            opentelemetry-instrumentation-flask opentelemetry-instrumentation-psycopg2

opentelemetry-instrument \
  --service_name worker \
  --exporter_otlp_endpoint http://jaeger:4318/v1/traces \
  python worker.py

Same pattern as Node: the opentelemetry-instrument launcher auto-patches supported libraries.

Cross-service propagation

Nothing to configure if both services use auto-instrumentation: outgoing HTTP calls get the traceparent header injected, incoming requests extract it, and Jaeger stitches the tree. Only custom transports (raw sockets, queues without instrumentation) need manual extraction/injection of the W3C propagator.

Correlating traces with Loki logs

Inject trace IDs into log lines so {container="api"} |= <trace_id> works from both directions:

// Pino + OTel: add trace context to every log record
const pino = require("pino");
const { trace, context } = require("@opentelemetry/api");

function logWithTrace(logger, obj, msg) {
  const span = trace.getSpan(context.active());
  if (span) {
    const sc = span.spanContext();
    obj.trace_id = sc.traceId;
    obj.span_id = sc.spanId;
  }
  logger.info(obj, msg);
}

Python equivalent with the standard logging filter:

from opentelemetry import trace
class TraceIdFilter(logging.Filter):
    def filter(self, record):
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else "-"
        return True

In Loki you then search {job="docker"} |= "a1b2c3..." for the trace ID copied from Jaeger; from a log line, paste the trace ID into Jaeger's search box. This closes the loop described in metrics-vs-logs.md.

Sampling configuration

For all-in-one testing keep 100% sampling (SDK default is parent-based always-on). Once comfortable, cut volume via env vars:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1        # keep ~10%

Errors still slip through at 10% — that's expected with head sampling; tail-based sampling (needs the Jaeger remote-storage/opentelemetry-collector-contrib layer) is the upgrade path if that matters.

Practical Examples

Example 1: Two-service demo in 10 minutes

docker compose up -d jaeger
# terminal 1
OTEL_SERVICE_NAME=frontend OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces \
  node --require ./tracing.js frontend.js   # listens :8080, proxies /api
# terminal 2
OTEL_SERVICE_NAME=api OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces \
  node --require ./tracing.js api.js        # sleeps 200ms then queries pg
curl http://localhost:8080/api/orders

Open localhost:16686 → Service frontend → Find Traces: one trace, two services, waterfall showing api's sleep dominating.

Example 2: Find the N+1 query

Filter traces by duration > 500ms, open one, expand db.query spans: 40 nearly identical SELECT ... FROM order_items WHERE order_id = $1 spans under one handler = classic N+1. The trace names the exact handler span — something a metrics histogram could never reveal.

Example 3: Trace-to-log drill-down

Copy trace_id from a slow trace → Grafana Explore → Loki:

{job="docker"} |= "6e0c5d2f9b4a47f18d3e2a77c1b0f9e4"

Every line the request logged, in order, across containers — the incident report writes itself.

Common Pitfalls & Troubleshooting

Problem	Cause	Fix
No traces appear	SDK loaded after server starts, or wrong OTLP endpoint	Import/init tracing first; endpoint must include `/v1/traces` on port 4318
Spans exist but are orphaned per service	Context not propagated between hops	Ensure HTTP client libs are instrumented (auto-instrumentations), not raw fetch wrappers
All traces vanish after restart	In-memory all-in-one storage	Use Badger storage as configured above, accept loss, or move to Elasticsearch/Tempo backend
Collector rejects old Thrift clients	Jaeger v2/OTLP-only default	Send OTLP (4317/4318); update agent configs
Trace data floods disk/memory	Sampling left at 100% in production	Set `parentbased_traceidratio`; monitor Jaeger container resources first
Manual spans missing children	Span created but `.end()` never called, or not using active context	Always call `span.end()`; prefer `startActiveSpan` so children nest
High overhead on hot paths	Synchronous exporter per span	Batch processor (default in SDKs); raise batch delay, lower sample rate

Next Steps / Ops Actions

Re-read the signal mapping to place tracing correctly in maturity order: metrics-vs-logs
Alert on error-rate metrics rather than individual traces: simple-alerts
Keep shipped logs carrying trace_id: loki-logging
When moving toward Kubernetes, evaluate Grafana Tempo instead of all-in-one: k0s-monitoring

Sources & Related Articles

External references consulted:

Change Log

2026-08-26

Initial creation by kb-writing session (ox-alpha).