fogserv.cloud fogserv.cloud

Loki Logging - Centralized Log Shipping and LogQL

Status: Active
Last Updated: 2026-08-26
Category: Observability - Logs
Prerequisites: prometheus-basics, grafana-dashboards, docker-compose-intro
Time: 2-3 hours
Tags: loki, promtail, alloy, logql, logging, grafana, docker

Summary

Loki stores logs as label-indexed streams — "Prometheus, but for logs" — and queries them with LogQL from Grafana. This article deploys Loki plus a log agent (Alloy or Promtail) to ship Docker container and system logs, wires the Grafana datasource, covers the LogQL patterns you'll use daily, and configures retention so the disk doesn't fill.

What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. Context / Why This Matters
  2. Implementation / Core Content
  3. Practical Examples
  4. Common Pitfalls & Troubleshooting
  5. Next Steps / Ops Actions

Context / Why This Matters

metrics-vs-logs.md established that logs answer "what exactly happened?" while metrics answer "is it healthy?". Until now your logs have lived in docker logs on each host — fine for one box, useless when a problem spans three machines or survives a docker rm. Loki gives you one queryable place for all of it, reusing the Grafana instance you already run (grafana-dashboards.md) and the same label mental model as Prometheus (prometheus-basics.md).

Key architectural difference from Elasticsearch: Loki indexes only labels, not line content. Storage is cheap (compressed chunks in object store or local disk); querying relies on time + label selection first, then filtering text. That's why low-cardinality labels and structured logs matter even more here than in Prometheus.

Implementation / Core Content

Architecture in one diagram

containers → docker json log files ─┐
journald / syslog ──────────────────┼→ Alloy/Promtail (agent) → HTTP push → Loki → Grafana
app files (/var/log/...) ───────────┘

The agent tails files locally, applies pipelines (parse/drop/relabel), and pushes batches to Loki's /loki/api/v1/push endpoint.

Docker Compose deployment

services:
  loki:
    image: grafana/loki:3.1.1
    container_name: loki
    restart: unless-stopped
    ports:
      - "127.0.0.1:3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/config.yml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/config.yml

  alloy:
    image: grafana/alloy:v1.2.1
    container_name: alloy
    restart: unless-stopped
    user: "0"                       # needs root to read docker logs
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log:/var/log:ro
      - ./alloy-config.alloy:/etc/alloy/config.alloy:ro
    command: run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy

volumes:
  loki-data:

Prefer the legacy agent? Swap the alloy service for:

  promtail:
    image: grafana/promtail:3.1.1
    container_name: promtail
    restart: unless-stopped
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/log:/var/log:ro
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml

Promtail is in LTS-only maintenance; new deployments should use Alloy. The concepts below map one-to-one (pipeline stages ≈ processing stages), and both are covered.

Loki server config (loki-config.yml)

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory          # single-node only

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb               # tsdb is the current default engine
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 720h        # 30 days
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_query_series: 500

compactor:
  working_directory: /loki/compactor
  retention_enabled: true       # REQUIRED or retention_period is ignored

The classic gotcha is right there in the comment: setting retention_period without compactor.retention_enabled: true silently keeps data forever.

Agent config — Alloy river syntax (alloy-config.alloy)

// Discover all running containers
discovery.docker "linux" {
  host             = "unix:///var/run/docker.sock"
  refresh_interval = "5s"
}

discovery.relabel "docker" {
  targets = discovery.docker.linux.targets

  // label = container name
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
  // label = compose project
  rule {
    source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
    target_label  = "compose_project"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.docker.output
  forward_to = [loki.process.docker.receiver]
  labels     = { job = "docker" }
}

// Parse the docker json wrapper so level/message are searchable
loki.process "docker" {
  stage.json {
    expressions = { level = "log", stream = "stream", time = "time" }
  }
  stage.labels {
    values = { stream = "" }   // promote stdout/stderr to a label
  }
  // keep ERROR/WARN lines longer by tagging them; drop chatty DEBUG entirely
  stage.match {
    selector   = "{job=\"docker\"}"
    stages {
      stage.drop {
        expression = "\"level\":\"DEBUG\""
      }
    }
  }
  forward_to = [loki.write.default.receiver]
}

// Also ship kernel/auth logs via journal
loki.source.journal "host" {
  max_age    = "12h"
  labels     = { job = "journal", host = env("HOSTNAME") }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

Equivalent Promtail snippet for the same docker scrape (scrape_configs section):

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        regex: "/(.*)"
        target_label: container
      - source_labels: [__meta_docker_container_label_com_docker_compose_project]
        target_label: compose_project
    pipeline_stages:
      - json:
          expressions:
            level: log
            stream: stream
      - labels:
          stream:
      - drop:
          expression: '"level":"DEBUG"'

Grafana datasource

Add via UI (Connections → Data sources → Loki → URL http://loki:3100) or provision:

apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    isDefault: false

Verify with Explore → pick Loki → query {job="docker"} — you should see live tailing output within seconds.

LogQL essentials

LogQL has two layers: a log selector (labels, like PromQL selectors) then optional filters/parsing:

# All lines from one container
{container="nginx-proxy"}

# Label match + line contains
{job="docker", container="api"} |= "error"

# Chained filters: contains but not this
{job="docker"} |= "error" != "healthcheck"

# Regex filter (case-insensitive)
{job="docker"} |~ "(?i)(fatal|panic)"

# Parse JSON and filter/return parsed fields
{container="api"} | json | level="ERROR" | line_format "{{.time}} {{.message}}"

# Count matches over time — logs become metrics!
sum(rate({job="docker"} |= "error" [5m])) by (container)

# Top offenders: error count per container in the last hour
topk(5, sum(count_over_time({job="docker"} |= "error" [1h])) by (container))

That last family of expressions is the bridge back to metrics: you can graph and alert on log-derived rates directly in Grafana panels, no Prometheus required.

Practical Examples

Example 1: Investigate a 500-spike end to end

1. Metrics panel shows nginx 5xx rate climbing at 14:02.
2. Explore → Loki: {container="nginx-proxy"} |~ " 5[0-9][0-9] "
   → upstream timed out while reading response header
3. Pivot to the app: {container="api"} |= "timeout" | json | line_format "{{.msg}}"
   → waiting on postgres pool, pool_size=10 exhausted
4. Fix known issue / restart; confirm error rate returns to baseline.
Total time: minutes, across two hosts' logs, zero SSH.

Example 2: Log-derived alert panel

Grafana alert (or Loki ruler) on errors appearing in any production container:

sum by (container) (count_over_time({job="docker", compose_project="prod"} |= "FATAL" [5m])) > 0

Example 3: Verify retention actually deletes

# Check oldest chunk timestamp still present after a week:
curl -s 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query=sum(bytes_over_time({job="docker"}[24h]))' \
  --data-urlencode "start=$(date -d '35 days ago' +%s)000000000" \
  --data-urlencode "end=$(date -u +%s)000000000" | jq .
# Empty result beyond ~30 days == compaction working

Common Pitfalls & Troubleshooting

Problem Cause Fix
Loki ignores retention_period, disk grows forever compactor.retention_enabled not set Add it under compactor: and restart
"entry out of order" rejected Two agents shipping same file, or clock skew behind NAT Run exactly one agent per source; NTP-sync hosts
Too many streams error / Loki OOMs High-cardinality labels (request ID, trace ID as labels) Keep labels to container/compose_project/host/stream; put IDs in the line
No docker logs visible Alloy lacks socket/container dir mounts, or not root Mount /var/run/docker.sock and /var/lib/docker/containers; user: "0"
Duplicate log lines Same file scraped by both journal and file sources Pick one source per log stream
Queries slow on big time ranges Scanning months of chunks Narrow the time range first; add label selectors before line filters
DEBUG noise fills storage App log level too verbose Set level in app config, or add stage.drop in the pipeline

Next Steps / Ops Actions

Sources & Related Articles

External references consulted:

Related knowledge-base articles:

Change Log

2026-08-26

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse