CI Monitoring - Pipeline Metrics, Failure Alerts, and Dashboards

Status: Active
Last Updated: 2026-08-26
Category: CI/CD - Phase 2: Operations
Prerequisites: woodpecker-installation, prometheus-basics, alertmanager-config
Time: 2-3 hours
Tags: woodpecker, monitoring, prometheus, alerting, grafana, metrics

Summary

A pipeline that fails silently is worse than no pipeline. Expose Woodpecker's Prometheus metrics, scrape them with your existing stack, build failure-rate and duration dashboards in Grafana, and alert when builds break or deploys stop happening.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Enable Woodpecker server metrics for Prometheus
✅ Record pipeline success rate and duration as metrics
✅ Alert on repeated failures and stalled deployments
✅ Build a CI health dashboard in Grafana

Context / Why This Matters
Implementation / Core Content
Practical Examples
Troubleshooting & Common Pitfalls
Next Steps / Ops Actions
Sources & Related

Context / Why This Matters

You already monitor hosts and services (why-monitor, node-exporter-setup). The CI system deserves the same treatment: it is the automation layer every other service depends on. If Woodpecker stops succeeding — runner down, queue wedged, disk full on the runner — nothing tells you until someone notices a stale deployment.

The good news: Woodpecker exposes native Prometheus metrics on the server, and it slots straight into the Prometheus + Alertmanager + Grafana pattern from prometheus-basics, alertmanager-config, and grafana-dashboards.

Implementation / Core Content

Enable the metrics endpoint

Woodpecker serves metrics at /metrics on the server. Restrict exposure — either keep it internal-only behind your reverse proxy or protect it with a bearer token (WOODPECKER_METRICS_AUTH_TOKEN):

# docker-compose.yml (woodpecker-server)
environment:
  - WOODPECKER_METRICS_AUTH_TOKEN=${METRICS_TOKEN}

Scrape config on your Prometheus host:

scrape_configs:
  - job_name: woodpecker
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/woodpecker_metrics.token
    static_configs:
      - targets: ["ci.fogserv.cloud:443"]
    scheme: https

Verify locally first:

curl -sH "Authorization: Bearer $METRICS_TOKEN" https://ci.fogserv.cloud/metrics | grep woodpecker_ | head

Key Woodpecker metrics

Metric	Meaning
`woodpecker_pipeline_total_builds`	Total pipelines by status (counter)
`woodpecker_pipeline_time_builds_seconds`	Build duration by status (histogram)
`woodpecker_pipeline_queue_pending_pipelines`	Pipelines waiting for a runner
`woodpecker_pipeline_queue_running_builds`	Currently executing
`woodpecker_agent_info`	Registered agents (gauge)

Derived recording rules

Add rules to compute rates and durations so dashboards stay cheap:

groups:
  - name: ci-health
    interval: 1m
    rules:
      - record: ci:pipeline_success_rate_1h
        expr: |
          sum(rate(woodpecker_pipeline_total_builds{status="success"}[1h]))
          /
          sum(rate(woodpecker_pipeline_total_builds[1h]))
      - record: ci:queue_depth
        expr: max(woodpecker_pipeline_queue_pending_pipelines)
      - record: ci:agents_online
        expr: count(woodpecker_agent_info)

Alerts

Hook into your existing Alertmanager routing (alertmanager-config):

groups:
  - name: ci-alerts
    rules:
      - alert: CIPipelineFailureSpike
        expr: ci:pipeline_success_rate_1h < 0.5 and sum(rate(woodpecker_pipeline_total_builds[1h])) > 3
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "CI success rate below 50% over the last hour"

      - alert: CIQueueBackedUp
        expr: ci:queue_depth > 5
        for: 10m
        labels:
          severity: warning
          summary: "Pipelines waiting > 5 — runner saturated or down?"

      - alert: CIAllAgentsDown
        expr: ci:agents_online == 0
        for: 5m
        labels:
          severity: critical

      - alert: CIDeploysStalled
        # No successful tagged-release pipeline in 14 days on an active repo
        expr: increase(woodpecker_pipeline_total_builds{status="success",event="tag"}[14d]) == 0
        labels:
          severity: info
        annotations:
          summary: "No successful release pipeline in two weeks"

The last rule catches "automation quietly died" — the failure mode where nothing is broken but nothing is deploying either.

Also ship Woodpecker server/agent container logs to Loki per loki-logging so alert links land somewhere greppable.

Practical Examples

Example: minimal Grafana dashboard rows

Build one dashboard "CI Health" with four panels (grafana-dashboards covers layout):

# Panel 1: Success rate (stat)
ci:pipeline_success_rate_1h * 100

# Panel 2: Builds per hour by status (timeseries)
sum by (status) (rate(woodpecker_pipeline_total_builds[1h]))

# Panel 3: p95 duration (timeseries)
histogram_quantile(0.95,
  sum by (le) (rate(woodpecker_pipeline_time_builds_seconds_bucket{status="success"}[6h])))

# Panel 4: Queue depth + agents (timeseries)
ci:queue_depth
ci:agents_online

Expected reading: success rate pinned near 100% on quiet days; a dip to ~40% usually means one broken branch being hammered, not systemic failure — check the status dimension before paging anyone.

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
`/metrics` returns 404/401	Metrics token not set or wrong auth header	Set `WOODPECKER_METRICS_AUTH_TOKEN` and send matching Bearer token
Metrics show zero builds after restart	Counters reset; storage-backed series need time to repopulate	Expected behavior — rely on `rate()` windows, not absolute counters
Failure-spike alert fires constantly	One noisy repo with flaky tests dominates rate	Add per-repo recording rules and exclude known-flaky repos from the global alert
Agents online gauge = 1 but builds queue	Agent registered but workflow slots exhausted	Check `WOODPECKER_MAX_WORKFLOWS` on agent; see matrix-builds concurrency section
Alert fatigue	Severity everything `critical`	Reserve critical for all-agents-down; route warnings to digest channel per alertmanager-config

Next Steps / Ops Actions

Gate what gets deployed before worrying about how fast it deploys: security-scanning
Keep runners fast enough that queue alerts stay quiet: caching-strategies

Sources & Related

External references consulted:

Change Log

2026-08-26

Initial creation by KB writing session.