Failure Recovery Patterns - Retries, Checkpoints, and Resuming Agent Work

Status: Active
Last Updated: 2026-08-26
Category: Agentic - Patterns
Prerequisites: agentic-workflows, multi-agent-communication
Time: 2 hours
Tags: agentic, reliability, retry, backoff, idempotency, checkpoints, dead-letter

Summary

How to design agent work so failures are recoverable instead of catastrophic: exponential backoff with jitter, checkpoint files that survive context loss, idempotent step design, dead-letter handling for poison tasks, and procedures for resuming interrupted runs.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Implement retry with exponential backoff + jitter correctly
✅ Structure work into idempotent, resumable steps
✅ Write checkpoints that let a fresh agent session continue mid-task
✅ Route poison jobs to a dead-letter area with diagnosis data
✅ Resume interrupted work without redoing or duplicating effects

Context / Why This Matters
Retry with Backoff
Idempotent Steps
Checkpoints
Dead-Letter Handling
Resuming Interrupted Work

Context / Why This Matters

Agent sessions die — timeouts, context limits, host reboots, upstream API outages. The KB's core discipline from agentic-workflows is that each turn must leave a shippable state; this article supplies the mechanics: what to persist, how to retry safely, and how a different agent (or a fresh context of the same one) picks up exactly where things stopped.

The design rule underneath everything: assume any step can run zero, one, or many times. Systems built on that assumption recover; systems assuming exactly-once do not.

Implementation / Core Content

1. Retry with Backoff

Blind immediate retries amplify outages. Standard recipe:

delay(n) = min(base * 2^n + jitter, max_delay)

#!/bin/bash
# retry.sh CMD... — up to 5 attempts, exp backoff + jitter, cap 5m
MAX=5; BASE=2; CAP=300
for attempt in $(seq 1 $MAX); do
  if "$@"; then exit 0; fi
  rc=$?
  if [ "$attempt" -eq "$MAX" ]; then echo "giving up after $MAX attempts" >&2; exit $rc; fi
  delay=$(( BASE ** attempt + RANDOM % BASE ))
  [ $delay -gt $CAP ] && delay=$CAP
  echo "attempt $attempt failed (rc=$rc); sleeping ${delay}s" | logger -t retry
  sleep $delay
done

Rules:

Jitter is not optional. Without it, all agents retry in lockstep and DDoS the recovering service.
Only retry transient errors (timeouts, 429/503, network). A 401 or a syntax error will fail identically forever — fail fast to dead-letter instead.
Budget total time, not attempts. "Retry until 10 minutes elapsed" survives slow upstreams better than "3 tries."
Honor Retry-After headers when present.

2. Idempotent Steps

An operation is idempotent when running it again after success changes nothing. Techniques:

Technique	Example
Deterministic output names	Write to `backups/db-2026-08-26.sql.gz` — rerun overwrites same file
Existence check before effect	Skip restore if schema marker table exists at target version
Idempotency keys	Send `X-Idempotency-Key: <task-id>` to APIs that support it
Create-if-absent primitives	`mkdir -p`, `INSERT ... ON CONFLICT DO NOTHING`
Move-based completion	Write `.tmp`, then atomic `mv` to final name

Classify every step:

Naturally idempotent (reads, deterministic writes) — safe to retry freely.
Effectful but keyable (API calls, emails) — wrap with keys/checks.
Non-idempotent (rm, transfers without checksums) — gate behind a recorded "step done" marker so retries never reach them twice.

3. Checkpoints

A checkpoint is durable state written before relying on it, letting a fresh process resume. Keep it simple and human-readable:

// /srv/scratch/migration-42/state.json  (write via tmp+mv)
{
  "task": "migration-42",
  "updated_at": "2026-08-26T11:30:00Z",
  "steps": {
    "backup":      {"status": "done"},
    "schema-v3":   {"status": "done", "evidence": "applied 2026-08-26T11:12Z"},
    "data-copy":   {"status": "running", "cursor": {"last_id": 18423}},
    "verify":      {"status": "pending"}
  }
}

Checkpoint discipline:

Update the checkpoint after each step completes, atomically (state.json.tmp → mv). A torn checkpoint is worse than none.
Store enough to resume: cursors for batch work, applied-version markers for migrations.
Record evidence, not intentions ("done" must mean verifiably done).
The first thing a resumed agent does is read the checkpoint — never trust memory over disk.

4. Dead-Letter Handling

Some tasks exhaust retries: malformed input, unreproducible API errors, contradictions in instructions. Retry harder and you just burn tokens. Route them aside:

/srv/queue/
  new/       # pending tasks, one JSON per file
  done/      # completed
  dead/      # failed permanently, kept for autopsy
    task-77.attempts5.last-error-timeout.json

A dead-letter entry must contain: original payload, error history (per-attempt), timestamps, and agent identity. Naming the failure in the filename makes triage a single ls.

Then handle the dead letter like any ticket: classify (transient-but-exhausted vs. genuinely poisoned), fix root cause, and either discard, repair-and-requeue, or escalate to a human. Never silently delete.

5. Resuming Interrupted Work

Procedure for an agent picking up unknown-state work:

Read the checkpoint / scratchpad first (multi-agent-communication, handoff discipline).
Reconcile claims vs. reality. The checkpoint says data-copy running — did it finish? Verify with side-effect evidence (row counts, file mtimes, API state), not optimism.
Mark ambiguous steps done-or-redo based on idempotency class: idempotent steps — just redo them; non-idempotent — investigate before acting.
Continue from the first non-done step, updating the checkpoint per step.
Log the resume itself, so audit trails show discontinuities.

Practical Examples

Example 1: Idempotent batch sync with cursor checkpointing

#!/bin/bash
STATE=/srv/scratch/sync/state.json
CURSOR=$(jq -r '.cursor // 0' "$STATE")
LAST=$(fetch_batch --after-id "$CURSOR" > batch.json && jq '.max_id' batch.json)
apply_items_from batch.json          # ON CONFLICT DO NOTHING → safe on retry
echo "{\"cursor\": $LAST}" > "$STATE.tmp" && mv "$STATE.tmp" "$STATE"

Kill it anywhere; rerunning resumes from $CURSOR. Duplicate applies are absorbed by conflict handling.

Example 2: Supervisor-level retry with dead-letter fallback

if ! retry.sh ./worker.sh "$(cat task.json)"; then
  mv task.json dead/task-$(date +%s).json
  echo '{"event":"dead_letter","task":"'"$TASK_ID"'"}' | logger -t supervisor
fi

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Duplicate emails/deploys after crash-retry	Non-keyed effectful step retried	Idempotency keys; check-before-execute
Resumed agent redoes half the migration	No checkpoint, or trusted memory	Persist per-step status with evidence
Retry storm during outage	No jitter/backoff cap	Exponential + jitter + total-time budget
Task stuck "in progress" forever	Worker died holding claim	Lease TTLs + reaper (multi-agent-communication)
Poison task loops for hours	Retrying non-transient errors	Classify errors; dead-letter fast on permanent ones
Torn state.json confuses recovery	In-place writes	Always write temp + atomic rename

Next Steps / Ops Actions

Add checkpoints to every multi-step fogserv.cloud automation (backups, migrations) scheduled per scheduler-patterns.
Wire dead-letter directories into alerting per simple-alerts — a non-empty dead/ should page someone eventually.
Keep workflow governance aligned with agentic-workflows.

Sources & Related Articles

External references consulted:

Change Log

2026-08-26

Initial creation covering retry/backoff, idempotency, checkpoints, dead-letter handling, and resume procedures.