Multi-Agent Communication - Coordination Patterns for Agent Fleets

Status: Active
Last Updated: 2026-08-26
Category: Agentic - Patterns
Prerequisites: agentic-workflows
Time: 2 hours
Tags: agentic, multi-agent, coordination, message-passing, blackboard, concurrency

Summary

Concrete patterns for how multiple agents coordinate on shared work: blackboard stores, message passing, supervisor/worker hierarchies, and shared scratchpad files — plus the one rule that prevents most multi-agent failures: one writer per resource.

🎯 What You'll Learn

By the end of this article, you'll be able to:

• ✅ Choose between blackboard, message passing, and supervisor/worker topologies
• ✅ Use shared scratchpad files safely for cross-session state
• ✅ Apply "one writer per resource" to avoid clobbering
• ✅ Design handoffs that survive context loss

---

Table of Contents

1. Context / Why This Matters
2. Pattern: Blackboard
3. Pattern: Message Passing
4. Pattern: Supervisor / Worker
5. Pattern: Shared Scratchpad Files
6. Conflict Avoidance

---

Context / Why This Matters

A single agent with a clear task is reliable; two agents editing without a protocol are a race condition with a vocabulary. The KB's orchestration baseline (agentic-workflows) already establishes orchestrator–worker separation and evaluator loops. This article goes one level deeper: the actual communication mechanisms between agents, when each fits, and the ownership rules that keep them from destroying each other's output.

The same principles apply whether the "agents" are LLM sessions, CI jobs, or shell scripts — they are all concurrent writers to shared state.

---

Implementation / Core Content

1. Pattern: Blackboard

A blackboard is a shared structured store that all agents read and selectively write. Nobody sends messages directly; agents poll the board, claim work, and post results.

            ┌────────────────────┐
   agent A ─►│                    │◄─ agent B
            │   BLACKBOARD        │
   agent C ─►│ (state + claims)   │◄─ evaluator
            └────────────────────┘

Good fits:

• Tasks decomposable into independent pieces with shared context (triage queues, research digests).
• Agents with different lifetimes that can't hold live connections to each other.

Implementation options ranked by durability:

Store	When
Git-tracked markdown files (kb/tasks.md-style)	Human-auditable, survives everything, slowest
SQLite file	Atomic claims via transactions, single host
Redis / queue-backed KV	Fast, multi-host, adds an ops dependency

A minimal git-blackboard claim protocol:

# Claim atomically-ish: create your claim file BEFORE editing the task
CLAIM=claims/task-42.$(date +%s).$$.json
echo '{"agent":"worker-3","task":"task-42","claimed_at":"2026-08-26T10:00Z"}' > $CLAIM
git add $CLAIM && git commit -m "claim task-42" && git push
# If push fails (someone else claimed first), pull --rebase, drop your claim, pick another.

Weakness: polling latency and stale reads. Mitigate with short claim TTLs — a claim older than its expires_at is void.

2. Pattern: Message Passing

Agents exchange discrete messages through channels (queues, pipes, webhooks). Each agent owns an inbox; state lives inside the agent, not on a shared board.

# Directory-based inbox (simplest durable form)
mkdir -p /srv/agent-mail/{builder,out}/{new,cur}

# Sender drops a job
cat > /srv/agent-mail/builder/new/job-$(date +%s%N).json <<EOF
{"type":"build","ref":"main","reply_to":"out/coordinator"}
EOF

# Receiver processes and moves to cur (at-least-once semantics)
for f in /srv/agent-mail/builder/new/*.json; do
  process "$f" && mv "$f" /srv/agent-mail/builder/cur/
done

Rules that make message passing safe:

1. Messages must be self-contained. Include every input needed; never reference "as discussed."
2. Acknowledge only after durably handling (move/rename/delete after success), giving at-least-once delivery → handlers must therefore be idempotent (failure-recovery-patterns).
3. Reply channels, not broadcast replies, so results don't race in the sender's inbox.

3. Pattern: Supervisor / Worker

One coordinator owns the goal, decomposes it, spawns workers, merges results. This is the orchestrator–worker separation from agentic-workflows, stated as a communication contract:

• Supervisor → worker: exactly one task description, inputs included, plus an output contract ("return a conclusion, not raw evidence").
• Worker → supervisor: one final result message. No partial streaming unless designed for it.
• Failure policy declared up front: retry count, timeout, and what the supervisor does when a worker dies (re-dispatch to a fresh worker with the same idempotent task).

Why this topology wins most of the time: workers need no knowledge of each other; the supervisor is the single serialization point, which eliminates write conflicts by construction. Cost: the supervisor is a bottleneck and a single point of confusion — keep its role narrow (dispatch + merge) and give it bounded budgets.

4. Pattern: Shared Scratchpad Files

For same-host collaboration, plain files in a scratch directory are the lowest-tech and often best channel: append-only notes, JSON status blobs, plan files. Conventions that keep them sane:

/srv/scratch/
  mission-<id>/
    plan.md          # written once by supervisor; workers treat as read-only
    status/
      worker-1.json  # each worker writes ONLY its own file
    notes/
      *.md           # append-only observations; filename = author + timestamp
    done/
      worker-1.json  # moved here on completion (atomic rename)

Key conventions:

• Append-only over edit-in-place: notes/*.md entries are added, never rewritten — history stays intact and merge conflicts become impossible.
• Status via atomic rename: write worker-1.json.tmp, then mv to worker-1.json. Readers never see half-written JSON.
• Filenames carry identity: <author>-<timestamp>.md so two agents can never target the same path accidentally.

5. Conflict Avoidance

The unifying rule: one writer per resource. Every file, table row, lock, and service instance has exactly one owner at any moment; everyone else reads.

Enforcement techniques, cheapest first:

1. Partition by name. Per-agent output paths (above). Zero coordination cost. Default choice.

2. File locks. flock around any read-modify-write section (same mechanism as scheduler-patterns):

   exec 9>/srv/scratch/mission-7/state.lock
   flock -w 30 9 || exit 75      # busy: retry later
   # ...read state, modify, write...
   flock -u 9
   

3. Claim/lease protocol on the blackboard (section 1): claim before work, lease expiry for crashed claimants.

4. Git as arbiter. Commit-claim-push; push failure = someone won the race, rebase and re-decide. Slow but fully auditable.

Anti-patterns to refuse:

• Two agents "co-editing" the same document concurrently — serialize instead (one finishes, hands off explicitly).
• Locking by convention only ("we agreed not to touch X") with no mechanism — under context loss, agreements evaporate.
• Shared mutable counters without locks — last-writer-wins silently corrupts them.

Handoff discipline: because agent contexts die frequently, every handoff message must contain (a) what was done, (b) what remains, (c) where the durable state lives. A handoff that requires asking a follow-up question has failed.

---

Practical Examples

Example 1: Parallel research fan-out with scratchpad merge

Supervisor writes plan.md listing 5 questions, spawns 5 workers each writing status/worker-N.json + notes/worker-N-findings.md. Supervisor polls done/ until 5 files exist, then synthesizes. No locks needed anywhere because every writer owns distinct filenames.

Example 2: Exclusive deploy right via flock

Only one agent may run deploys at a time:

flock -n /var/lock/fogserv-deploy.lock ./deploy.sh \
  || { echo '{"error":"deploy_lock_busy"}'; exit 75; }

Workers that get exit 75 back off and retry later rather than queueing — keeps failure simple.

---

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
Worker A's edits overwrite B's	Two writers on one resource	Partition filenames or add flock/claims
Agent redoes completed work	Status lost with context	Durable scratchpad/status files, checked before acting
Duplicate side effects (double deploy, double email)	At-least-once messaging without idempotency	Idempotency keys per task; check-before-execute
Stale claims block tasks forever	Claimant died holding lease	Lease TTL + reaper that voids expired claims
Messages reference missing context	Non-self-contained messages	Include full inputs in every message
Deadlock: A waits on B, B waits on A	Circular wait on two locks	Global lock ordering; or use single supervisor serialization

---

Next Steps / Ops Actions

• Design every multi-step workflow's failure behavior with failure-recovery-patterns.
• Keep fleet-level governance (lifecycle states, audit logs) aligned with agentic-workflows.
• For infrastructure hosting agent services, apply the ops baseline in ai-server-management.

Sources & Related Articles

External references consulted:

• https://man7.org/linux/man-pages/man1/flock.1.html
• https://en.wikipedia.org/wiki/Blackboard_system

Related knowledge-base articles:

• agentic-workflows
• failure-recovery-patterns
• ai-server-management
• scheduler-patterns

Change Log

2026-08-26

• Initial creation covering blackboard, message passing, supervisor/worker, scratchpad files, and conflict avoidance.