Multi Agent Orchestrator

Multi-Agent Orchestrator

> Multi-agent orchestration built as production infrastructure, not a demo. > A supervisor that plans in typed contracts, specialists that act only through a > sandboxed tool registry, persistent memory whose usefulness is A/B-measured, > human-in-the-loop escalation with measured trigger quality, and deterministic > replay of every execution.

[](https://github.com/anbsamsam17/multi-agent-orchestrator/actions/workflows/ci.yml) [](#whats-inside-by-phase) [](#) [](LICENSE) [](#whats-inside-by-phase)

The differentiator is not the stack — it is provable termination + structural safety + measured behavior + replayability. An unbounded retry loop, an approval gate with a race, an escalation metric gamed by escalating everything, or a replay that only "mostly" matches would each destroy the entire signal. Those failure modes are exactly what this codebase exists to get right — and to prove it got right.

> ✅ Complete. All seven phases implemented, 255 offline tests, CI green — no API > key, no Docker, no network. Every evaluation number below reproduces from a harness > run (make eval / make eval-ab), provenance-stamped; the termination figures are > pinned by the test suite (make test). Nothing is declared without a measurement > behind it.

---

What most "AI agent" repos skip — and how this one answers it

Most agent repos are a single agent in a while-loop with untyped tool calls, no memory, no human gate, and no way to know whether the machinery helps at all. This one is built around the five questions that actually matter when agents run unattended:

| The question that matters in production | How this repo answers it — with a number | |---|---| | **Does it terminate — provably? | The graph's recursion limit is derived from the loop budgets, not a magic constant, and a worst-case test pins it: 96 node visits ≤ 229. | | Can a specialist do anything unsafe? | No side-effect path exists except a typed registry; sensitive tools are fail-closed behind a race-safe one-shot human grant at the registry chokepoint. | | Does the multi-agent machinery actually help? | A paired A/B (multi- vs single-agent, memory on/off) with bootstrap CI95 — a confidence interval straddling zero is reported "not significant," never a win. | | Do the escalation triggers work? | Reported as precision AND recall with the full confusion matrix — escalating everything games recall, never escalating games precision, so neither ships alone. | | Can you see and re-run what happened? | Every decision is a typed, costed trace span; any past run replays span-by-span against fake transports and fails loudly** on the slightest divergence. |

Results at a glance

The evaluation rows reproduce with make eval; the A/B rows with make eval-ab (golden set sha256:7f937d03…, bootstrap seed 1750, B=10000); the termination row with make test. All key-free.

| | Measured | The honest fine print | |---|---|---| | Provable termination | worst case 96 node visits vs a derived recursion limit of 229 | both numbers are pinned by tests in test_graph.py — the 96 by a worst-case node-visit count, the 229 as a function of the loop budgets, not a tuned constant | | Plan quality (n=8 golden) | 0.896 | scores the committed deterministic behavior vs labeled ground truth (reproducible); a live-model score needs ANTHROPIC_API_KEY | | Tool-call correctness | 0.875 | an unexpected sensitive call is a hard 0 — never partly right | | Escalation triggers | precision 0.50 · recall 0.50 · F1 0.50 (tp1 fp1 fn1 tn5) | precision AND recall, always both, with the full confusion matrix; n=8 (small by design — the set deliberately ships an FP and an FN) | | A/B — multi vs single agent | Δ +0.042 plan-quality, CI95 [−0.125, +0.250] (fixture) | directional, not significant; fixture mode exercises the paired-bootstrap machinery, it does not evidence a benefit (the live A/B does) | | A/B — memory on vs off | Δ 0.000, CI95 [0, 0] | an honest non-result: the scripted model ignores injected memory; a real effect needs the live A/B | | Deterministic replay | full-span + prompt-fingerprint identity | a divergence (incl. a non-deterministic tool re-executing) fails loudly, never a plausible-but-wrong replay | | Safety | sensitive tools structurally unreachable | a human one-shot grant is required at the registry chokepoint; the gate is race-safe and fails closed |

Every row is honest about what it measures. The evaluation figures score the repo's committed deterministic scenario behavior against labeled ground truth — that is what makes them reproducible and key-free; measuring a live model and running the live A/Bs is make eval-ab with a key, whose verdicts (even "no gain") are published as-is.

Architecture

                                 ┌────────────────────────────┐
                    ┌──────────► │  Reviewer (≠ producer)     │
                    │            └────────────┬───────────────┘
                    │                         │ Verdict (typed)
┌──────────┐   ┌────┴─────┐   ┌───────────────▼──────────────┐   ┌───────────────┐
│  Task    │ ─►│Supervisor│ ─►│  Specialists                  │ ─►│  Synthesis /  │
│  intake  │   │ typed    │   │  research · analysis ·        │   │  delivery     │
└──────────┘   │ Execution│   │  writing · code               │   └───────────────┘
               │ Plan     │   └────────────┬──────────────────┘
               └────┬─────┘                │  ToolCalls only
          memory    │              ┌───────▼────────┐
          retrieval │              │ Tool registry   │  allowlists · rate limits ·
          (attributed)             │ (MCP-mappable)  │  sandboxes · sensitive-flag
                    │              └───────┬────────┘
     ┌──────────────▼───┐                  │ sensitive=True → structurally gated
     │ Memory           │          ┌───────▼────────┐
     │ working (Redis)  │          │ HITL approval   │  notify / approve-action /
     │ semantic (Qdrant)│          │ gate (fail-     │  approve-plan / take-over
     └──────────────────┘          │ closed, race-   │
                                   │ safe, resumable)│
                                   └────────────────┘
     Every decision → typed TraceSpan (model id, tokens, cost) → deterministic replay

LangGraph state machine: intake → plan → select → execute → review → synthesize, with conditional edges for bounded retries, reviewer rejections, and human escalation. Full module map in [docs/architecture.md](docs/architecture.md); every load-bearing decision is an ADR in [docs/decisions/](docs/decisions/).

What the adversarial review caught

Every phase was attacked by a read-only adversarial reviewer before merge — a separate opus agent whose only job is to break the diff. It did not rubber-stamp. It found real defects that passed the happy path, each fixed with a regression test:

| Phase | What the red-team caught | The fix | |---|---|---| | 3 · Memory | Blocker: the prompt-injection fence was escapable — poisoned content containing the closing tag broke out of quarantine, and my own tests hid it (they used poison without the delimiter). | neutralize any forgeable delimiter in content before fencing; test the real attack | | 4 · HITL | Four majors, all green on the happy path: a non-atomic decision (a REJECT overwritten by a concurrent APPROVE), a double-consumed grant, no auth on the approve endpoint, and a grant burned on tool failure (wedging the task). | atomic SETNX decision latch, a lock on consume, bearer auth, release-on-failure | | 5 · Replay | "Deterministic replay" was over-claimed — the identity check compared only the span skeleton, and tools re-executed (a non-deterministic tool could replay differently and still pass). | strengthened to full-span + prompt-fingerprint identity; a divergence now fails loudly | | 7 · Showcase | A README number (the 96 node visits) wasn't reproducible from any command — it was prose. | pinned it as a test, so the number can only move when the behavior does |

A junior says "it works." The point of this repo is to show who tried to break the code, what they found, and how it was proven fixed.

Quickstart

git clone https://github.com/anbsamsam17/multi-agent-orchestrator && cd multi-agent-orchestrator
make install   # pip install -e ".[dev]"
make test      # 255 offline tests — no API key, no Docker, no network
make demo      # run the whole orchestration end-to-end on a scripted offline scenario
make eval      # score the golden scenarios + write provenance (meta.json); key-free
make eval-ab   # paired A/Bs with bootstrap CI95 (live with ANTHROPIC_API_KEY, else fixture)
make serve     # FastAPI service: POST /tasks, SSE /tasks/{id}/stream, the approval queue

make up starts the memory backends (Redis + Qdrant) via Docker Compose for a live deployment; the test suite never needs them (fakeredis + in-memory Qdrant + fake LLM transports). Set ORCHESTRATOR_API_TOKEN to require bearer auth on the API.

What's inside (by phase)

| Phase | Delivers | Tests (cumulative) | |---|---|---| | 1 · Contracts + tool registry | frozen Pydantic contracts; the registry as the only side-effect surface (allowlists, rate limits, caller-bounded timeouts, fail-closed sensitive gate, filesystem jail, MCP-mappable) | 65 | | 2 · Supervisor + graph | typed ExecutionPlan via structured output; registry-only specialists with corrective retries; reviewer ≠ producer; LangGraph with a derived, proven recursion limit | 93 | | 3 · Memory | working (Redis) + semantic (Qdrant) memory; typed extraction with secret-scrubbing; attributed, poison-safe injection; GDPR delete | 148 | | 4 · Human-in-the-loop + API | one-shot grants at the registry chokepoint; race-safe, fail-closed approval gate; FastAPI (submit, SSE, approvals, bearer auth); escalate→approve→resume | 189 | | 5 · Observability + replay | record at the LLM seam; deterministic replay (prompt-fingerprint + full-span check); pinned-price cost accounting; redaction-before-persistence | 214 | | 6 · Evaluation | golden scenarios; fixture-tested scorers; escalation P/R + confusion matrix; paired A/Bs with bootstrap CI95; provenance-stamped | 253 |

Counts are cumulative and measured on the dev box (Windows, where one symlink-escape test skips). Phase 7 (this showcase) adds one test — the pinned worst-case node-visit count — so the full suite is 255 tests: 254 pass locally, and the symlink test runs and passes in CI.

Engineering discipline

• Typed everything. Plans, tool calls, verdicts, memories, spans — Pydantic models at

every module seam; no free-text protocols between agents.

• Measured, never declared. Evaluation metrics exist only if reproducible from

make eval / make eval-ab, provenance-stamped (git SHA, scenario-set SHA-256, pinned model ids and seeds); structural guarantees like the termination bound are pinned by make test. Golden scenarios are hook-protected against agent mutation.

• Honest reporting. CIs that straddle zero are "directional, not significant"; n is

always stated; unfavorable verdicts and disclosed caveats (e.g. the fixture A/B's circularity) are published as-is.

• Fail loud, not silent. Every guarantee — termination, the approval gate, replay, the

scorers — is designed to raise on a violation rather than produce a plausible-but-wrong result.

• ADRs capture the seven load-bearing decisions in [docs/decisions/](docs/decisions/).

Built by a governed multi-agent system

The committed [.claude/](.claude/) directory is the agentic engineering setup that builds this repo — 10 specialized subagents (an architect, six domain engineers, an evaluation scientist, a read-only red-teamer, a docs historian), 5 orchestration commands, and safety hooks that guard the ground truth against agent Edit/Write mutation (a PreToolUse block — best-effort, fail-open). A multi-agent system, built by a governed multi-agent system, deliberately. The recursion is the point: the same rigor the product enforces at runtime is applied to its own development, and every phase was adversarially red-teamed before merge.

Part of a three-repo AI engineering portfolio

Each repo carries the same discipline — measured, never declared; bounded loops; reproducible from a harness:

• hybrid-rag-pipeline — hybrid

dense+sparse retrieval and the evaluation science (nDCG/recall@k, verified citations, bootstrap CI95).

• multi-agent-orchestrator (this repo) — agents as production infrastructure:

provable termination, structural safety, measured behavior, replay.

• Eval dataset generator (in progress) — mines production traces, like the ones this

repo emits, into an auto-labeled evaluation set (closing the improvement loop).

The through-line: I build (RAG), I orchestrate (agents), I industrialize the improvement loop (eval) — and in all three, every published number is reproducible from a harness.

License

[MIT](LICENSE)

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: anbsamsam17
• Source: anbsamsam17/multi-agent-orchestrator
• License: MIT

Install and usage instructions live in the source repository linked above.