# Multi Agent Orchestrator

> Multi-agent orchestration built as production infrastructure: typed planning, a sandboxed tool registry, race-safe human-in-the-loop, deterministic replay, and a provably-terminating LangGraph — measured, never declared. 255 offline tests, CI-green.

- **Type:** MCP server
- **Install:** `agentstack add mcp-anbsamsam17-multi-agent-orchestrator`
- **Verified:** Pending review
- **Seller:** [anbsamsam17](https://agentstack.voostack.com/s/anbsamsam17)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [anbsamsam17](https://github.com/anbsamsam17)
- **Source:** https://github.com/anbsamsam17/multi-agent-orchestrator

## Install

```sh
agentstack add mcp-anbsamsam17-multi-agent-orchestrator
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Multi-Agent Orchestrator

> **Multi-agent orchestration built as production infrastructure, not a demo.**
> A supervisor that plans in **typed contracts**, specialists that act *only* through a
> **sandboxed tool registry**, **persistent memory** whose usefulness is A/B-measured,
> **human-in-the-loop** escalation with measured trigger quality, and **deterministic
> replay** of every execution.

[](https://github.com/anbsamsam17/multi-agent-orchestrator/actions/workflows/ci.yml)
[](#whats-inside-by-phase)
[](#)
[](LICENSE)
[](#whats-inside-by-phase)

**The differentiator is not the stack — it is provable termination + structural safety +
measured behavior + replayability.** An unbounded retry loop, an approval gate with a
race, an escalation metric gamed by escalating everything, or a replay that only "mostly"
matches would each destroy the entire signal. Those failure modes are exactly what this
codebase exists to get right — and to *prove* it got right.

> ✅ **Complete.** All seven phases implemented, **255 offline tests**, CI green — no API
> key, no Docker, no network. Every *evaluation* number below reproduces from a harness
> run (`make eval` / `make eval-ab`), provenance-stamped; the termination figures are
> pinned by the test suite (`make test`). Nothing is declared without a measurement
> behind it.

---

## What most "AI agent" repos skip — and how this one answers it

Most agent repos are a single agent in a `while`-loop with untyped tool calls, no memory,
no human gate, and no way to know whether the machinery helps at all. This one is built
around the five questions that actually matter when agents run unattended:

| The question that matters in production | How this repo answers it — with a number |
|---|---|
| **Does it terminate — *provably*?** | The graph's recursion limit is **derived** from the loop budgets, not a magic constant, and a worst-case test pins it: **96 node visits ≤ 229**. |
| **Can a specialist do anything unsafe?** | No side-effect path exists except a typed registry; sensitive tools are **fail-closed** behind a race-safe one-shot human grant at the registry chokepoint. |
| **Does the multi-agent machinery actually help?** | A **paired A/B** (multi- vs single-agent, memory on/off) with **bootstrap CI95** — a confidence interval straddling zero is reported "not significant," never a win. |
| **Do the escalation triggers work?** | Reported as **precision AND recall** with the full confusion matrix — escalating everything games recall, never escalating games precision, so neither ships alone. |
| **Can you see and *re-run* what happened?** | Every decision is a typed, costed trace span; any past run **replays** span-by-span against fake transports and **fails loudly** on the slightest divergence. |

## Results at a glance

The evaluation rows reproduce with `make eval`; the A/B rows with `make eval-ab` (golden
set `sha256:7f937d03…`, bootstrap seed `1750`, `B=10000`); the termination row with
`make test`. All key-free.

| | Measured | The honest fine print |
|---|---|---|
| **Provable termination** | worst case **96 node visits** vs a **derived** recursion limit of **229** | both numbers are pinned by tests in `test_graph.py` — the 96 by a worst-case node-visit count, the 229 as a *function* of the loop budgets, not a tuned constant |
| **Plan quality** (n=8 golden) | **0.896** | scores the committed deterministic behavior vs labeled ground truth (reproducible); a live-model score needs `ANTHROPIC_API_KEY` |
| **Tool-call correctness** | **0.875** | an unexpected *sensitive* call is a hard 0 — never partly right |
| **Escalation triggers** | precision **0.50** · recall **0.50** · F1 **0.50** (tp1 fp1 fn1 tn5) | precision AND recall, always both, with the full confusion matrix; **n=8** (small by design — the set deliberately ships an FP and an FN) |
| **A/B — multi vs single agent** | Δ +0.042 plan-quality, CI95 [−0.125, +0.250] (fixture) | **directional, not significant**; fixture mode exercises the paired-bootstrap machinery, it does **not** evidence a benefit (the live A/B does) |
| **A/B — memory on vs off** | Δ **0.000**, CI95 **[0, 0]** | an honest non-result: the scripted model ignores injected memory; a real effect needs the live A/B |
| **Deterministic replay** | full-span + prompt-fingerprint identity | a divergence (incl. a non-deterministic tool re-executing) **fails loudly**, never a plausible-but-wrong replay |
| **Safety** | sensitive tools **structurally unreachable** | a human one-shot grant is required at the registry chokepoint; the gate is race-safe and fails closed |

Every row is honest about what it measures. The evaluation figures score the repo's
**committed deterministic scenario behavior** against labeled ground truth — that is what
makes them reproducible and key-free; measuring a *live model* and running the *live*
A/Bs is `make eval-ab` with a key, whose verdicts (even "no gain") are published as-is.

## Architecture

```
                                 ┌────────────────────────────┐
                    ┌──────────► │  Reviewer (≠ producer)     │
                    │            └────────────┬───────────────┘
                    │                         │ Verdict (typed)
┌──────────┐   ┌────┴─────┐   ┌───────────────▼──────────────┐   ┌───────────────┐
│  Task    │ ─►│Supervisor│ ─►│  Specialists                  │ ─►│  Synthesis /  │
│  intake  │   │ typed    │   │  research · analysis ·        │   │  delivery     │
└──────────┘   │ Execution│   │  writing · code               │   └───────────────┘
               │ Plan     │   └────────────┬──────────────────┘
               └────┬─────┘                │  ToolCalls only
          memory    │              ┌───────▼────────┐
          retrieval │              │ Tool registry   │  allowlists · rate limits ·
          (attributed)             │ (MCP-mappable)  │  sandboxes · sensitive-flag
                    │              └───────┬────────┘
     ┌──────────────▼───┐                  │ sensitive=True → structurally gated
     │ Memory           │          ┌───────▼────────┐
     │ working (Redis)  │          │ HITL approval   │  notify / approve-action /
     │ semantic (Qdrant)│          │ gate (fail-     │  approve-plan / take-over
     └──────────────────┘          │ closed, race-   │
                                   │ safe, resumable)│
                                   └────────────────┘
     Every decision → typed TraceSpan (model id, tokens, cost) → deterministic replay
```

LangGraph state machine: `intake → plan → select → execute → review → synthesize`, with
conditional edges for bounded retries, reviewer rejections, and human escalation. Full
module map in [`docs/architecture.md`](docs/architecture.md); every load-bearing decision
is an ADR in [`docs/decisions/`](docs/decisions/).

## What the adversarial review caught

Every phase was attacked by a **read-only adversarial reviewer before merge** — a separate
opus agent whose only job is to break the diff. It did not rubber-stamp. It found real
defects that passed the happy path, each fixed with a regression test:

| Phase | What the red-team caught | The fix |
|---|---|---|
| 3 · Memory | **Blocker:** the prompt-injection fence was escapable — poisoned content containing the closing tag broke out of quarantine, and my *own* tests hid it (they used poison without the delimiter). | neutralize any forgeable delimiter in content before fencing; test the *real* attack |
| 4 · HITL | **Four** majors, all green on the happy path: a non-atomic decision (a REJECT overwritten by a concurrent APPROVE), a double-consumed grant, **no auth** on the approve endpoint, and a grant burned on tool failure (wedging the task). | atomic SETNX decision latch, a lock on consume, bearer auth, release-on-failure |
| 5 · Replay | "Deterministic replay" was **over-claimed** — the identity check compared only the span *skeleton*, and tools re-executed (a non-deterministic tool could replay differently and still pass). | strengthened to full-span + prompt-fingerprint identity; a divergence now fails loudly |
| 7 · Showcase | A README number (the 96 node visits) wasn't reproducible from any command — it was prose. | pinned it as a test, so the number can only move when the behavior does |

A junior says "it works." The point of this repo is to show *who tried to break the code,
what they found, and how it was proven fixed.*

## Quickstart

```bash
git clone https://github.com/anbsamsam17/multi-agent-orchestrator && cd multi-agent-orchestrator
make install   # pip install -e ".[dev]"
make test      # 255 offline tests — no API key, no Docker, no network
make demo      # run the whole orchestration end-to-end on a scripted offline scenario
make eval      # score the golden scenarios + write provenance (meta.json); key-free
make eval-ab   # paired A/Bs with bootstrap CI95 (live with ANTHROPIC_API_KEY, else fixture)
make serve     # FastAPI service: POST /tasks, SSE /tasks/{id}/stream, the approval queue
```

`make up` starts the memory backends (Redis + Qdrant) via Docker Compose for a live
deployment; the test suite never needs them (fakeredis + in-memory Qdrant + fake LLM
transports). Set `ORCHESTRATOR_API_TOKEN` to require bearer auth on the API.

## What's inside (by phase)

| Phase | Delivers | Tests (cumulative) |
|---|---|---|
| 1 · Contracts + tool registry | frozen Pydantic contracts; the registry as the **only** side-effect surface (allowlists, rate limits, caller-bounded timeouts, fail-closed sensitive gate, filesystem jail, MCP-mappable) | 65 |
| 2 · Supervisor + graph | typed `ExecutionPlan` via structured output; registry-only specialists with corrective retries; reviewer ≠ producer; LangGraph with a **derived, proven recursion limit** | 93 |
| 3 · Memory | working (Redis) + semantic (Qdrant) memory; typed extraction with secret-scrubbing; **attributed, poison-safe** injection; GDPR delete | 148 |
| 4 · Human-in-the-loop + API | one-shot grants at the registry chokepoint; **race-safe, fail-closed** approval gate; FastAPI (submit, SSE, approvals, bearer auth); escalate→approve→resume | 189 |
| 5 · Observability + replay | record at the LLM seam; **deterministic replay** (prompt-fingerprint + full-span check); pinned-price cost accounting; redaction-before-persistence | 214 |
| 6 · Evaluation | golden scenarios; fixture-tested scorers; escalation P/R + confusion matrix; **paired A/Bs with bootstrap CI95**; provenance-stamped | 253 |

Counts are cumulative and measured on the dev box (Windows, where one symlink-escape test
skips). Phase 7 (this showcase) adds one test — the pinned worst-case node-visit count —
so the full suite is **255 tests**: 254 pass locally, and the symlink test runs and passes
in CI.

## Engineering discipline

- **Typed everything.** Plans, tool calls, verdicts, memories, spans — Pydantic models at
  every module seam; no free-text protocols between agents.
- **Measured, never declared.** Evaluation metrics exist only if reproducible from
  `make eval` / `make eval-ab`, provenance-stamped (git SHA, scenario-set SHA-256, pinned
  model ids and seeds); structural guarantees like the termination bound are pinned by
  `make test`. Golden scenarios are hook-protected against agent mutation.
- **Honest reporting.** CIs that straddle zero are "directional, not significant"; n is
  always stated; unfavorable verdicts and disclosed caveats (e.g. the fixture A/B's
  circularity) are published as-is.
- **Fail loud, not silent.** Every guarantee — termination, the approval gate, replay, the
  scorers — is designed to *raise* on a violation rather than produce a
  plausible-but-wrong result.
- **ADRs** capture the seven load-bearing decisions in [`docs/decisions/`](docs/decisions/).

## Built by a governed multi-agent system

The committed [`.claude/`](.claude/) directory is the agentic engineering setup that builds
this repo — **10 specialized subagents** (an architect, six domain engineers, an
evaluation scientist, a read-only red-teamer, a docs historian), **5 orchestration
commands**, and safety hooks that guard the ground truth against agent `Edit`/`Write`
mutation (a PreToolUse block — best-effort, fail-open). A multi-agent system, **built by a
governed multi-agent system, deliberately.** The recursion is the point: the same rigor the
product enforces at runtime is applied to its own development, and every phase was
adversarially red-teamed before merge.

## Part of a three-repo AI engineering portfolio

Each repo carries the same discipline — *measured, never declared; bounded loops;
reproducible from a harness*:

- **[`hybrid-rag-pipeline`](https://github.com/anbsamsam17/hybrid-rag-pipeline)** — hybrid
  dense+sparse retrieval and the **evaluation science** (nDCG/recall@k, verified citations,
  bootstrap CI95).
- **`multi-agent-orchestrator`** *(this repo)* — agents as production infrastructure:
  provable termination, structural safety, measured behavior, replay.
- **Eval dataset generator** *(in progress)* — mines production traces, like the ones this
  repo emits, into an auto-labeled evaluation set (closing the improvement loop).

The through-line: *I build (RAG), I orchestrate (agents), I industrialize the improvement
loop (eval) — and in all three, every published number is reproducible from a harness.*

## License

[MIT](LICENSE)

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [anbsamsam17](https://github.com/anbsamsam17)
- **Source:** [anbsamsam17/multi-agent-orchestrator](https://github.com/anbsamsam17/multi-agent-orchestrator)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: flagged — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-anbsamsam17-multi-agent-orchestrator
- Seller: https://agentstack.voostack.com/s/anbsamsam17
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.