# Agentic Tool Optimization > Local-first multi-runtime AI cockpit (CLI + desktop + MCP). War-rooms across Claude/Codex/Gemini, replay, regression detection, receipts in SQLite. MIT. - **Type:** MCP server - **Install:** `agentstack add mcp-willnigri-agentic-tool-optimization` - **Verified:** Pending review - **Seller:** [WillNigri](https://agentstack.voostack.com/s/willnigri) - **Installs:** 0 - **Category:** [Databases](https://agentstack.voostack.com/c/databases) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [WillNigri](https://github.com/WillNigri) - **Source:** https://github.com/WillNigri/Agentic-Tool-Optimization - **Website:** https://agentictool.ai ## Install ```sh agentstack add mcp-willnigri-agentic-tool-optimization ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # ATO β€” The cockpit where every AI follows your rules > **Your AIs already have hands. ATO is the seat you fly them from.** > Set the rules. Watch every tool call. Kill the runaway. Compare what each one actually did to your code. Multi-runtime. Local-first. MIT. ``` $ ato review --reviewer claude --reviewer codex --reviewer gemini --against main # illustrative output β€” your numbers (cost, duration, tool calls) will vary by runtime, model, and diff size review session 7F3A1B6E Β· 3 reviewers Β· your rules: read-only, repo-scoped, killable CLAUDE πŸ”§ verified via 4 tool calls flagged 2 issues β€” XSS in src/render.ts:142, auth bypass in api/login.ts:87 read_file Γ—3 Β· grep Γ—1 Β· 6.4s Β· $0.024 Β· 0 files written (read-only mode) CODEX πŸ”§ verified via 6 tool calls flagged 3 issues incl. SQL injection in db/queries.ts:64 ← caught one claude missed by grep'ing for raw string concatenation read_file Γ—4 Β· grep Γ—2 Β· 7.1s Β· $0.004 Β· 0 files written GEMINI ⚠️ prompt-only β€” didn't open the code flagged 2 issues from the diff alone (XSSΓ—2, CSRF) 5.9s Β· $0.001 Β· 0 tool calls Β· downweight in synthesis closer: 4 unique findings, 1 disputed. tags: security, sql-injection. cost: $0.029 total. every tool call, every byte read, archived to ~/.ato/local.db paste-ready transcript at .ato/reviews/7F3A1B6E.md ``` Three AIs all *could* have walked the code. **You see which ones did, and which one just replied.** That's the cockpit. --- ## In 30 seconds ```bash # macOS brew install willnigri/ato/ato && ato demo-war-room # Linux curl -fsSL https://agentictool.ai/install.sh | sh && ato demo-war-room # Desktop app (macOS Β· Windows Β· Linux) ``` **[Download β†’](https://github.com/WillNigri/Agentic-Tool-Optimization/releases/latest)** (Tauri 2.x Β· MIT) `demo-war-room` runs without API keys β€” picks your configured runtimes, falls back to Ollama. First receipt in 30 seconds. --- ## What's new on main (unreleased) - **The open-box router.** `ato bench run` (verifiable sandboxed code-exec grading of pinned LiveCodeBench slices, contamination-clean headlines, reproducibility hashes) + `ato route explain` (transparent advisory over YOUR local receipts) + `--cascade` coordination benchmarks. See [The open-box router](#the-open-box-router--bench-with-your-keys-route-with-receipts). ## What's new in v2.18.0 - **Real-time team participation β€” now bidirectional.** Share any war-room, session, or chat into a team workspace and teammates see a live rich card (title, summary, tags, runtime badges, πŸ‘₯ TEAM badge, member avatars) the moment it lands. Append a turn from the UI or CLI and every member's machine updates instantly β€” no refresh, both directions (HTTP append + Postgres NOTIFY + WebSocket push). Previously team shares were read-only snapshots. CLI: `ato war-rooms share --team `, `ato sessions share`, `ato chats share`; append a turn live with `ato war-rooms append-event --team --kind --json `. Requires Team tier + `ato login` on the prod app binary. Close summaries now prefer your **Claude Code subscription** (no API key billing): `ato war-rooms close --coordinator claude`. - **Browse your team workspaces from any browser.** Sign in to your cloud account on the web β€” every shared session, war-room, chat, loop, and mission renders with the same fidelity as the desktop. Mobile-responsive. - **Pair your browser to your desktop.** v2.17 tether: X25519 DH + AEAD, fingerprint-verified pairing. Your laptop becomes a secure oracle for the page you're looking at β€” no plaintext through the cloud relay. - **Create + manage teams from the web.** New "+ New team", invite by email, role changes, danger-zone delete. Account page with profile + plan + sign-out. (LLM keys, runtimes, and skills still live in the desktop where the OS keychain is.) - **`ato war-rooms sweep`** β€” auto-closes idle war-rooms with a coordinator-summary, single-JSON envelope output, clap-layer validators. Wire to cron and one-shot R1 reviews self-close. - **`ato subagent log`** β€” Claude Code's Agent (Task) tool dispatches now show up in execution_logs alongside outer-session work. Canonical `auth_mode` / `billing_surface` vocab so analytics group cleanly. Git commit SHA captured per receipt. - **Web sign-in + Onboarding redesigned.** Minimal centered sign-in card. Onboarding walks users through install / `.env` / wrap-the-client with explanations of *why* each step matters. - **Deprecated Google models auto-filtered** from the chat picker (no more `gemini-2.0-flash-001` 404s). - See [CHANGELOG.md](CHANGELOG.md) for the full list. --- ## Why ATO Claude Code, Codex, Gemini CLI already come with tools. They can grep your repo, read your files, run your tests, edit code. **That's the default now** β€” chat is a commodity and so are hands. What you can't do without a cockpit: - **Set rules across every runtime at once.** Read-only here. No network there. Repo-scoped everywhere. ATO speaks the permission flag each runtime understands (`--allowedTools` for Claude, the equivalent for Codex / Gemini) so one config governs them all. - **Watch every tool call.** Per-dispatch receipts: prompt, runtime, model, every `read_file` / `grep` / `git_log` with arguments, every byte returned, files written via mtime-snapshot diff. Verified-via-N-tool-calls vs prompt-only badges per seat so you know which findings were checked against the code and which are vibes. - **Compare what each AI actually did.** Side-by-side replay across runtimes. File attribution per dispatch. "Claude touched 3 files, Codex touched 5 β€” these two diverge here." - **Kill a runaway.** Live runs registry, one-click kill, across every runtime. - **Bring API models in as full teammates.** Claude Code, Codex, Gemini CLI, Hermes, and OpenClaw already have hands β€” they ship with their own coding tool layer. The runtimes that don't are API providers without a first-party coding agent β€” Grok, MiniMax, DeepSeek, Qwen, GLM, Yi, Kimi β€” they hit a prompt-in, text-out endpoint with no built-in `read_file` / `grep` / `bash`. ATO wraps them with the same tool loop the CLI runtimes use, so a Grok or DeepSeek call can review code alongside Claude Code under identical rules + receipts. One war room, every seat with hands. All of it local. AES-256 at rest, OS-keychain master key with a rotation ledger (v2.7.14+). No cloud round-trip unless you opt in. --- ## The open-box router β€” bench with your keys, route with receipts Closed routers (Sakana's Fugu Ultra and friends) are a black box: tokens in, a model choice out, numbers you can't reproduce on infrastructure you can't see. ATO does the opposite. You benchmark the models **you have keys for** on a **pinned public dataset**, every score ships with the hashes to re-run it, and the router shows every number behind its pick. ```bash # 1. Pin a LiveCodeBench slice yourself (we never vendor the corpus), bench your models ato bench run --dataset-file lcb-pin.jsonl \ --models "gemini-3-flash-preview,gpt-5,claude-sonnet-4-6,claude-fable-5" \ --provider-default-sampling --lcb-revision 0fe84c39… # 2. Ask the router. It reads YOUR local receipts β€” never a hosted leaderboard. ato route explain "solve a hard competitive programming problem" ``` ``` DECISION: claude-sonnet-4-6 (anthropic) ⚠ gemini-3-flash-preview, gpt-5, claude-fable-5 overlap the front-runner's 95% CI (accuracy cannot honestly separate them) β€” chosen by cost, then latency. claude-fable-5 85.7% CI [70.6%, 93.7%] (30/35) $0.152/task 25s/task gpt-5 79.6% CI [66.4%, 88.5%] (39/49, contamination-clean) $0.111/task 93s/task gemini-3-flash-preview 73.5% CI [59.7%, 83.8%] (36/49, contamination-clean) $0.040/task 50s/task claude-sonnet-4-6 69.4% CI [55.5%, 80.5%] (34/49) $0.028/task 19s/task dataset_hash: e2bcdcc9… harness_hash: 40dd5ac0… env_hash: 66c5d96c… re-run the evidence yourself: ato bench run --dataset-file --models … ``` Real run, real money (~$19 across two battles), 49 real LiveCodeBench problems, contamination checked per-model against vendor-stated cutoffs (post-cutoff for the rows tagged clean; the Claude rows' tasks predate their cutoffs and the table refuses to hide it; fable's leg holds 35 valid results β€” truncated by a billing cap, disclosed in the receipt). The intervals overlap, so the router says so out loud and picks on price instead of pretending it can rank them β€” while the paired McNemar test on shared tasks (fable never lost a discordant pair; p=0.031 over sonnet) already separates the leader route v2 will learn to declare. **That refusal to overclaim is the product.** Every scorecard records: pass-rate Β± Wilson CI, a contamination-clean headline (task release dates vs vendor-stated training cutoffs, 32 models cited in `ato bench cutoffs`), cost/latency/tokens, sandboxed execution (network-off), and the dataset/harness/env hashes that make "62% on LiveCodeBench" mean something. Two numbers are only comparable when their hashes match β€” the router enforces that, and discloses what it skipped. It benches **coordinations** too: `--cascade "claude-sonnet-4-6->gemini-3-flash-preview"` runs a cheap-model-first recipe that escalates only when the answer fails the problem's public tests β€” a verifiable signal, no judge model β€” and scores it as a virtual model in the same table. (Our first cascade run lost to plain sonnet on cost-effectiveness, and the receipts show exactly which three problems fooled the public-test signal. Honest negatives are still receipts.) Advisory is free and local, always: `route explain` never dispatches, never reads a hosted leaderboard, and with zero local evidence it tells you to run `ato bench` β€” not to trust ours. Acting on routes automatically is ATO Pro: `route apply` (ships today) dispatches the advisory's pick with the provenance stamped on the result; continuous `route watch` is the registered next gate. --- ## What you do in the cockpit ### 1. Multi-LLM code review β€” with shared findings `ato review` dispatches your diff + full file context + git log to N reviewers in a shared session. Each one can call `read_file`, `grep`, `git_log` to walk the live repo under your rules. The second reviewer sees the first's findings β€” real consensus, not parallel monologues. ```bash ato review --reviewer claude --reviewer codex --reviewer @security-specialist \ --against main --consensus ``` Untrusted-file-content guard: tool returns are wrapped in `` so a file under review can't hijack the reviewer with prompt injection. ### 2. War-rooms β€” N LLMs, one shared room Fire one prompt at N LLMs sharing a `--war-room-id`. A closer summarizes every reply with title, tags, category, and who agreed. ```bash WR=$(uuidgen) ato dispatch claude "should we ship the Postgres migration before the freeze?" --war-room-id $WR ato dispatch codex "should we ship the Postgres migration before the freeze?" --war-room-id $WR ato dispatch gemini "should we ship the Postgres migration before the freeze?" --war-room-id $WR ato war-rooms close $WR --human ``` The room becomes one card in the GUI; click any seat to see its tool calls. ### 3. Sticky sessions with `@runtime` bridge Multi-turn chats that span runtimes. `@gemini what do you think?` mid-thread bridges Gemini in; the bridge loops until `[CONSENSUS]` or the round cap. ```bash ato sessions new --runtime claude --title "auth-rewrite" ato dispatch claude "..." --session --tag-bridge --max-rounds 3 ``` ### 4. Receipts that the pilot can actually read Local SQLite at `~/.ato/local.db`. Every dispatch persists prompt, runtime, model, tokens, cost, duration, **files touched (mtime diff)**, every tool call with arguments, session id. The GUI renders them paste-ready. ```bash ato dispatches list --human ato files-touched # cross-run lineage per file ato traces show # full transcript + tool-call audit ``` ### 5. MCP server β€” 65 tools, your coding agent drives ATO Claude Code, Codex, Cursor β€” any MCP-aware runtime can drive ATO directly: `run_agent`, `list_agents`, `get_context_usage`, `get_usage_stats`, `get_mcp_status`, skill management, runtime health, agent logs, cache. The runtime calls into the cockpit instead of you opening tabs. ```bash npx ato-mcp # add to ~/.claude/settings.json mcpServers ``` ### And the rest of the cockpit - **Replay any past trace** against a different runtime/model β€” side-by-side diff with duration + cost delta. - **Compare workbench** β€” diff any two cloud traces of the same agent (duration, cost, files, ok-status). - **Live runs registry + kill** β€” every in-flight dispatch with agent, runtime, workspace, elapsed; one-click kill. - **Cross-runtime regression detection** β€” flags *"success rate dropped 17pp after the model swap"* by joining the config-change ledger with trace windows. - **Cost optimizer** β€” `ato optimize recommend --human`. Concrete swaps with quality guards (β‰₯30% cheaper, ok-rate within 10pp, eval-score within 5pp). - **Loop Composer** (v2.14) β€” visual + CLI graph editor for **LLM workflow loops**. First-class node types: dispatch, methodology run, diagnose, apply (Goodhart-defended), review, war-room, score. Persisted SQLite, scriptable via `ato loop run ` for headless boxes and MCP agents. *"Weekly: run methodology X, diagnose failures, apply the patch, re-run on the holdout, alert on regression"* is one loop. Reframed from Automations β€” same node-graph muscle, but the palette is LLM-aware. Reads the moment as Peter Steinberger put it: design loops that prompt your agents, not prompts in isolation. - **Embedded terminal** β€” xterm.js + portable-pty, scoped to active project, persistent across navigation. - **Dynamic prompts** β€” resolvers: env / file / SQL / MCP / computed JS. - **Agent wizard + skills marketplace** β€” writes the right file per runtime (`~/.claude/agents/`, `~/.codex/agents/`, `/.gemini/agents/`, `~/.openclaw/agents/`, `~/.hermes/agents/`). - **SSH remote runtimes** β€” `ato runtimes add-remote` for laptopβ†’server dispatches (v2.3.32). - **Eval-score ratchet** β€” `ato ratchet check` for CI gates (v2.6.x). - **External agent deploys** β€” bundle generators for Cloudflare Worker, Vercel Edge, Docker, Node + embed widget; customer's API key. --- ## After ~20 real dispatches: ATO recommends switches ```bash ato optimize recommend --human ``` ``` Cost Recommendations (based on YOUR data) 1. Switch CLAUDE β†’ GOOGLE on @code-writer Savings: 85% per round ($0.0245 β†’ $0.0037) Evidence: head-to-head rounds from your local trace database Quality guard: ok-rate within 4pp Β· eval-score within 2pp Β· PASS ``` Computed from rounds where both runtimes answered the same prompt and a judge scored both. Not vibes. **Receipts.** This is **chapter 2** β€” what happens once the cockpit has data on YOUR work. The cockpit is the daily-use loop; cost recs are the payoff. --- ## Free vs Pro **The principle**: you can run every primitive yourself for free. We charge for the codified automation we package on top. Same model as GitLab, Sentry, Supabase. Full mapping in [`docs/tiers.md`](./docs/tiers.md). | | Free | Pro ($29/mo) | Team ($49/mo) | |---|:---:|:---:|:---:| | `ato review` with `read_file` / `grep` / `git_log` | βœ“ | βœ“ | βœ“ | | War-rooms Β· sessions Β· cross-runtime bridge | βœ“ | βœ“ | βœ“ | | Tool-call passthrough + permission rules | βœ“ | βœ“ | βœ“ | | Replay Β· file attribution Β· live runs Β· kill | βœ“ | βœ“ | βœ“ | | Receipts (local SQLite, AES-256 at rest) | βœ“ | βœ“ | βœ“ | | MCP server (65 tools incl. 12 methodology) | βœ“ | βœ“ | βœ“ | | Cost optimizer (`optimize recommend`) | βœ“ | βœ“ | βœ“ | | Tauri desktop Β· embedded terminal Β· skills marketplace | βœ“ | βœ“ | βœ“ | | **Methodology runner** (`create / run / adopt / score / runs / margin / calibrate`) | βœ“ | βœ“ | βœ“ | | Methodology Insights panel (p … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source β€” we do not rehost the code. - **Author:** [WillNigri](https://github.com/WillNigri) - **Source:** [WillNigri/Agentic-Tool-Optimization](https://github.com/WillNigri/Agentic-Tool-Optimization) - **License:** MIT - **Homepage:** https://agentictool.ai Install and usage instructions live in the source repository linked above. ## Pricing - **Free** β€” Free ## Security capabilities Automated source analysis of v0.1.0 β€” what this tool can access: - **Network access:** yes - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source β€” more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** β€” security scan: flagged β€” Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-willnigri-agentic-tool-optimization - Seller: https://agentstack.voostack.com/s/willnigri - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack β€” the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.