# Shadow > Behavior contracts for AI agents - **Type:** MCP server - **Install:** `agentstack add mcp-manav8498-shadow` - **Verified:** Pending review - **Seller:** [manav8498](https://agentstack.voostack.com/s/manav8498) - **Installs:** 0 - **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [manav8498](https://github.com/manav8498) - **Source:** https://github.com/manav8498/Shadow - **Website:** https://manav8498.github.io/Shadow/ ## Install ```sh agentstack add mcp-manav8498-shadow ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Shadow [](https://github.com/manav8498/Shadow/actions/workflows/ci.yml) [](#license) [](SPEC.md) [](CHANGELOG.md) [](rust-toolchain.toml) [](python/pyproject.toml) **Find the exact change that broke your AI agent.** A teammate edits a system prompt or swaps a model. Code review looks fine. Unit tests pass. You merge. A week later a customer reports the refund agent stopped asking for confirmation. Nobody saw it coming because the code looked harmless. Shadow catches that bug class on the PR. `shadow diagnose-pr` answers, in one PR comment: 1. **Did agent behavior change?** Yes / no, with severity. 2. **How many traces are affected?** Out of N representative production-like cases. 3. **Which exact candidate change caused it?** Prompt edit, model swap, tool-schema rename, config delta — named with a `file:line` citation. 4. **With what confidence?** ATE + 95% bootstrap CI + E-value when run with `--backend live`; deterministic delta-attribution otherwise. 5. **What fix should `verify-fix` confirm before merge?** ```bash shadow diagnose-pr \ --traces prod-traces/ \ --candidate-traces candidate-traces/ \ --baseline-config baseline.yaml \ --candidate-config candidate.yaml \ --policy shadow-policy.yaml \ --pr-comment comment.md ``` → Try the [60-second runnable demo](examples/refund-causal-diagnosis/), or read [`docs/features/causal-pr-diagnosis.md`](docs/features/causal-pr-diagnosis.md) for the full flow. See a real Shadow PR comment ↗  ·  launch video (84 s)  ·  MP4 / WebM ## Why ordinary CI can't catch this Unit tests assert what code returns, not how an agent *behaves*. The refund agent that drops "always confirm" is still parseable Python. The model swap from GPT-4o to a cheaper one still returns valid responses. The tool schema renamed `order_id` → `id` still typechecks both sides. Every guardrail your normal CI gives you is irrelevant — the regression is in the agent's decisions, not the code. **Shadow turns "how should this agent behave?" into a YAML contract** — your CI tests every PR against it; your runtime enforces it against every tool call. Same rule, both places. When the contract breaks, Shadow names the line in the PR that broke it. ## What Shadow does, in one screen Given a baseline `.agentlog` and a candidate change, Shadow answers three questions on the PR: 1. **What behavior changed?** A nine-axis diff scores response meaning, tool calls, refusals, length, latency, cost, output format, and more — with a plain-English summary on top. 2. **Why did it change?** If the PR touched multiple things at once, regression attribution names the specific change that most likely explains each regression. 3. **Is it safe to merge?** A YAML policy declares rules the agent must follow (tool ordering, output shape, forbidden outputs). The same policy enforces at runtime. The report lands in the PR comment. No dashboard, no separate login, no trace upload. Traces stay on your disk. ## Install **Step 1.** Check you have Python 3.11 or newer: ```bash python3 --version # 3.11.x or higher ``` **Step 2.** Install Shadow from PyPI: ```bash pip install shadow-diff ``` That's it. `shadow --help` should now work, and `shadow quickstart` runs the demo. No clone, no Rust toolchain, no separate setup step — pre-built wheels ship for Linux x86_64, macOS arm64 (Apple Silicon), and Windows x86_64. **On other platforms** (Intel Mac, ARM Linux, older glibc, Alpine, FreeBSD), pip falls back to the source distribution and builds the Rust core locally. Install Rust first: ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh pip install shadow-diff ``` For byte-identical installs (CI repro, enterprise audit), see [docs/quickstart/locked-install.md](docs/quickstart/locked-install.md). ### Optional extras Shadow's core install is lean. Most users want one of: ```bash pip install 'shadow-diff[anthropic]' # if your agent uses Claude pip install 'shadow-diff[openai]' # if your agent uses GPT pip install 'shadow-diff[embeddings]' # better paraphrase-robust diff ``` All optional extras (framework adapters, dashboard, signing, OTel, …) | Extra | Pulls in | Use case | |---|---|---| | `shadow-diff[anthropic]` | `anthropic` | Live Anthropic client wrapper for `shadow record` | | `shadow-diff[openai]` | `openai` | Live OpenAI client wrapper for `shadow record` | | `shadow-diff[embeddings]` | `sentence-transformers` | Paraphrase-robust semantic-similarity axis. The default lexical TF-IDF path stays fast and dependency-free; the `Embedder` trait accepts any backend (this extra, ONNX runtime, HF Inference API, OpenAI embeddings, …). | | `shadow-diff[otel]` | `opentelemetry-sdk` | Export traces to any OTel-compatible backend | | `shadow-diff[serve]` | `fastapi`, `uvicorn`, `websockets` | `shadow serve` (live `.shadow/` dashboard) and `shadow dashboard --report report.json` (single-report viewer) | | `shadow-diff[mcp]` | `mcp` | `shadow mcp-serve` Model Context Protocol server | | `shadow-diff[multimodal]` | `Pillow`, `imagehash` | Image / multimodal diff | | `shadow-diff[sign]` | `sigstore` | Sigstore keyless signing for ABOM certificates | | `shadow-diff[langgraph]` | `langgraph` | LangGraph agent adapter | | `shadow-diff[crewai]` | `crewai` | CrewAI agent adapter | | `shadow-diff[ag2]` | `ag2` | AG2 / Autogen agent adapter | | `shadow-diff[all]` | everything above | One-shot install for trying the full feature surface | | `shadow-diff[dev]` | test/lint/type-check tooling | Contributing to Shadow itself | Combine extras with comma-separated form: ```bash pip install 'shadow-diff[anthropic,openai,embeddings,otel]' ``` ### Telemetry: off by default, opt-in only Shadow ships an opt-in usage-telemetry hook. When enabled, each event includes: CLI command / event name, SDK version, Python version, OS, CPU architecture, and an anonymous install ID (random UUID4 generated once, stored at `~/.shadow/install_id`, never tied to identity). **No traces, no prompts, no user data.** Telemetry is **off by default** — the first-run prompt asks before enabling. CI environments are detected and skipped automatically. Hard-disable with `SHADOW_TELEMETRY=off` in your shell. Source: [`python/src/shadow/_telemetry.py`](python/src/shadow/_telemetry.py). Shadow never uploads `.agentlog` content, prompt text, response text, or tool arguments. The only fields collected when telemetry is enabled are listed in the module docstring. ## Try it in sixty seconds ```bash git clone https://github.com/manav8498/Shadow.git cd Shadow/examples/refund-causal-diagnosis ./demo.sh ``` That runs `shadow diagnose-pr` end-to-end on a refund-agent scenario where the candidate config drops a "always confirm before refunding" instruction. Shadow names the prompt change as the dominant cause and tells you the fix: ``` ## Shadow verdict: STOP This PR violates a critical policy and must not merge as-is. This PR changes agent behavior on **3** / **3** production-like traces. ### Dominant cause `prompt.system` appears to be the main cause. - Axis: `trajectory` - ATE: `+0.60` - 95% CI: `[0.60, 0.60]` - E-value: `6.7` ### Why it matters 3 traces violate the `confirm-before-refund` policy rule. ### Suggested fix Review the prompt change at `prompts/candidate.md` — restore the instruction or constraint it removed. ``` For the underlying nine-axis behavior diff (without the causal layer), `shadow demo` runs the same fixtures through `shadow diff` and prints a severity table — useful when you want raw signal numbers: ```bash shadow demo ``` That output looks like this (abbreviated): ``` signal baseline candidate change severity ───────────────────────────────────────────────────────────── response meaning 1.000 0.435 -0.565 severe tool calls 0.000 0.000 +0.000 none refusals 0.000 0.333 +0.333 severe response length 26.000 52.000 +26.000 minor response time 98.000 412.000 +314.000 severe output format 1.000 0.000 -1.000 severe top divergences: #1 turn 0 — tool set changed: removed `search_files(query)`, added `search_files(limit,query)` #2 turn 2 — stop_reason changed: `end_turn` → `content_filter` recommendations: error Refusal rate is up severely. Check for stricter system instructions. error Review tool-schema change at turn 0: call shape diverged. warning Review response text at turn 1: semantic content shifted. ``` Three things to read top-to-bottom: the severity column tells you which signals moved, the top-divergences list names the specific changes, and the recommendations tell you what to check first. **A reviewer doesn't need to know any Shadow vocabulary** — the recommendation lines speak plain English. Use `shadow quickstart` when you want a writable copy of the demo files (agent.py, configs, fixtures) to edit and re-run: ```bash shadow quickstart shadow diff shadow-quickstart/fixtures/baseline.agentlog \ shadow-quickstart/fixtures/candidate.agentlog ``` ## Daily workflow — Shadow as `pytest` for agent behavior The four commands you'll actually run all day: ```bash shadow inspect trace.agentlog # debug a single trace shadow scan baseline_traces/ candidate_traces/ # block secret leaks shadow baseline create baseline_traces/ # pin the gold standard shadow gate-pr ... # gate every PR ``` **`shadow inspect`** opens a `.agentlog` file in your terminal — turn, role, tokens, latency, cost, redactions, first divergence. What `pytest -v` is to test runs, this is to recorded traces. Pass two paths to compare side-by-side; the first divergent turn is highlighted in red. **`shadow scan`** walks your `.agentlog` files looking for OpenAI / Anthropic / GitHub / AWS keys, JWTs, PEM private keys, emails, phone numbers, and Luhn-valid credit cards. Exit code is non-zero on any hit, so it composes into CI before `shadow gate-pr`. Add company-specific patterns via `--patterns ci/extra-secrets.txt`. **`shadow baseline`** is a frozen-trace workflow modeled on Insta and Jest snapshots: ```bash shadow baseline create baseline_traces/ # pin the hash into shadow.yaml shadow baseline verify # exit non-zero on drift shadow baseline update --force # re-pin after a deliberate regen shadow baseline approve candidate_traces/ --force # promote candidate to baseline ``` The hash lives in `shadow.yaml` so PRs that change the baseline show up in `git diff` as a single line. Reviewers see "the baseline hash changed in this PR" — a deliberate signal that the agent's expected behavior was reset. **`shadow gate-pr`** prints a 1-screen failure summary that reads like a pytest assertion failure: ``` ✗ Shadow gate-pr: STOP (exit 2) — 3/3 traces affected Cause: prompts/refund.md:17 appears to be the main cause. - removed: 4. Always confirm the refund amount before issuing the refund. trajectory: ATE=+0.60 95% CI=[0.50, 0.70] E=6.7 Policy: 3 new violation(s) of confirm-before-refund Fix: Restore the prompt instruction at prompts/refund.md:17 Verify: shadow verify-fix --report .shadow/diagnose-pr/report.json ``` Pass `-v` / `--verbose` to also dump the full JSON report. The PR-comment markdown is always written via `--pr-comment` regardless of verbosity. Exit code: `0` ship, `1` hold/probe, `2` stop, `3` internal error. ## Get a Shadow comment on every PR (≈ 10 minutes) The end-to-end setup, from a fresh repo to seeing your first Shadow comment land on a real PR. Skip any step you've already done. **1. Install Shadow** (one line): ```bash pip install shadow-diff ``` **2. Record baseline + candidate traces.** Wrap the place where your agent runs in a `Session` block. Shadow auto-instruments OpenAI, Anthropic, LiteLLM, LangChain `ChatOpenAI`, and Vercel AI SDK (the `ai` package) calls and writes content-addressed `.agentlog` files: ```python # scripts/record_traces.py from shadow.sdk import Session with Session(output_path="baseline.agentlog"): run_my_agent() # whatever your existing entry-point is ``` Run it once on baseline (`main`), once on the candidate (your PR branch), and commit both `.agentlog` directories. Traces stay on your disk and inside your repo — nothing is uploaded. **3. Drop in the diagnose-pr GitHub Action.** Scaffold a workflow that runs `shadow diagnose-pr` on every PR — names the exact change that broke the agent, posts the verdict + suggested fix as a comment, and gates the merge with verdict-mapped exit codes (`ship`=0, `hold`/`probe`=1, `stop`=2): ```bash shadow init --github-action git add .github/workflows/shadow-diagnose-pr.yml && git commit -m "ci: shadow diagnose-pr" ``` The generated workflow uses the [`shadow-diagnose-pr`](.github/actions/shadow-diagnose-pr/action.yml) composite action with the `recorded` backend by default (offline, no API spend). Switch to `--backend live` (and add an `OPENAI_API_KEY` secret) when you want real intervention-based ATE + bootstrap CI + E-value on the dominant cause. For a hand-rolled workflow, the action is a one-liner: ```yaml - uses: manav8498/Shadow/.github/actions/shadow-diagnose-pr@main with: baseline-traces: fixtures/baseline_traces candidate-traces: fixtures/candidate_traces baseline-config: configs/baseline.yaml candidate-config: configs/candidate.yaml policy: configs/shadow-policy.yaml backend: recorded ``` Forked PRs run with a read-only `GITHUB_TOKEN`, so the action falls back to the workflow summary instead of attempting to post a comment that would fail. The marker-dedup means a re-run on the same PR edits the existing comment rather than stacking new ones. **4. Open a PR.** The Shadow comment lands automatically. It leads with the verdict, names the dominant cause (or lists likely candidates when attribution can't crown one), explains the policy violation in plain English, and ends with the `shadow verify-fix` command to confirm a fix before merge. See [`docs/sample-pr-comment.md`](docs/sample-pr-comment.md) for an example, or [`examples/refund-causal-diagnosis/`](examples/refund-causal-diagnosis/) for a runnable scenario you can cargo-cult into your own repo. That's it. The earlier path of `shadow init --github-action --legacy-diff` (raw nine-axis `shadow diff` without causal attribution) still ships for repos that prefer the older flow; everything below is optional depth. ## Writing behavior rules The diff tells you what changed. A policy tells you what is not allowed to change. Write one YAML file that declares the agent's behavioral contract: ```yaml # shadow-policy.yaml rules: - id: confirm-before-refund kind: must_call_before params: { first: confirm_refund_amount, then: issue_refund } severity: error - id: never-leak-ssn kind: forbidden_text params: { text: "SSN:" } severity: error - id: finish-cleanly kind: required_stop_reason params: { allowed: [end_turn, tool_use] } severity: error - id: cost-ceiling kind: max_total_tokens params: { limit: 100000 } ``` Run: ```bash shadow diff baseline.agentlog candidate.agentlog --policy shadow-policy.yaml ``` The candidate trace is checked against every rule. Violations that are new in the candidate are flagged as regressions. Violations that existed in the baseline and are now cleared are flagged as fixes. Twelve rule kinds ship today: `must_call_before`, `must_call_once`, `no_call`, `max_turns`, `required_stop_reason`, `max_total_tokens`, `must_include_text`, `forbidden_text`, `must_match_json_schema`, `must_remain_consistent`, `must_followup`, `must_be_grounded` (cheap lexical grounding gate, not NLI-backed faithfulness — see [docs/features/policy.md](docs/featur … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [manav8498](https://github.com/manav8498) - **Source:** [manav8498/Shadow](https://github.com/manav8498/Shadow) - **License:** Apache-2.0 - **Homepage:** https://manav8498.github.io/Shadow/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** yes - **Filesystem access:** no - **Shell / process execution:** yes - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: flagged — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-manav8498-shadow - Seller: https://agentstack.voostack.com/s/manav8498 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.