# Auditable > Audit any agent decision across its past, present, and future, on one typed graph. - **Type:** MCP server - **Install:** `agentstack add mcp-yzhao062-auditable` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [yzhao062](https://agentstack.voostack.com/s/yzhao062) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [yzhao062](https://github.com/yzhao062) - **Source:** https://github.com/yzhao062/auditable - **Website:** https://auditable-ai.readthedocs.io/ ## Install ```sh agentstack add mcp-yzhao062-auditable ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # auditable **`auditable` is an open-source system of record for AI-agent decisions: it captures what each decision relied on, replays it against live state, and rolls back the committed action when it no longer holds.** *Your logs show what the agent did. `auditable` shows what it relied on, replays it under live state, and rolls it back when it no longer holds.* [](https://pypi.org/project/auditable/) [](https://pypi.org/project/auditable/) [](LICENSE) [](https://github.com/yzhao062/auditable) [Compare](#auditable-vs-the-tools-you-already-use) · [Flagship Demo](#the-flagship-moment) · [Plug In Your Agent](#plug-in-your-agent) · [Lifecycle](#the-lifecycle) · [Install](#install) · [Docs](https://auditable-ai.readthedocs.io/) · [Roadmap](#roadmap) ### auditable vs. the Tools You Already Use | Capability | **auditable** | Tracing and Observability[LangSmith](#ref-1), [Langfuse](#ref-2), [Phoenix](#ref-3) | Eval Harnesses[DeepEval](#ref-5), [Ragas](#ref-6), [promptfoo](#ref-7) | Guardrails[NeMo](#ref-9), [Guardrails AI](#ref-10), [Lakera](#ref-11) | |---|:---:|:---:|:---:|:---:| | Captures the dependency state a decision relied on | ✅ | 🟡 | ❌ | ❌ | | Re-decides under live state (replay) | ✅ | 🟡 | ❌ | ❌ | | Reverses a committed action (rollback) | ✅ | ❌ | ❌ | ❌ | | One graph across plan, run, and review | ✅ | 🟡 | ❌ | ❌ | | Framework-agnostic, dependency-free core | ✅ | 🟡 | 🟡 | 🟡 | ✅ yes · 🟡 partial · ❌ no, per each tool's public docs (2026). These categories are complementary to `auditable`, not competitors: tracing captures execution, evals score quality, guardrails block at runtime. `auditable` is the audit-and-recovery layer that plugs into them. See [References](#references). **`auditable` is the only column with the full set.** What makes it new: - **A unified graph representation for agentic AI.** Every agent run becomes one typed graph that PRE, LIVE, and POST all read. One representation from plan to live operation to review, instead of three disconnected tools. - **Recover, do not just observe.** `auditable` captures the dependency state a decision relied on, replays it under the state that is live now, and reverses the committed action through a compensating rail when it no longer holds. Logging tells you what broke; `auditable` undoes it. - **One decision, three spans, judged together.** Data, model, and harness are bound in a single signed, hash-chained record, so a decision is audited as one unit, not three disconnected logs. ### Proven on Public Agent Benchmarks The graph is not just structure. Across six public agent corpora (GRADE, [arXiv:2606.22741](https://arxiv.org/abs/2606.22741)), the dependency layer predicts which runs fail at **ROC-AUC 0.805** where run length carries no signal, and the execution layer localizes the faulting step at **Top-3 0.614**. See [the benchmark detail](docs/post-analysis.md) for the full numbers and corpora. Figures from GRADE (arXiv:2606.22741). Agents act on dependency state that quietly drifts. A budget read minutes ago can fall below the amount already committed; a price pinned at plan time can move before the action lands. Most tools log what happened, yet they cannot re-decide under the state that is live now, so a stale decision stands until a human notices. `auditable` closes that recovery gap across the lifecycle on one graph: it captures the decision, replays it against live state, and reverses the committed action when it no longer holds. ## The Flagship Moment One payment, walked through the whole lifecycle in 18 `auditable` calls. The agent approves a $2,083.20 vendor payment against a budget snapshot that covered it. Six days later the live budget has dropped below the amount. `replay` re-decides on the live state, and the gate reverses the committed payment. This is recovery, not a log line. ```bash pip install "auditable[graph]" python examples/example_end_to_end.py ``` The run prints a single audit report: a REVIEW verdict, the keystone with its coverage reason, six findings with severity tags and recommended actions, and the LIVE recovery that rolled the payment back. Paste it into a pull request or an issue. ## Plug In Your Agent `auditable` works with the agent you already run. Wrap a LangGraph `StateGraph` of plain sync or async function nodes over TypedDict state, and every node's reads and writes over the state channels become **observed** dependency edges, matched across the superstep barrier, with no change to your node logic: ```python from langgraph.graph import StateGraph from auditable import analyze_run from auditable.integrations.langgraph import instrument builder = instrument(StateGraph(State)) # 1) wrap once, then build / compile / invoke as usual ... graph = builder.compile() graph.invoke(initial_state) report = analyze_run(builder, adapter=builder) # 2) the observed dependency graph for that run print(report.keystone) # the step the rest of the run rests on ``` ```bash pip install "auditable[langgraph]" python examples/example_langgraph_capture.py # a real LangGraph run -> observed=100%, keystone named ``` The same wrapped builder also yields **replayable** records, so the LIVE pillar runs on the captured run, not a hand-built dict. `builder.to_records(decisions={"approve": "vendor_payment"})` lowers a marked decision node into a `DecisionRecord` carrying the state it relied on, and `replay(record, live_state=...)` re-decides it under state that is live now and routes a rollback. See [`example_langgraph_live_replay.py`](examples/example_langgraph_live_replay.py): one real LangGraph run, captured, ranked (POST), then replayed and reversed under a drifted budget (LIVE). Want a real model in the loop? [`example_langgraph_llm_agent.py`](examples/example_langgraph_llm_agent.py) runs the same capture with live LLM nodes against any OpenAI-compatible endpoint (a plain OpenAI key, a gateway, or a local vLLM or Ollama server). The captured edges are identical, because the capture sees the state channels a node read and wrote, not the model call inside it. Not on LangGraph? A one-line callback captures any LangChain runnable (`auditable.integrations.langchain`), an in-place `instrument` records every tool call at an MCP server boundary (`auditable.integrations.mcp`), and the framework-agnostic `TouchRecorder` captures the same observed edges from any loop (a raw OpenAI or Anthropic agent, your own scheduler) by declaring each step's `reads()` and `writes()`. See [`examples/example_touch_capture.py`](examples/example_touch_capture.py). **Roadmap:** CrewAI and OpenTelemetry. ## Examples and Integrations Point `auditable` at a scenario and it builds the same typed graph. Each row links a runnable example; browse them all in [`examples/README.md`](examples/README.md). | Scenario | Pillar | Builds the graph from | Run | |---|---|---|---| | Capture a real LangGraph agent | LIVE → POST | a live `StateGraph` run (`instrument`) | [`example_langgraph_capture.py`](examples/example_langgraph_capture.py) | | Capture a real LLM agent (live model) | LIVE → POST | a model-driven `StateGraph` run (`instrument` + OpenAI-compatible) | [`example_langgraph_llm_agent.py`](examples/example_langgraph_llm_agent.py) | | Capture a real LangChain runnable | LIVE → POST | a callback on any runnable (`LangChainCallbackHandler`) | [`example_langchain_capture.py`](examples/example_langchain_capture.py) | | Capture any tool loop by hand | LIVE → POST | declared resource touches (`TouchRecorder`) | [`example_touch_capture.py`](examples/example_touch_capture.py) | | Capture MCP tool traffic at the server boundary | LIVE → POST | an instrumented FastMCP server (`instrument`) | [`example_mcp_capture.py`](examples/example_mcp_capture.py) | | Capture a LangGraph agent, then replay and reverse it | LIVE → POST | a live `StateGraph` run lowered to replayable records (`instrument` + `to_records`) | [`example_langgraph_live_replay.py`](examples/example_langgraph_live_replay.py) | | Lint a declared plan before deploy | PRE | a framework-agnostic plan dict (`declared_plan_v1`) | [`example_pre_lint_plan.py`](examples/example_pre_lint_plan.py) | | Recover a payment as the budget drifts | LIVE | a decision captured live (`audit` + `replay`) | [`example_live_replay.py`](examples/example_live_replay.py) | | Monitor a run as it streams, name the keystone live | LIVE | a run scored prefix by prefix (`LiveSession`) | [`example_live_monitor.py`](examples/example_live_monitor.py) | | Rank a tau-bench run, name the keystone | POST | a tau-bench trajectory (`tau_bench_prior_db_reads_v1`) | [`example_post_rank_run.py`](examples/example_post_rank_run.py) | | Walk one payment through every pillar | PRE, LIVE, POST | the full lifecycle (`own_record_v1` for POST) | [`example_end_to_end.py`](examples/example_end_to_end.py) | The first six rows capture a real run; see [Plug In Your Agent](#plug-in-your-agent) for the two-line setup. **Roadmap:** CrewAI and OpenTelemetry. ## The Lifecycle `auditable` runs the same detection-and-report pass over one typed decision graph at three points in an agent's life. The graph kernel stays constant; only the pillar changes (when it fires, what it scores). | Pillar | When It Fires | Public Entry | Focus | |---|---|---|---| | **PRE** | Before deploy | `analyze_plan` | Read-only structural lints on a declared plan. Names the control-flow chokepoint. Dependency-state risk withheld. | | **LIVE** | While running | `audit` + `replay` + `ActionGate` | Capture a decision, re-decide under live state, route a fix (allow, block, review, rollback) through a rail. The sharpest pillar. | | **POST** | After a run | `analyze_run` | Rank a finished run by structural blast share. Name the keystone the run rests on, so you review that step first. | ## Install ```bash pip install auditable ``` The core is dependency-free and torch-free. Structural-graph analysis (`analyze_plan` for PRE, `analyze_run` for POST) needs the optional graph extra (NetworkX): ```bash pip install "auditable[graph]" pip install "auditable[anomaly]" ``` The graph extra adds NetworkX. The anomaly extra adds the fitted `DataAuditor` path through PyOD; freshness fallback and the LIVE example run on the core install alone. Docs: https://auditable-ai.readthedocs.io/ ## The Three Pillars, in Detail PRE: lint the plan before deploy (four read-only lints, the chokepoint, a coverage report) Point `analyze_plan` at a declared plan (a plain dict, the neutral target a LangGraph, CrewAI, or AutoGen front-end would lower into) and it runs read-only structural lints over the plan graph. Every check is a pure NetworkX query: no value is executed, and every finding is a structural design warning, not a validated failure prediction. ```python from auditable.graph.pre import analyze_plan from auditable.graph.adapters import declared_plan_v1 plan = { "nodes": [ # 0: read a volatile price, but grant scope far beyond what it read. {"idx": 0, "agent": "planner", "kind": "tool_call", "reads": [{"id": "price", "volatile": True}], "scope": ["price", "ledger", "vendor_db"]}, # 1: a decision that rests on the unpinned, un-revalidated price. {"idx": 1, "agent": "planner", "kind": "decision", "reads": [{"id": "price", "producer": 0, "volatile": True}], "control_preds": [0]}, # 2: a consequential write of 'order', with no prior read of 'order' # and no re-read of 'price' between the volatile read and the action. {"idx": 2, "agent": "executor", "kind": "tool_call", "reads": [{"id": "price", "producer": 0, "volatile": True}], "writes": ["order"], "control_preds": [1]}, ] } report = analyze_plan(plan, adapter=declared_plan_v1) print(report) ``` The four shipping lints, all read-only queries at `severity='warning'`: | Lint | Fires When | |---|---| | `write_with_no_prior_read` | A node writes a resource that nothing in its backward slice ever read. | | `flippable_dependency_annotation` | An unpinned, non-revalidated volatile dependency feeds a decision. This is an annotation; the would-it-flip question needs runtime values and is out of scope at PRE. | | `scope_vs_snapshot` | Granted tool scope strictly exceeds the snapshot the node read. | | `missing_revalidation_barrier` | A volatile read reaches a consequential action with no intervening re-read. Drift confirmation needs runtime values and is out of scope at PRE. | The report also names the **execution-topology keystone**: the structural chokepoint of the declared plan, the node that the most other nodes transitively follow in control flow (the argmax of `execution_reach` over the `handoff_to` projection). This is a structural design lint, a separate concept from the POST blast-radius keystone, and it does not predict failure. Alongside the lints, the **Preflight Coverage Report** is a descriptive coverage-readiness view, explicitly not a risk score. It reports the dependency-edge grade mix, the observed fraction, the saturation ratio, the exact no-score reason the runtime scorer would apply, which declared reads, writes, and edges still lack a resource identity, and the declared revalidation barriers per resource. Two boundaries, stated plainly. Dependency-state blast-share risk is **withheld** at PRE: the declared dependency layer is declared-only (observed fraction zero), so `analyze_plan` returns `state_b_risk=None` with `state_b_withheld=True` and a reason string, and it raises rather than emit a number if a scored verdict ever came back. A table-stakes OWASP-Agentic and CWE rule floor is **planned**, not shipping. See [`examples/example_pre_lint_plan.py`](examples/example_pre_lint_plan.py) and the [PRE rules reference](docs/pre-rules.md). POST: rank a finished run, find the keystone (`analyze_run` over a recorded trajectory) `analyze_run` reads a recorded agent run, builds one decision graph, and ranks every step by how much of the run transitively rests on it, so you review the keystone first. On a tau-bench airline trajectory, the one reservation read that both later writes depend on is the keystone. ```python from auditable import analyze_run from auditable.graph.adapters import tau_bench_prior_db_reads_v1 report = analyze_run(run, adapter=tau_bench_prior_db_reads_v1) k = report.keystone print(k.idx, k.node_attrs["tool"]) # 2 get_reservation_details ``` The score is an uncalibrated triage ranking, not a calibrated probability. In a no-score state (`no_score:single_decision`, `no_score:low_coverage`) the scores are `None`, so a withheld score never reads as zero risk. The corpus write-to-read edges are modeled (a conservative prior-read upper bound, not a causal label); the report carries these caveats in `report.notes`. The trajectory is modeled on [tau-bench](https://github.com/sierra-research/tau-bench) (Sierra Research, MIT). See [`examples/example_post_rank_run.py`](examples/example_post_rank_run.py) and the [POST analysis reference](docs/post-analysis.md). How It Works: the two-layer graph, the signed record, the rail (kernel internals) *auditable links a run into one graph with two edge layers: execution (control flow, observed from the trace) over dependency (what each step relied on). When a step rested on a value that has since gone stale, like `price`, `replay` catches it. The record itself binds three spans per decision (data, model, harness).* The graph kernel has two edge layers, and the distinction is load-bearing. Execution edges (`emits`, `handoff_to`) are observed from the trace. Dependency edges (`depends_on`) are declared, inferred, or observed, and every edge records how it is known: the corpus and plan adapters declare or infer them, while the LangGraph capture path and the `TouchRecorder` read them off a real run as observed channel touches. PRE and POST both run over this same typed graph; `audit()` is the ergonomi … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [yzhao062](https://github.com/yzhao062) - **Source:** [yzhao062/auditable](https://github.com/yzhao062/auditable) - **License:** Apache-2.0 - **Homepage:** https://auditable-ai.readthedocs.io/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** yes *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-yzhao062-auditable - Seller: https://agentstack.voostack.com/s/yzhao062 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.