Auditable

auditable

auditable is an open-source system of record for AI-agent decisions: it captures what each decision relied on, replays it against live state, and rolls back the committed action when it no longer holds.

Your logs show what the agent did. auditable shows what it relied on, replays it under live state, and rolls it back when it no longer holds.

[](https://pypi.org/project/auditable/) [](https://pypi.org/project/auditable/) [](LICENSE) [](https://github.com/yzhao062/auditable)

[Compare](#auditable-vs-the-tools-you-already-use) · [Flagship Demo](#the-flagship-moment) · [Plug In Your Agent](#plug-in-your-agent) · [Lifecycle](#the-lifecycle) · [Install](#install) · Docs · [Roadmap](#roadmap)

auditable vs. the Tools You Already Use

| Capability | auditable | Tracing and Observability[LangSmith](#ref-1), [Langfuse](#ref-2), [Phoenix](#ref-3) | Eval Harnesses[DeepEval](#ref-5), [Ragas](#ref-6), [promptfoo](#ref-7) | Guardrails[NeMo](#ref-9), [Guardrails AI](#ref-10), [Lakera](#ref-11) | |---|:---:|:---:|:---:|:---:| | Captures the dependency state a decision relied on | ✅ | 🟡 | ❌ | ❌ | | Re-decides under live state (replay) | ✅ | 🟡 | ❌ | ❌ | | Reverses a committed action (rollback) | ✅ | ❌ | ❌ | ❌ | | One graph across plan, run, and review | ✅ | 🟡 | ❌ | ❌ | | Framework-agnostic, dependency-free core | ✅ | 🟡 | 🟡 | 🟡 |

✅ yes · 🟡 partial · ❌ no, per each tool's public docs (2026). These categories are complementary to auditable, not competitors: tracing captures execution, evals score quality, guardrails block at runtime. auditable is the audit-and-recovery layer that plugs into them. See [References](#references).

auditable is the only column with the full set. What makes it new:

• A unified graph representation for agentic AI. Every agent run becomes one typed graph that PRE, LIVE, and POST all read. One representation from plan to live operation to review, instead of three disconnected tools.
• Recover, do not just observe. auditable captures the dependency state a decision relied on, replays it under the state that is live now, and reverses the committed action through a compensating rail when it no longer holds. Logging tells you what broke; auditable undoes it.
• One decision, three spans, judged together. Data, model, and harness are bound in a single signed, hash-chained record, so a decision is audited as one unit, not three disconnected logs.

Proven on Public Agent Benchmarks

The graph is not just structure. Across six public agent corpora (GRADE, arXiv:2606.22741), the dependency layer predicts which runs fail at ROC-AUC 0.805 where run length carries no signal, and the execution layer localizes the faulting step at Top-3 0.614. See [the benchmark detail](docs/post-analysis.md) for the full numbers and corpora.

Figures from GRADE (arXiv:2606.22741).

Agents act on dependency state that quietly drifts. A budget read minutes ago can fall below the amount already committed; a price pinned at plan time can move before the action lands. Most tools log what happened, yet they cannot re-decide under the state that is live now, so a stale decision stands until a human notices. auditable closes that recovery gap across the lifecycle on one graph: it captures the decision, replays it against live state, and reverses the committed action when it no longer holds.

The Flagship Moment

One payment, walked through the whole lifecycle in 18 auditable calls. The agent approves a $2,083.20 vendor payment against a budget snapshot that covered it. Six days later the live budget has dropped below the amount. replay re-decides on the live state, and the gate reverses the committed payment. This is recovery, not a log line.

pip install "auditable[graph]"
python examples/example_end_to_end.py

The run prints a single audit report: a REVIEW verdict, the keystone with its coverage reason, six findings with severity tags and recommended actions, and the LIVE recovery that rolled the payment back. Paste it into a pull request or an issue.

Plug In Your Agent

auditable works with the agent you already run. Wrap a LangGraph StateGraph of plain sync or async function nodes over TypedDict state, and every node's reads and writes over the state channels become observed dependency edges, matched across the superstep barrier, with no change to your node logic:

from langgraph.graph import StateGraph
from auditable import analyze_run
from auditable.integrations.langgraph import instrument

builder = instrument(StateGraph(State))          # 1) wrap once, then build / compile / invoke as usual
...
graph = builder.compile()
graph.invoke(initial_state)
report = analyze_run(builder, adapter=builder)   # 2) the observed dependency graph for that run
print(report.keystone)                           # the step the rest of the run rests on

pip install "auditable[langgraph]"
python examples/example_langgraph_capture.py     # a real LangGraph run -> observed=100%, keystone named

The same wrapped builder also yields replayable records, so the LIVE pillar runs on the captured run, not a hand-built dict. builder.to_records(decisions={"approve": "vendor_payment"}) lowers a marked decision node into a DecisionRecord carrying the state it relied on, and replay(record, live_state=...) re-decides it under state that is live now and routes a rollback. See [example_langgraph_live_replay.py](examples/examplelanggraphlive_replay.py): one real LangGraph run, captured, ranked (POST), then replayed and reversed under a drifted budget (LIVE).

Want a real model in the loop? [example_langgraph_llm_agent.py](examples/examplelanggraphllm_agent.py) runs the same capture with live LLM nodes against any OpenAI-compatible endpoint (a plain OpenAI key, a gateway, or a local vLLM or Ollama server). The captured edges are identical, because the capture sees the state channels a node read and wrote, not the model call inside it.

Not on LangGraph? A one-line callback captures any LangChain runnable (auditable.integrations.langchain), an in-place instrument records every tool call at an MCP server boundary (auditable.integrations.mcp), and the framework-agnostic TouchRecorder captures the same observed edges from any loop (a raw OpenAI or Anthropic agent, your own scheduler) by declaring each step's reads() and writes(). See [examples/example_touch_capture.py](examples/exampletouchcapture.py). Roadmap: CrewAI and OpenTelemetry.

Examples and Integrations

Point auditable at a scenario and it builds the same typed graph. Each row links a runnable example; browse them all in [examples/README.md](examples/README.md).

| Scenario | Pillar | Builds the graph from | Run | |---|---|---|---| | Capture a real LangGraph agent | LIVE → POST | a live StateGraph run (instrument) | [example_langgraph_capture.py](examples/examplelanggraphcapture.py) | | Capture a real LLM agent (live model) | LIVE → POST | a model-driven StateGraph run (instrument + OpenAI-compatible) | [example_langgraph_llm_agent.py](examples/examplelanggraphllmagent.py) | | Capture a real LangChain runnable | LIVE → POST | a callback on any runnable (LangChainCallbackHandler) | [example_langchain_capture.py](examples/examplelangchaincapture.py) | | Capture any tool loop by hand | LIVE → POST | declared resource touches (TouchRecorder) | [example_touch_capture.py](examples/exampletouchcapture.py) | | Capture MCP tool traffic at the server boundary | LIVE → POST | an instrumented FastMCP server (instrument) | [example_mcp_capture.py](examples/examplemcpcapture.py) | | Capture a LangGraph agent, then replay and reverse it | LIVE → POST | a live StateGraph run lowered to replayable records (instrument + to_records) | [example_langgraph_live_replay.py](examples/examplelanggraphlivereplay.py) | | Lint a declared plan before deploy | PRE | a framework-agnostic plan dict (declared_plan_v1) | [example_pre_lint_plan.py](examples/exampleprelintplan.py) | | Recover a payment as the budget drifts | LIVE | a decision captured live (audit + replay) | [example_live_replay.py](examples/examplelivereplay.py) | | Monitor a run as it streams, name the keystone live | LIVE | a run scored prefix by prefix (LiveSession) | [example_live_monitor.py](examples/examplelivemonitor.py) | | Rank a tau-bench run, name the keystone | POST | a tau-bench trajectory (tau_bench_prior_db_reads_v1) | [example_post_rank_run.py](examples/examplepostrankrun.py) | | Walk one payment through every pillar | PRE, LIVE, POST | the full lifecycle (own_record_v1 for POST) | [example_end_to_end.py](examples/exampleendto_end.py) |

The first six rows capture a real run; see [Plug In Your Agent](#plug-in-your-agent) for the two-line setup. Roadmap: CrewAI and OpenTelemetry.

The Lifecycle

auditable runs the same detection-and-report pass over one typed decision graph at three points in an agent's life. The graph kernel stays constant; only the pillar changes (when it fires, what it scores).

| Pillar | When It Fires | Public Entry | Focus | |---|---|---|---| | PRE | Before deploy | analyze_plan | Read-only structural lints on a declared plan. Names the control-flow chokepoint. Dependency-state risk withheld. | | LIVE | While running | audit + replay + ActionGate | Capture a decision, re-decide under live state, route a fix (allow, block, review, rollback) through a rail. The sharpest pillar. | | POST | After a run | analyze_run | Rank a finished run by structural blast share. Name the keystone the run rests on, so you review that step first. |

Install

pip install auditable

The core is dependency-free and torch-free. Structural-graph analysis (analyze_plan for PRE, analyze_run for POST) needs the optional graph extra (NetworkX):

pip install "auditable[graph]"
pip install "auditable[anomaly]"

The graph extra adds NetworkX. The anomaly extra adds the fitted DataAuditor path through PyOD; freshness fallback and the LIVE example run on the core install alone.

Docs: https://auditable-ai.readthedocs.io/

The Three Pillars, in Detail

PRE: lint the plan before deploy (four read-only lints, the chokepoint, a coverage report)

Point analyze_plan at a declared plan (a plain dict, the neutral target a LangGraph, CrewAI, or AutoGen front-end would lower into) and it runs read-only structural lints over the plan graph. Every check is a pure NetworkX query: no value is executed, and every finding is a structural design warning, not a validated failure prediction.

from auditable.graph.pre import analyze_plan
from auditable.graph.adapters import declared_plan_v1

plan = {
    "nodes": [
        # 0: read a volatile price, but grant scope far beyond what it read.
        {"idx": 0, "agent": "planner", "kind": "tool_call",
         "reads": [{"id": "price", "volatile": True}],
         "scope": ["price", "ledger", "vendor_db"]},
        # 1: a decision that rests on the unpinned, un-revalidated price.
        {"idx": 1, "agent": "planner", "kind": "decision",
         "reads": [{"id": "price", "producer": 0, "volatile": True}],
         "control_preds": [0]},
        # 2: a consequential write of 'order', with no prior read of 'order'
        #    and no re-read of 'price' between the volatile read and the action.
        {"idx": 2, "agent": "executor", "kind": "tool_call",
         "reads": [{"id": "price", "producer": 0, "volatile": True}],
         "writes": ["order"], "control_preds": [1]},
    ]
}

report = analyze_plan(plan, adapter=declared_plan_v1)
print(report)

The four shipping lints, all read-only queries at severity='warning':

| Lint | Fires When | |---|---| | write_with_no_prior_read | A node writes a resource that nothing in its backward slice ever read. | | flippable_dependency_annotation | An unpinned, non-revalidated volatile dependency feeds a decision. This is an annotation; the would-it-flip question needs runtime values and is out of scope at PRE. | | scope_vs_snapshot | Granted tool scope strictly exceeds the snapshot the node read. | | missing_revalidation_barrier | A volatile read reaches a consequential action with no intervening re-read. Drift confirmation needs runtime values and is out of scope at PRE. |

The report also names the execution-topology keystone: the structural chokepoint of the declared plan, the node that the most other nodes transitively follow in control flow (the argmax of execution_reach over the handoff_to projection). This is a structural design lint, a separate concept from the POST blast-radius keystone, and it does not predict failure.

Alongside the lints, the Preflight Coverage Report is a descriptive coverage-readiness view, explicitly not a risk score. It reports the dependency-edge grade mix, the observed fraction, the saturation ratio, the exact no-score reason the runtime scorer would apply, which declared reads, writes, and edges still lack a resource identity, and the declared revalidation barriers per resource.

Two boundaries, stated plainly. Dependency-state blast-share risk is withheld at PRE: the declared dependency layer is declared-only (observed fraction zero), so analyze_plan returns state_b_risk=None with state_b_withheld=True and a reason string, and it raises rather than emit a number if a scored verdict ever came back. A table-stakes OWASP-Agentic and CWE rule floor is planned, not shipping.

See [examples/example_pre_lint_plan.py](examples/exampleprelint_plan.py) and the [PRE rules reference](docs/pre-rules.md).

POST: rank a finished run, find the keystone (analyze_run over a recorded trajectory)

analyze_run reads a recorded agent run, builds one decision graph, and ranks every step by how much of the run transitively rests on it, so you review the keystone first. On a tau-bench airline trajectory, the one reservation read that both later writes depend on is the keystone.

from auditable import analyze_run
from auditable.graph.adapters import tau_bench_prior_db_reads_v1

report = analyze_run(run, adapter=tau_bench_prior_db_reads_v1)
k = report.keystone
print(k.idx, k.node_attrs["tool"])   # 2  get_reservation_details

The score is an uncalibrated triage ranking, not a calibrated probability. In a no-score state (no_score:single_decision, no_score:low_coverage) the scores are None, so a withheld score never reads as zero risk. The corpus write-to-read edges are modeled (a conservative prior-read upper bound, not a causal label); the report carries these caveats in report.notes. The trajectory is modeled on tau-bench (Sierra Research, MIT). See [examples/example_post_rank_run.py](examples/examplepostrankrun.py) and the [POST analysis reference](docs/post-analysis.md).

How It Works: the two-layer graph, the signed record, the rail (kernel internals)

auditable links a run into one graph with two edge layers: execution (control flow, observed from the trace) over dependency (what each step relied on). When a step rested on a value that has since gone stale, like price, replay catches it. The record itself binds three spans per decision (data, model, harness).

The graph kernel has two edge layers, and the distinction is load-bearing. Execution edges (emits, handoff_to) are observed from the trace. Dependency edges (depends_on) are declared, inferred, or observed, and every edge records how it is known: the corpus and plan adapters declare or infer them, while the LangGraph capture path and the TouchRecorder read them off a real run as observed channel touches. PRE and POST both run over this same typed graph; audit() is the ergonomi

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yzhao062
• Source: yzhao062/auditable
• License: Apache-2.0
• Homepage: https://auditable-ai.readthedocs.io/

Install and usage instructions live in the source repository linked above.