# Harness Audit

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-contextosai-skills-harness-audit`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [contextosai](https://agentstack.voostack.com/s/contextosai)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [contextosai](https://github.com/contextosai)
- **Source:** https://github.com/contextosai/skills/tree/main/skills/harness-audit

## Install

```sh
agentstack add skill-contextosai-skills-harness-audit
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Agent Harness Audit

You are running a **production readiness audit** on an AI agent's harness. The
governing rule is the one thing that makes this audit worth anything:

> **No artifact, no pass.** Judge the runtime record, not the architecture
> diagram or the README. A control that is only *described* — in prose, in a
> design doc, in the system prompt — is a **Fail**. A harness is evidence, not
> confidence. And prefer evidence from channels the agent cannot author: a
> harness-emitted tool-call / resource-access / message record outranks anything
> the model self-reports, because a self-report is gameable.

Your job is to find the evidence in the target repository (and run traces, if
available), score each control against that evidence, and hand back an action
plan the team can execute. You are not here to be reassured; you are here to
find what will break in production.

## Inputs and audit scope

- **Target**: the path to the agent's repository (default: the current working
  directory). If the user names a different path, audit that.
- **Run traces** (optional): paths or a directory of real run logs/traces. If
  present, inspect at least two — one ordinary successful run and one **boundary
  run** that hit a policy denial, tool error, approval gate, evaluator failure,
  escalation, rollback, or fallback. The happy path shows what the harness does
  when nothing is stressed; the boundary path shows whether the harness exists
  when it matters.
- **Declared maturity band** (optional): ask the user, or infer and state your
  assumption. The band determines which failures block launch (see Maturity).

Do not block on missing optional inputs. Default the target to the current
directory, infer the intended band from real deployment signals, and label the
result as an assumption. Record exclusions such as external policy services,
managed gateways, unavailable traces, generated code, and inaccessible CI.
An inaccessible surface is **Unverified**, not evidence of absence; map it to
Fail for launch scoring and name the access needed to verify it.

## Protocol

Work in this order. Do not skip the prescan, and do not score a control Pass
without a concrete artifact reference.

### 1. Orient (≈2 min)

Identify the substrate before judging it. Read `package.json` /
`pyproject.toml` / lockfiles / `README` to determine the framework (LangGraph,
OpenAI Agents SDK, Google ADK, Semantic Kernel, CrewAI, Mastra, or custom),
the language, and where the agent's entry point, tools, prompts, policies,
evals, and telemetry live. State what you found in one short paragraph.

Also determine whether this is **single-agent or multi-agent** (handoffs,
sub-agents, planner/worker roles, a graph with multiple agent nodes). If
multi-agent, controls #42 (outbound disclosure) and #43 (communication policy)
are in scope and the inter-agent message channel is a first-class audit surface
— coordination expands the risk surface. If single-agent, mark #43 N/A and say so.

Build a short **surface map** before scoring: model entry points, context and
memory paths, every tool dispatcher and side-effecting adapter, identity/policy
boundaries, inter-agent channels, output boundary, trace sink, and eval/release
path. Scope each control only after this map exists. Do not mark a generally
applicable control N/A merely because its feature is absent; absence is Fail.

### 2. Prescan for evidence (≈3 min)

Run the deterministic prescan to get a fast map of candidate evidence:

```bash
node "$SKILL_DIR/scripts/prescan.mjs" 
```

(If Node is unavailable or the script errors, fall back to `rg`/`grep`
manually using the search hints in the checklist.) The prescan classifies hits
by artifact type and only *locates*
candidate evidence — it never decides Pass/Fail. **You** must open each hit and
verify it actually enforces the control at the right boundary. A keyword match
is not a control. Prescan deliberately skips dotenv files and redacts likely
secrets; never paste secret-bearing source or trace payloads into the report.

### 3. Trace two representative paths

If traces exist, select them deliberately:

1. Choose one ordinary run that exercises the primary intent and tools.
2. Choose one boundary run that exercises the highest-risk touched surface.
3. Reconcile trace events against code/config versions. A trace from an unknown
   release can demonstrate behavior but cannot prove the current release.
4. Record trace IDs and timestamps; redact user content, credentials, and
   sensitive tool arguments.

If no boundary trace exists, do not manufacture one or execute risky tools.
Score trace-dependent guarantees no higher than Partial from code/tests alone,
and require a runtime re-audit. A fixture may prove deterministic logic; it
does not prove production wiring.

### 4. Score the 44 controls

Open `reference/checklist.md`. It lists every control with: the audit question,
minimum pass evidence, the immediate fail signal, severity, where to look, and
the ContextOS plane/doc that owns the remediation. For each control:

- **Pass** — a script, trace, manifest, config, test, or record proves the
  control for a real path. Cite the artifact as `path:line`.
- **Partial** — the control exists but coverage is incomplete, manual, delayed,
  undocumented, or enforced at the wrong boundary. Cite what exists and what's
  missing.
- **Fail** — absent, unenforced, unverifiable, or prose-only. Say where you
  looked.

Use the evidence-strength rules at the top of the checklist. For a Pass, cite
the enforcement point and either a boundary test/trace or a declarative runtime
binding that makes bypass impossible. Configuration alone does not prove it is
loaded; a helper alone does not prove it is called. When evidence conflicts,
use the weaker score and explain the conflict.

Apply the **five-minute rule**: if you cannot find the evidence within ~5
minutes of searching, score it **Fail**. A control that can't be found under
pressure won't protect the system under pressure. Note it as "not found within
budget" rather than asserting it doesn't exist — but it still scores Fail.

Severity (`P0`/`P1`/`P2`) comes from the checklist and is **independent** of the
pass state — a P0 control that is `Partial` is still a launch blocker.

### 5. Check arithmetic and decide

- Roll the 44 control scores into the **eight outcome groups** (context-aware,
  policy-governed, tool-controlled, validated, observable, reversible,
  measurable, continuously improving).
- Determine the **maturity band** the evidence actually supports (see below),
  and compare it to the band the team *claims* to operate at. The gap between
  claimed and evidenced maturity is the headline finding.
- Apply the launch decision: **any P0 Fail/Partial blocks production** unless
  the agent is fully read-only, isolated from real users, and confined to a
  controlled beta.
- Reconcile Pass + Partial + Fail + N/A to 44. Use N/A only where the checklist
  explicitly permits it (#43 for a confirmed single-agent harness). Rollups
  must reconcile to control detail. Do not invent a weighted aggregate score;
  worst-severity gaps and evidence quality matter more than a percentage.

### 6. Emit the report

Produce the scorecard using `reference/scorecard-template.md`. Include audit
limitations and verdict confidence. The report MUST
end with a **fix queue**: every gap gets an owner placeholder, severity, the
concrete fix, the *expected evidence* that would flip it to Pass, and the
ContextOS plane/doc to read. Order the queue by the dependency chain, not by
control number — fix the most load-bearing failure first (a broken context
compiler undermines grounding, validation, observability, and replay; a missing
policy engine undermines tool control, approval, and privacy). Tell the user to
fix one well and re-run the audit. Do not hand back dozens of parallel workstreams.

## Maturity bands

The band is not a label; it determines which failures block launch.

| Band | Appropriate use | Required controls | Not allowed |
|---|---|---|---|
| **Prototype** | Internal exploration, no real side effects, synthetic/low-risk data | Agent charter, basic eval set, trace capture, tool sandbox | Real users, PII, money movement, durable memory |
| **Controlled beta** | Limited users, explicit supervision, compensating controls | P0 controls for touched surfaces, approval gates, offline evals, trace review, fix queue | Direct high-risk execution without a human gate |
| **Production** | Real users, real tools, monitored release lifecycle | Full P0 pass, P1 gaps owned, live validation, replay, rollback, incident playbook | Unversioned prompt/model/tool changes |
| **Regulated / high-risk** | Regulated data, money movement, legal/health/security, destructive actions | Full P0/P1 pass, red-team audit, retention policy, evidence retention, formal release governance | Informal approval, undocumented memory, non-replayable action |

## Discipline reminders

- **Equivalent evidence, not ContextOS terminology.** The target may use any
  framework. You are looking for the *control*, however it's named — a Pydantic
  guardrail, a LangGraph interrupt, an OPA policy, an OTEL span. The checklist's
  ContextOS mapping is for the remediation pointer, not a naming requirement.
- **Boundary over happy path.** If you only inspect a successful run, you have
  audited the demo, not the harness.
- **Completion is not safety.** A run that finishes the task is evidence of
  nothing about whether it stayed in bounds — task completion and safety are
  routinely misaligned, and a higher completion rate often means *more* boundary
  crossings, not fewer. A run that completed the task while crossing a boundary
  is a **Fail**, not a partial credit.
- **Watch the object, not just the tool.** The most common live violation is the
  *right tool on the wrong object* — a valid refund/lookup/file call applied to
  an out-of-scope customer, record, or path. Schema validation passes; the
  boundary is still crossed. Scrutinize resource-scope binding (#41) and the
  argument-level axis of trajectory evals (#27) accordingly.
- **Be specific and falsifiable.** Every Pass cites `path:line`. Every Fail says
  where you looked. Never soften a P0 Fail into a suggestion.
- **Don't fabricate.** If traces weren't provided, say the trace-dependent
  controls were scored from code only and flag them for a runtime re-audit.
- **Separate existence, wiring, and behavior.** A dependency proves capability,
  configuration proves intent, and runtime wiring plus a boundary test/trace
  proves enforcement. Score the strongest complete chain, not its strongest
  isolated artifact.
- **Sample honestly.** Do not extrapolate one protected tool, route, tenant, or
  role to all others. Partial is the correct score for incomplete coverage.

## What this audit is grounded in

This skill is the runnable form of the ContextOS eight-property harness audit:
https://contextosai.com/blog/eight-property-harness-audit — 44 controls grouped
into eight outcomes, evidence required for every pass. The eight properties are
defined at https://contextosai.com/docs/foundations/harness-engineering.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [contextosai](https://github.com/contextosai)
- **Source:** [contextosai/skills](https://github.com/contextosai/skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-contextosai-skills-harness-audit
- Seller: https://agentstack.voostack.com/s/contextosai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.