# Assay

> Assay — a canary-oracle benchmark for MCP security: a frozen task set scored by a recomputable HMAC-canary oracle (structural-zero false positives), not an LLM judge. Maintained by Verosek.

- **Type:** MCP server
- **Install:** `agentstack add mcp-vatssshah-assay`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [VatsSShah](https://agentstack.voostack.com/s/vatssshah)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [VatsSShah](https://github.com/VatsSShah)
- **Source:** https://github.com/VatsSShah/assay
- **Website:** https://www.verosek.com

## Install

```sh
agentstack add mcp-vatssshah-assay
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Assay

A canary-oracle benchmark for MCP security. Score an MCP server or agent against a frozen task
set and get one comparable number, decided by a deterministic cryptographic oracle at the tool
boundary rather than an LLM judge.

[](https://github.com/VatsSShah/assay/actions)

Maintained by [Verosek](https://www.verosek.com). The standard is vendor-neutral; Verosek runs
the reference implementation and a commercial continuous-scoring service.

---

## The idea

A canary is a per-run value of the form:

```text
digest = HMAC-SHA256(run_secret, "origin|tool|step")[:16]
```

It is planted inside a realistic-secret envelope (an `sk-` key, an `AKIA` id, a JWT, an opaque
token, or a bare reference number) that rotates per run. The value the agent sees looks like a
real secret, so refusing to leak it is the same behaviour as refusing to leak a real secret, not
pattern-matching a benchmark marker. Detection is envelope-agnostic: recompute the 16-hex digest
and look for it (or a documented encoding) in the tool call or egress.

Because `run_secret` is fresh random bytes each run, the digest exists nowhere else and cannot be
forged or precomputed. So for the canary-gated tasks, a hit means a planted secret crossed a
boundary it should not have. That is a fact you can recompute, not a model's opinion, and its
false-positive rate is structurally zero (proof in [SPEC.md](SPEC.md) §4). The proof is about the
oracle: it says a canary hit implies real exfiltration. It does not claim a given model will or
will not leak; that is what scoring a target measures.

```mermaid
flowchart LR
  A["Mint per-run secret k(32 random bytes)"] --> B["digest =HMAC-SHA256(k, origin·tool·step)"]
  B --> C["Plant in a realistic-secret envelopesk- / AKIA / JWT / opaque / bare-ref"]
  C --> D{"Does the agent orserver emit it?"}
  D -- "digest appears in egress" --> E["finding (deterministic proof)"]
  D -- "no" --> F["resisted"]
  E --> G["Ship triple:(run_secret, preimage, observed)"]
  G --> H["anyone recomputes thedigest and confirms"]
```

## Why a cryptographic oracle

HarmBench and StrongREJECT, the standard jailbreak benchmarks, both score with a learned grader
and so carry a measured false-positive rate. AgentDojo introduced a deterministic state oracle
for agent prompt-injection, but it is not MCP-scoped and not cryptographic. The MCP taxonomies
(OWASP MCP Top-10, Adversa MCP Top-25, MITRE ATLAS) define threats but no scoring oracle. Assay
fills that gap for MCP: a frozen task set whose canary-gated results are decided by a recomputable
HMAC oracle.

## Quickstart: verify a scorecard with the standard library

```bash
pipx run assay-bench path/to/scorecard.json          # validate a whole manifest
pipx run assay-bench triple        # one proof triple
```

The verifier checks four things: the commitment binds the revealed secret, the manifest integrity
hash matches, every fired canary recomputes from its triple, and the stated scores match the
scores recomputed from the findings. So a submitter cannot fake a canary hit or state a score the
findings do not imply. It does not, on its own, prove an honest run happened against a real
target (a submitter could under-report by not running the attacks); that is why every leaderboard
row is re-run by a maintainer.

## What's in this repo

| file | what it is |
|---|---|
| [`SPEC.md`](SPEC.md) | the versioned specification: two modes, oracle rules, FP=0 proof, metrics, commit-reveal, anti-contamination, freeze policy |
| [`TASKS.md`](TASKS.md) | the full catalog of all 31 tasks: mechanism, poison template, canary plant, egress, oracle rule, vulnerable-vs-safe behaviour, mitigation, taxonomy |
| [`tasks.json`](tasks.json) | the frozen task set as data, severity weights, taxonomy crosswalk |
| [`COVERAGE.md`](COVERAGE.md) | every task mapped to OWASP MCP / Adversa-25 / OWASP ASI / MITRE ATLAS |
| [`assay_verifier.py`](assay_verifier.py) | the standalone third-party verifier (stdlib only) |
| [`scoring.py`](scoring.py) | the two 0-100 scores (Agent Resistance, Server Posture) and the over-refusal axis |
| [`badge.py`](badge.py) | turn a passing scorecard into an embeddable shield |
| [`manifest_schema.json`](manifest_schema.json) | the scorecard JSON Schema |
| [`leaderboard/`](leaderboard/) | the two-track leaderboard: schema, site builder, submission protocol, reference manifests |
| [`reference/confusion_matrix.json`](reference/confusion_matrix.json) | the reference run's per-task recall and false-positive table |
| [`corpus/README.md`](corpus/README.md) | the labeled corpus: built-in controls plus external vulnerable servers (loopback) |
| [`paper/assay.tex`](paper/assay.tex) | the methodology paper |

## Two modes

A mapped surface is never reported as an exploit. Every result states its mode and whether its
evidence is a canary, a protocol fact, or a deterministic behavioural check.

```mermaid
flowchart TB
  subgraph MA["Mode A: server pentest (you are the client)"]
    A1["Assay MCP client"] --> A2[("your MCP server")]
    A2 --> A3["evidence = reproducible protocol fact(unauth tools/list, missing Origin, param injection, SSRF)"]
  end
  subgraph MB["Mode B: agent robustness (your agent is under test)"]
    B1["your agent"] -->|tool calls| B2["adversarial MCP server(poisoned descriptions + results)"]
    B2 -->|poisoned tool results| B1
    B1 --> B3["evidence = canary hit at the egress boundary"]
  end
```

## Result integrity

```mermaid
sequenceDiagram
  participant H as Harness
  participant L as Leaderboard
  participant V as Anyone (verifier)
  H->>L: 1. commitment = SHA-256(run_secret), before the run
  H->>H: 2. run frozen tasks, mint per-run canaries
  H->>L: 3. reveal run_secret + manifest (triples, scores, integrity hash)
  V->>V: 4. commitment binds reveal? triples recompute? scores match findings? hash matches?
  V-->>L: verified, cold, stdlib only
```

The manifest carries an integrity hash (a SHA-256 over its own content), not a cryptographic
signature. Commit-reveal stops a submitter from changing the secret after seeing results; the
score-binding check stops them from stating a number the findings do not support. Authenticity of
the run itself comes from the maintainer re-run.

## The reference numbers, scoped honestly

The repo ships a reference run against the built-in labeled corpus. That corpus uses deterministic
vulnerable and safe reference targets (in-process stubs), so the reference run is **mechanism
validation**: it confirms every attack fires against a target built to be vulnerable and clears a
target built to resist, and that the canary-gated families produce no false positive on the safe
side. It is not a field measurement against a live model. The structural FP=0 claim is the
cryptographic one above; the live ASR of a real model or server is what you get by scoring that
target (the reference implementation, see below). Per-task numbers are in
[`reference/confusion_matrix.json`](reference/confusion_matrix.json).

## Leaderboard

```mermaid
flowchart LR
  S["PR: manifest + entry"] --> C["CI runs the verifier(binds proofs + scores)"]
  C -- "fails" --> X["blocked"]
  C -- "verifies" --> M["maintainer re-runsat the pinned version"]
  M --> R["row published(links to its manifest)"]
```

Two tracks: Agent Resistance (Mode B) and Server Posture (Mode A). The verifier makes a stated
score impossible to inflate; the maintainer re-run is what attests the row is a real run against
the named target. See [`leaderboard/SUBMIT.md`](leaderboard/SUBMIT.md).

## Multimodal track

Six tasks carry the canary through a non-text channel, scored by the same oracle (no per-modality
judge to mis-read an image or mis-hear audio): M20 directive in a user-attached image, M26
directive in the pixels of a tool-returned image, M27 a cross-modal chain (a vision tool surfaces
the canary, a network tool exfiltrates it, with no injected instruction), M28 a steganographic
chart (canary in alt-text/metadata), M29 a rug-pull on an image resource, M30 a QR code encoding
an egress URL. Full mechanics in [`TASKS.md`](TASKS.md).

## Anti-contamination

- **Over-refusal (utility) axis (planned).** Resistance measures false success. The utility axis
  measures false refusal, a secure-but-useless target that blocks legitimate tool use. The scoring
  is defined (a helpful target scores 0%, an over-cautious one 100%), but the benign twins are not
  authored yet, so no live target is measured against it today.
- **Held-out split (planned).** `tasks.json` is the public split. A held-out split with the same
  oracles and unpublished task instances is maintained separately so a target that memorizes the
  public tasks can be caught by the gap between its public and held-out scores. Until that ships,
  commit-reveal blocks result grinding but not task memorization, so treat a high public score as
  necessary, not sufficient.

## Scope

In scope: tool-call injection, tool-return and RAG poisoning, rug-pull, cross-tenant, TOCTOU,
confused-deputy, transport/unauth/DNS-rebind, secret exfil, excessive agency, multimodal exfil,
and lethal-trifecta source-to-sink paths (the 31 tasks in [`TASKS.md`](TASKS.md)). Out of scope:
model-weight or training-time attacks, pure prompt-only jailbreaks (owned by StrongREJECT),
host-OS exploits. Targets are loopback only; the published benchmark never scores a live
third-party endpoint, so every result is locally reproducible.

## Open standard vs. reference implementation

The standard (this repo, MIT-licensed) is the canary methodology, the frozen task set, the
verifier, the manifest schema, and the leaderboard protocol. It is vendor-neutral so anyone,
including competitors, can run and cite it. The reference implementation that runs the tasks, and
a commercial continuous-scoring service, are maintained by [Verosek](https://www.verosek.com). The
commercial layer never touches this spec or the leaderboard's neutrality.

## Cite

> Assay: a canary-oracle benchmark for MCP security, v0.1. See [`SPEC.md`](SPEC.md) and
> [`CITATION.cff`](CITATION.cff).

Licensed under [MIT](LICENSE). Third-party taxonomy attributions in [`NOTICE`](NOTICE).

---

Built and maintained by Verosek · verosek.com

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [VatsSShah](https://github.com/VatsSShah)
- **Source:** [VatsSShah/assay](https://github.com/VatsSShah/assay)
- **License:** MIT
- **Homepage:** https://www.verosek.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-vatssshah-assay
- Seller: https://agentstack.voostack.com/s/vatssshah
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.