# Mcp Test Harness

> a Testing framework for MCP servers

- **Type:** MCP server
- **Install:** `agentstack add mcp-vaquarkhan-mcp-test-harness`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vaquarkhan](https://agentstack.voostack.com/s/vaquarkhan)
- **Installs:** 0
- **Category:** [Integrations](https://agentstack.voostack.com/c/integrations)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [vaquarkhan](https://github.com/vaquarkhan)
- **Source:** https://github.com/vaquarkhan/mcp-test-harness
- **Website:** https://vaquarkhan.github.io/mcp-test-harness/

## Install

```sh
agentstack add mcp-vaquarkhan-mcp-test-harness
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# MCP Test Harness

[](https://pypi.org/project/mcp-test-harness/)
[](https://pepy.tech/project/mcp-test-harness)
[](https://pypi.org/project/mcp-test-harness/)
[](https://github.com/vaquarkhan/mcp-test-harness/actions/workflows/validate.yml)
[](LICENSE)
[]()
[](CONTRIBUTING.md#develop--test)
[](https://github.com/vaquarkhan/mcp-test-harness/pkgs/container/mcp-test-harness)
[](https://vaquarkhan.github.io/mcp-test-harness/)

**Latest Release** - install from **[PyPI](https://pypi.org/project/mcp-test-harness/)** (`pip install mcp-test-harness`) or use the **OCI image** on **GitHub Container Registry:** **`ghcr.io/vaquarkhan/mcp-test-harness`**. Tags: **`latest`** and **`3.0.9`** (runtime, `mcp-test` entrypoint), **`dev`** and **`3.0.9-dev`** (pytest + dev extras). [Browse tags on GHCR](https://github.com/vaquarkhan/mcp-test-harness/pkgs/container/mcp-test-harness/versions) · `docker run --rm ghcr.io/vaquarkhan/mcp-test-harness:latest --version` · [docs/DOCKER.md](docs/DOCKER.md) · [docs/RELEASING.md](docs/RELEASING.md)

Author: [Vaquar Khan](https://github.com/vaquarkhan) · **License:** [MIT](LICENSE) ([NOTICE](NOTICE)) · **Cite:** [CITATION.cff](CITATION.cff) · **Sponsors:** [SPONSORS.md](SPONSORS.md)

---

## For C-suite & directors

**What it is:** a CI gate for [MCP](https://modelcontextprotocol.io/) servers - the connectors that let AI agents call tools, data, and APIs. Teams write deterministic tests once; every pull request proves the server still works before it ships.

**Why it matters**

| Business need | What the harness delivers |
|---------------|---------------------------|
| Ship AI features without silent breakage | Automated pass/fail on every PR - no manual Inspector click-through |
| Audit & governance evidence | JUnit / SARIF / HTML reports, conformance badges, security payload packs |
| Lower cost of quality | One tool for functional, regression, performance, and resiliency - not three vendors |
| Trust the gate | **840+** self-tests, **100%** library coverage, and e2e dogfood in CI |

**Not an LLM eval product.** It does not score model answers. It proves *your MCP server* is correct, fast enough, and hardened enough for production. Pair with runtime security ([MCP-Bastion](https://github.com/vaquarkhan/MCP-Bastion)) and IDE config scanning ([mcp-shark](https://github.com/mcp-shark/mcp-shark)). Positioning: [docs/POSITIONING.md](docs/POSITIONING.md) · comparison: [docs/COMPARISON.md](docs/COMPARISON.md).

---

## For architects & tech leads

**Role in the stack:** deterministic, code-first test automation between local MCP Inspector exploration and production. Fits stdio / SSE / HTTP transports; plugs into GitHub Actions (Marketplace Action), JUnit consumers, and Code Scanning (SARIF).

**Architecture (one run)**

```text
mcp-test CLI → config → discover test_*.py → schedule (optional parallel)
  → real MCP session (lifecycle + transport) → assertions → reports
```

| Concern | Approach |
|---------|----------|
| Correctness | Protocol-aware fixtures (`mcp_server`), schema checks, snapshots |
| Performance | `assert_latency` / `assert_throughput` / `assert_stateless_throughput`, baselines, SLO params |
| Security / resiliency | Payload packs, chaos faults, experiment catalog + scorecard (RFC-005) |
| Conformance signal | Levels + README badge via `mcp-test try` / `conformance` (RFC-002); **stateless** via `mcp-test conformance stateless` (RFC-006) |
| Adoption speed | `init`, `generate`, `record` (RFC-001) - live session → suite |
| Extensibility | Plugins (assertions, fixtures, reporters, transports) |

Deep dives: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) · [docs/DECISIONS.md](docs/DECISIONS.md) · [docs/ENTERPRISE_GOVERNANCE.md](docs/ENTERPRISE_GOVERNANCE.md) · RFCs under [docs/design/](docs/design/).

---

## For developers

**Fastest path**

```bash
pip install mcp-test-harness
mcp-test init --server-command "python your_server.py"
# edit tests/test_*.py - then:
mcp-test --config mcp-test.yaml
```

Zero-config probe: `mcp-test try --server-command "…"`. CI: Marketplace Action [`mcp-test-harness`](https://github.com/marketplace/actions/mcp-test-harness). Docs hub: [docs/QUICK_START.md](docs/QUICK_START.md) → [docs/DEVELOPER_GUIDE.md](docs/DEVELOPER_GUIDE.md) → [docs/CI_AND_REPORTS.md](docs/CI_AND_REPORTS.md).

MCP Test Harness is a pytest-style framework for MCP servers: the `mcp-test` CLI discovers, runs, and reports tests automatically - replacing much of the manual validation you might do in the MCP Inspector. **Repository:** [github.com/vaquarkhan/mcp-test-harness](https://github.com/vaquarkhan/mcp-test-harness).

> **Documentation:** [QUICK_START](docs/QUICK_START.md) · [DEVELOPER_GUIDE](docs/DEVELOPER_GUIDE.md) · [CI & reports](docs/CI_AND_REPORTS.md) · [performance](docs/PERFORMANCE.md) · [comparison](docs/COMPARISON.md) · [discovery checklist](docs/DISCOVERY.md). **Community:** Issues for bugs; PRs for docs and examples.

---

### What ships end-to-end

Everything below is **implemented, tested, and documented** in this repo (840+ tests, 100% lib coverage gate, e2e dogfood in CI). Version pins live only in [PyPI](https://pypi.org/project/mcp-test-harness/), [CHANGELOG.md](CHANGELOG.md), and install/Docker tags - not in this map.

| Area | Feature | Where to start |
|------|---------|----------------|
| **CLI** | **`mcp-test`** - discover, run, filter (`-k` / `-m`), parallel, watch, reports; subcommands `init`, `try`, `record`, `generate`, `experiment`, `conformance` | [CLI Reference](#cli-reference) · [Quick Start](#quick-start) |
| **Automation** | Scaffold (`init`), zero-config probe (`try`), live **`record`** / **`generate`**, `--watch` re-runs, config-driven CI gates | [Quick Start](#quick-start) · [Core Features](#core-features) |
| **Docker** | OCI image on **GHCR** (`ghcr.io/vaquarkhan/mcp-test-harness`) - runtime + `:dev`; local [`Dockerfile`](Dockerfile) | [Docker](#docker) · [docs/DOCKER.md](docs/DOCKER.md) |
| **Hooks** | **pre-commit** hook (`mcp-test-try`) + example config; plugin / discovery hooks for custom automation | [examples/pre-commit-config.yaml](examples/pre-commit-config.yaml) · [Plugins](#plugins) |
| **Diagnostics** | **MCP trace** - per-test JSON-RPC timeline in HTML/JSON reports; stdio pollution hints | [example_mcp_trace.md](examples/example_mcp_trace.md) |
| **Resiliency** | **Chaos testing** - `@marker(tags=["chaos"], chaos_faults=[...])` (delay, 503, truncate, schema drift) | [example_chaos_testing.md](examples/example_chaos_testing.md) |
| **Resiliency** | **Experiment catalog** - `mcp-test experiment run --suite core` with guardrails and scorecard (RFC-005) | [docs/design/RFC-005-resiliency-experiments.md](docs/design/RFC-005-resiliency-experiments.md) |
| **Conformance** | **Levels + badge** - `mcp-test try` / `conformance badge` (RFC-002); **stateless SEP-2575** via `mcp-test conformance stateless` (RFC-006); README shields.io seal | [docs/design/RFC-002-conformance-levels.md](docs/design/RFC-002-conformance-levels.md) · [RFC-006](docs/design/RFC-006-stateless-mcp.md) |
| **Productivity** | **`mcp-test record`** (RFC-001) + **`generate`** - live calls → suite + snapshots; schema drafts | [RFC-001](docs/design/RFC-001-record-to-suite.md) · [example_generate_scaffold.md](examples/example_generate_scaffold.md) |
| **Marketplace** | GitHub Action with conformance PR outputs | [Marketplace](https://github.com/marketplace/actions/mcp-test-harness) · [example_github_actions.md](examples/example_github_actions.md) |
| **Platform QA** | Tool **coverage map**, **unified portal** in HTML/JSON, **security payload** packs, **resiliency** assertions, **performance baselines**, **`assert_throughput`** / **`assert_stateless_throughput`** SLO params | [docs/POSITIONING.md](docs/POSITIONING.md) · [docs/SECURITY_TESTING.md](docs/SECURITY_TESTING.md) · [RFC-006](docs/design/RFC-006-stateless-mcp.md) |
| **CI / security** | **SARIF** export, OWASP MCP rule metadata, **PR summary** markdown + GitHub Action `pr-comment` | [docs/CI_AND_REPORTS.md](docs/CI_AND_REPORTS.md) |
| **Self-test** | **E2E dogfood** - pytest spawns real `mcp-test` CLI against bundled FastMCP fixtures; **100% coverage** enforced on every PR | [We eat our own dogfood](#we-eat-our-own-dogfood) |
| **Examples** | Runnable **platform-qa** demo pack (trace + chaos + generate) | [feature-demo/platform-qa](examples/feature-demo/platform-qa/README.md) |
| **Website** | Marketing site at [vaquarkhan.github.io/mcp-test-harness](https://vaquarkhan.github.io/mcp-test-harness/) (deploys `html/` via GitHub Actions) | [html/index.html](html/index.html) |

Full history: [CHANGELOG.md](CHANGELOG.md) · roadmap: [docs/ROADMAP.md](docs/ROADMAP.md).

---

## Visual guide

  

One command from your repo to a gated MCP test run - local or in CI.

**Developer journey**

  

`pip install` → `mcp-test init` → write `test_*.py` → `mcp-test` → merge with confidence.

**Three testing modes**

  

Functional correctness, snapshot regression, and SLO-style performance - not three separate tools.

  

Under the hood: config → discovery → scheduling → real MCP sessions → deterministic assertions → multi-format reports.

### All visual assets

Every diagram lives in [`docs/images/`](docs/images/). Quick reference:

| Image | What it shows |
|-------|----------------|
| [`hero-banner.png`](docs/images/hero-banner.png) | Project hero - pytest-style MCP testing |
| [`end-to-end-flow.png`](docs/images/end-to-end-flow.png) | Full pipeline: CLI → server → assertions → reports |
| [`developer-journey.png`](docs/images/developer-journey.png) | Install → init → write tests → run → CI |
| [`three-testing-modes.png`](docs/images/three-testing-modes.png) | Functional, regression, and performance pillars |
| [`architecture-flow.svg`](docs/images/architecture-flow.svg) | Internal modules and data path (vector) |
| [`stateless-dual-mode.svg`](docs/images/stateless-dual-mode.svg) | **Dual mode:** stateful session vs stateless HTTP (SEP-2575 / RFC-006) |
| [`mcp-testobarness-feature.png`](docs/images/mcp-testobarness-feature.png) | Core feature map |
| [`assertions-grid.png`](docs/images/assertions-grid.png) | Assertion library at a glance |
| [`report-formats.png`](docs/images/report-formats.png) | Console, JUnit, JSON, and HTML format overview (infographic) |
| [`html-dashboard.png`](docs/images/html-dashboard.png) | **Live HTML dashboard** - format previews (click to expand), stat cards, charts, filters, PDF/CSV export |
| [`transport-options.png`](docs/images/transport-options.png) | stdio, SSE, and HTTP transports |
| [`parallel-execution.png`](docs/images/parallel-execution.png) | Multi-worker scheduling and module grouping |
| [`ci-pipeline.png`](docs/images/ci-pipeline.png) | GitHub Actions PR gate workflow |
| [`docker-distribution.png`](docs/images/docker-distribution.png) | PyPI, GHCR container, and standalone binary |
| [`ecosystem-map.png`](docs/images/ecosystem-map.png) | Position vs Inspector, conformance, evals, Bastion |
| [`harness-bastion-pairing.png`](docs/images/harness-bastion-pairing.png) | Test in CI, secure in production |
| [`testherness.png`](docs/images/testherness.png) | Quick-start overview |
| [`dogfood-e2e.svg`](docs/images/dogfood-e2e.svg) | **Dogfood:** pytest runs `mcp-test` against bundled MCP fixtures (840+ tests, 100% lib coverage) |

---

## We eat our own dogfood

A test harness should prove itself. This repo runs **840+ pytest cases** with a **100% line coverage gate** on `src/mcp_test_harness`, plus **end-to-end dogfood** tests that spawn the real `mcp-test` CLI against a bundled FastMCP server:

  

```bash
# Same gate CI enforces on every PR
coverage run -m pytest tests/ --ignore=tests/test_workspace.py -q
coverage report --fail-under=100

# E2E dogfood only
python -m pytest tests/test_harness_dogfood_e2e.py -m e2e -v
```

Fixtures live under [`tests/fixtures/`](tests/fixtures/) (`minimal_mcp_server.py`, `harness_self_test/`). See [docs/DEVELOPER.md](docs/DEVELOPER.md#end-to-end-dogfood) and [CONTRIBUTING.md](CONTRIBUTING.md#develop--test).

---

## Why teams adopt this

  

MCP Test Harness is the **deterministic, CI-native gate** for MCP servers - one run proves correctness, speed, and security baselines without an LLM in the loop. Full positioning: **[docs/POSITIONING.md](docs/POSITIONING.md)**.

### Four differentiators

| # | Differentiator | What you get |
|---|----------------|--------------|
| **1** | **Tri-modal CI architecture** | Functional + regression + performance in one `mcp-test` run - `assert_latency` (p95/p99 + warmup), `assert_throughput` (`min_rps`, `max_p99_ms`, `max_error_rate`), `assert_tool_idempotent`, JSON **performance baselines** |
| **2** | **Protocol-aware fixtures** | `mcp_server` / `mcp_server_session` - spawn, handshake, inject session, teardown; parallel module grouping |
| **3** | **Code-first workflows** | Multi-step agent flows in async Python (anti-DSL); `@marker(tags=[...])` for smoke/security/perf |
| **4** | **Unified portal** | One HTML/JSON artifact: category scores, **tool coverage map** (advertised vs tested vs missing auth), security/resiliency packs |

### Three testing modes in one tool

- **Functional:** `assert_tool_call`, `assert_resource_read`, schema validation
- **Regression:** `assert_snapshot`, `assert_tool_idempotent`
- **Performance:** `assert_latency`, `assert_throughput`, `assert_latency_within_baseline`

Pair with **[mcp-shark](https://github.com/mcp-shark/mcp-shark)** for IDE config security and **[MCP-Bastion](https://github.com/vaquarkhan/MCP-Bastion)** for runtime protection - see [docs/COMPARISON.md](docs/COMPARISON.md).

MCP Test Harness supports **Responsible AI** and governance programs: schema contract checks, security payload packs (`@marker(tags=["security"])`), coverage gaps in reports, and EU AI Act demo packs.

## Documentation

**Hub (table of all guides + suggested reading order):** [docs/README.md](docs/README.md)

| Document | Contents |
|----------|----------|
| [docs/QUICK_START.md](docs/QUICK_START.md) | **Fastest path** - install, `mcp-test init`, run |
| [docs/DEVELOPER_GUIDE.md](docs/DEVELOPER_GUIDE.md) | **Canonical reference** - setup, config, stdio/parallel/validation, assertions, reporting |
| [docs/CI_AND_REPORTS.md](docs/CI_AND_REPORTS.md) | **CI, JUnit, JSON, HTML** - do you need to *publish* test reports? (usually: no) |
| [docs/POSITIONING.md](docs/POSITIONING.md) | **Why we're different** - four differentiators, enterprise governance, mcp-shark pairing |
| [docs/PERFORMANCE_TESTING_STRATEGY.md](docs/PERFORMANCE_TESTING_STRATEGY.md) | **Product pitch** - why MCP performance testing belongs in the harness; roadmap and scope |
| [docs/ROADMAP.md](docs/ROADMAP.md) | **Roadmap** - now/next/later plan plus **prioritized value bets** (unified reports, security packs, resiliency, coverage map, SLO load testing) |
| [docs/SECURITY_TESTING.md](docs/SECURITY_TESTING.md) | **Security testing** - MCP-aware security assertions and CI guidance |
| [docs/CONTRACT_AND_COMPAT.md](docs/CONTRACT_AND_COMPAT.md) | **Contracts & compatibility** - drift protection and protocol/client matrix strategy |
| [docs/ENTERPRISE_GOVERNANCE.md](docs/ENTERPRISE_GOVERNANCE.md) | **Enterprise** - audit/policy/tenant governance guidance (including EU AI Act evidence mapping notes) |
| [docs/PLUGIN_REGISTRY.md](docs/PLUGIN_REGISTRY.md) | **Plugin registry** - extension catalog and integration categories |
| [docs/TUTORIAL.md](docs/TUTORIAL.md) | Step-by-step tutorial |
| [docs/DECISIONS.md](docs/DECISIONS.md) | Architecture and product decisions |
| [docs/IMPLEMENTATION_CHECKLIST.md](docs/IMPLEMENTATION_CHECKLIST.md) | Maintainer: features vs. code locations |
| [docs/COMPARISON.md](docs/COMPARISON.md) | **Ecosystem** - where this harness fits alongside conformance/eval/benchmark categories |
| [docs/LLM_TEST_GENERATION.md](docs/LLM_TEST_GENERATION.md) | **LLM + tests** - draft-with-review: good; **auto** trusted in CI: bad

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vaquarkhan](https://github.com/vaquarkhan)
- **Source:** [vaquarkhan/mcp-test-harness](https://github.com/vaquarkhan/mcp-test-harness)
- **License:** MIT
- **Homepage:** https://vaquarkhan.github.io/mcp-test-harness/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-vaquarkhan-mcp-test-harness
- Seller: https://agentstack.voostack.com/s/vaquarkhan
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.