Mcp Test Harness

MCP Test Harness

[](https://pypi.org/project/mcp-test-harness/) [](https://pepy.tech/project/mcp-test-harness) [](https://pypi.org/project/mcp-test-harness/) [](https://github.com/vaquarkhan/mcp-test-harness/actions/workflows/validate.yml) [](LICENSE) []() [](CONTRIBUTING.md#develop--test) [](https://github.com/vaquarkhan/mcp-test-harness/pkgs/container/mcp-test-harness) [](https://vaquarkhan.github.io/mcp-test-harness/)

Latest Release - install from PyPI (pip install mcp-test-harness) or use the OCI image on GitHub Container Registry: ghcr.io/vaquarkhan/mcp-test-harness. Tags: latest and 3.0.9 (runtime, mcp-test entrypoint), dev and 3.0.9-dev (pytest + dev extras). Browse tags on GHCR · docker run --rm ghcr.io/vaquarkhan/mcp-test-harness:latest --version · [docs/DOCKER.md](docs/DOCKER.md) · [docs/RELEASING.md](docs/RELEASING.md)

Author: Vaquar Khan · License: [MIT](LICENSE) ([NOTICE](NOTICE)) · Cite: [CITATION.cff](CITATION.cff) · Sponsors: [SPONSORS.md](SPONSORS.md)

For C-suite & directors

What it is: a CI gate for MCP servers - the connectors that let AI agents call tools, data, and APIs. Teams write deterministic tests once; every pull request proves the server still works before it ships.

Why it matters

| Business need | What the harness delivers | |---------------|---------------------------| | Ship AI features without silent breakage | Automated pass/fail on every PR - no manual Inspector click-through | | Audit & governance evidence | JUnit / SARIF / HTML reports, conformance badges, security payload packs | | Lower cost of quality | One tool for functional, regression, performance, and resiliency - not three vendors | | Trust the gate | 840+ self-tests, 100% library coverage, and e2e dogfood in CI |

Not an LLM eval product. It does not score model answers. It proves your MCP server is correct, fast enough, and hardened enough for production. Pair with runtime security (MCP-Bastion) and IDE config scanning (mcp-shark). Positioning: [docs/POSITIONING.md](docs/POSITIONING.md) · comparison: [docs/COMPARISON.md](docs/COMPARISON.md).

For architects & tech leads

Role in the stack: deterministic, code-first test automation between local MCP Inspector exploration and production. Fits stdio / SSE / HTTP transports; plugs into GitHub Actions (Marketplace Action), JUnit consumers, and Code Scanning (SARIF).

Architecture (one run)

mcp-test CLI → config → discover test_*.py → schedule (optional parallel)
  → real MCP session (lifecycle + transport) → assertions → reports

| Concern | Approach | |---------|----------| | Correctness | Protocol-aware fixtures (mcp_server), schema checks, snapshots | | Performance | assert_latency / assert_throughput / assert_stateless_throughput, baselines, SLO params | | Security / resiliency | Payload packs, chaos faults, experiment catalog + scorecard (RFC-005) | | Conformance signal | Levels + README badge via mcp-test try / conformance (RFC-002); stateless via mcp-test conformance stateless (RFC-006) | | Adoption speed | init, generate, record (RFC-001) - live session → suite | | Extensibility | Plugins (assertions, fixtures, reporters, transports) |

Deep dives: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) · [docs/DECISIONS.md](docs/DECISIONS.md) · [docs/ENTERPRISEGOVERNANCE.md](docs/ENTERPRISEGOVERNANCE.md) · RFCs under [docs/design/](docs/design/).

For developers

Fastest path

pip install mcp-test-harness
mcp-test init --server-command "python your_server.py"
# edit tests/test_*.py - then:
mcp-test --config mcp-test.yaml

Zero-config probe: mcp-test try --server-command "…". CI: Marketplace Action mcp-test-harness. Docs hub: [docs/QUICKSTART.md](docs/QUICKSTART.md) → [docs/DEVELOPERGUIDE.md](docs/DEVELOPERGUIDE.md) → [docs/CIANDREPORTS.md](docs/CIAND_REPORTS.md).

MCP Test Harness is a pytest-style framework for MCP servers: the mcp-test CLI discovers, runs, and reports tests automatically - replacing much of the manual validation you might do in the MCP Inspector. Repository: github.com/vaquarkhan/mcp-test-harness.

> Documentation: [QUICKSTART](docs/QUICKSTART.md) · [DEVELOPERGUIDE](docs/DEVELOPERGUIDE.md) · [CI & reports](docs/CIANDREPORTS.md) · [performance](docs/PERFORMANCE.md) · [comparison](docs/COMPARISON.md) · [discovery checklist](docs/DISCOVERY.md). Community: Issues for bugs; PRs for docs and examples.

What ships end-to-end

Everything below is implemented, tested, and documented in this repo (840+ tests, 100% lib coverage gate, e2e dogfood in CI). Version pins live only in PyPI, [CHANGELOG.md](CHANGELOG.md), and install/Docker tags - not in this map.

| Area | Feature | Where to start | |------|---------|----------------| | CLI | mcp-test - discover, run, filter (-k / -m), parallel, watch, reports; subcommands init, try, record, generate, experiment, conformance | [CLI Reference](#cli-reference) · [Quick Start](#quick-start) | | Automation | Scaffold (init), zero-config probe (try), live record / generate, --watch re-runs, config-driven CI gates | [Quick Start](#quick-start) · [Core Features](#core-features) | | Docker | OCI image on GHCR (ghcr.io/vaquarkhan/mcp-test-harness) - runtime + :dev; local [Dockerfile](Dockerfile) | [Docker](#docker) · [docs/DOCKER.md](docs/DOCKER.md) | | Hooks | pre-commit hook (mcp-test-try) + example config; plugin / discovery hooks for custom automation | [examples/pre-commit-config.yaml](examples/pre-commit-config.yaml) · [Plugins](#plugins) | | Diagnostics | MCP trace - per-test JSON-RPC timeline in HTML/JSON reports; stdio pollution hints | [examplemcptrace.md](examples/examplemcptrace.md) | | Resiliency | Chaos testing - @marker(tags=["chaos"], chaos_faults=[...]) (delay, 503, truncate, schema drift) | [examplechaostesting.md](examples/examplechaostesting.md) | | Resiliency | Experiment catalog - mcp-test experiment run --suite core with guardrails and scorecard (RFC-005) | [docs/design/RFC-005-resiliency-experiments.md](docs/design/RFC-005-resiliency-experiments.md) | | Conformance | Levels + badge - mcp-test try / conformance badge (RFC-002); stateless SEP-2575 via mcp-test conformance stateless (RFC-006); README shields.io seal | [docs/design/RFC-002-conformance-levels.md](docs/design/RFC-002-conformance-levels.md) · [RFC-006](docs/design/RFC-006-stateless-mcp.md) | | Productivity | mcp-test record (RFC-001) + generate - live calls → suite + snapshots; schema drafts | [RFC-001](docs/design/RFC-001-record-to-suite.md) · [examplegeneratescaffold.md](examples/examplegeneratescaffold.md) | | Marketplace | GitHub Action with conformance PR outputs | Marketplace · [examplegithubactions.md](examples/examplegithubactions.md) | | Platform QA | Tool coverage map, unified portal in HTML/JSON, security payload packs, resiliency assertions, performance baselines, assert_throughput / assert_stateless_throughput SLO params | [docs/POSITIONING.md](docs/POSITIONING.md) · [docs/SECURITYTESTING.md](docs/SECURITYTESTING.md) · [RFC-006](docs/design/RFC-006-stateless-mcp.md) | | CI / security | SARIF export, OWASP MCP rule metadata, PR summary markdown + GitHub Action pr-comment | [docs/CIANDREPORTS.md](docs/CIANDREPORTS.md) | | Self-test | E2E dogfood - pytest spawns real mcp-test CLI against bundled FastMCP fixtures; 100% coverage enforced on every PR | [We eat our own dogfood](#we-eat-our-own-dogfood) | | Examples | Runnable platform-qa demo pack (trace + chaos + generate) | [feature-demo/platform-qa](examples/feature-demo/platform-qa/README.md) | | Website | Marketing site at vaquarkhan.github.io/mcp-test-harness (deploys html/ via GitHub Actions) | [html/index.html](html/index.html) |

# Same gate CI enforces on every PR
coverage run -m pytest tests/ --ignore=tests/test_workspace.py -q
coverage report --fail-under=100

# E2E dogfood only
python -m pytest tests/test_harness_dogfood_e2e.py -m e2e -v