Ghostlab

Ghostlab

> A local, end-to-end testing lab for MCP servers and agent skills — coding agents role-play > real users, drive your tools over multiple turns, and the harness captures > traces, scores outcomes, and even renders and clicks through MCP Apps UI > widgets.

[](https://github.com/sajjadGG/Ghostlab/actions) [](https://sajjadgg.github.io/Ghostlab/) [](pyproject.toml) [](LICENSE) [](llms.txt)

Test your MCP server the way it's actually used — not with unit tests against the protocol, but with a real coding agent (Codex / Claude) that picks tools, makes mistakes, and tries to accomplish goals, while a second agent plays the user. Protocol-level checks (schema errors, a tool call that 500s) are useful sanity checks, but they aren't the real test — the real test is whether an agent can actually get a task done through your MCP, end to end.

📖 Docs wiki: https://sajjadgg.github.io/Ghostlab/ · 🤖 For agents: [llms.txt](llms.txt) · 🛠 Contributing: [CONTRIBUTING.md](CONTRIBUTING.md)

> Naming: the project and repo are Ghostlab (formerly Rehearsal). The CLI > is ghostlab, with rehearsal kept as an alias, and the installed Python > package is rehearsal — all the same project.

Quickstart

python3 -m venv .venv                # Python 3.10+
.venv/bin/pip install -e .            # add '.[ui]' for the web UI, '.[apps]' for widget rendering

ghostlab create

That's the whole flow. ghostlab create walks you through everything, end to end:

1. Name + target — the only two prompts. A target is an MCP URL/config, or

a local SKILL.md supplied with --skill.

1. Discover — connects to the target, lints its contract (schema errors,

risk labels), and probes any MCP Apps ui:// widgets.

1. Configure semantic testing — if codex is on your $PATH, offers to

wire it up as the agent-under-test: a real coding-agent session with your MCP's tools available, so the dual-agent conversations below have something to actually drive. (No codex? The wizard still finishes — protocol-level suites still run, semantic/security just skip until a host is configured; see [Runner Configs](#runner-configs).)

1. Generate a test plan — personas × scenarios for the semantic/security

suites, plus deterministic coverage for every discovered tool (test-plan.yaml), all editable afterward.

1. Pick which suites to run — defaults to everything; narrow it to just

semantic while you're iterating, or the full set for a release check.

1. Run + review — executes the plan against your configured host(s),

writes a colored pass/fail summary plus a dashboard, and prints the readiness/gate verdict.

Everything the wizard does is one of discover / plan / test / review under the hood — run any of them standalone afterward to iterate without repeating the whole wizard:

ghostlab discover --job     # re-inspect after the target changes
ghostlab plan --job         # regenerate/curate the test plan
ghostlab test --job         # rerun (add --suite semantic to narrow it)
ghostlab test --job  --resume  # keep completed cases; retry harness outages
ghostlab create --name  --resume --yes  # continue the full job pipeline
ghostlab review --job       # the readiness/gate report on its own

A job is a self-contained folder: jobs//job.yaml (target, hosts, generation/test defaults, gates — all editable), test-plan.yaml, workspace/ (discover/generated/test artifacts + a local sqlite db), and runs/.

To evaluate a skill instead of an MCP server:

ghostlab create --name release-notes-skill --skill ./skills/release-notes --yes

Skill discovery reads SKILL.md; planning generates persona-grounded semantic and adversarial cases; testing injects the skill instructions into the AUT and judges observable compliance. MCP-only protocol and Apps suites are omitted.

What you get

| Stage | What it produces | | --- | --- | | Discover | A deterministic contract.json (schema lint, risk labels, MCP Apps metadata checks) and a refreshed capabilities: section in job.yaml | | Plan | A coverage-driven test-plan.yaml: deterministic protocol cases for every tool, plus generated persona/scenario cases for the semantic/security suites | | Test | Multi-host execution results (results.json/results.md), a standalone HTML dashboard, and — for conversational cases — full dual-agent transcripts with structured tool-call capture | | Review | A readiness report: pass/fail gate verdict, failure clusters, and prioritized repairs |

See it in action

Watch a real coding-agent drive your MCP, turn by turn — every tool call is captured with its pass/fail status. Below is the live trace of a Hugging Face MCP run, including two hf_hub_query calls that failed against the server:

Get a standalone HTML dashboard — pass rate, per-case verdicts, and suite/host tags at a glance:

Drill into any case — the goal and persona, the judge's verdict with its reasoning, and the full dual-agent transcript with inline tool calls:

Goal

Build a repeatable, sandboxed tester that can:

• Run any target MCP app in an isolated environment.
• Launch one coding-agent session as the agent-under-test (with target MCP injected).
• Launch another coding-agent session as the user emulator (persona + goal driven).
• Drive multi-turn interactions between them.
• Capture full traces, tool activity, failures, and outcomes.

This lets you test with your existing Codex/Claude usage path, instead of wiring a separate LLM provider deployment just for E2E testing.

Scope

Ghostlab is intentionally app-agnostic:

• Works with any MCP server reachable by stdio/SSE/streamable HTTP.
• Supports local or remote MCP endpoints.
• Supports multiple coding-agent runners (Codex, Claude Code, and future adapters).

No Cortex-specific assumptions are required in the core harness.

Reference

Everything below is the individual-command reference and advanced usage — useful once you're past the first ghostlab create run, or scripting CI.

Spec vs job

There are two ways to hold an evaluation's config; for almost everyone the answer is a job.

• Job (recommended) — a self-contained jobs// folder created by

ghostlab create. Every command takes --job (or auto-detects job.yaml in the current dir). This is the mainstream path the whole Quickstart uses.

• Spec (advanced) — a single standalone ghostlab.yaml produced by

ghostlab init, addressed with --spec . Useful for scripting or keeping config outside the jobs/ layout. Unless you specifically need that, prefer a job.

The commands overlap (discover/plan/test/review accept either --job or --spec); pick one model per evaluation and stay with it.

Job folder layout

jobs//
  job.yaml          # target, setup, hosts, generation, test, prompts, gates
  test-plan.yaml    # produced by `ghostlab plan`
  workspace/        # discover/, generated/, test/ artifacts + ghostlab.sqlite3
  runs/             # dual-agent run output

Core dual-harness architecture

1. AUT Harness (Agent Under Test) — starts a coding-agent session (Codex or

Claude Code), injects the target MCP server config into it, and exposes a controlled I/O bridge so it can receive user messages and return replies/tool results.

1. User Emulator Harness — starts a second coding-agent session, gives it a

scenario file (persona, goals, constraints, success criteria), and asks it to act like a realistic user, sending messages turn-by-turn to the AUT.

1. Orchestrator — coordinates turn-taking, timeouts, retries, and stop

conditions; logs every message/event in structured format; produces a run report with bug candidates and reproduction context.

Target configuration model

Each test run points to a target definition:

• target.id: unique name (filesystem-mcp-local, my-app-staging)
• transport: stdio | sse | streamable-http | skill
• connection: command+args+env (stdio) or URL+headers (network transports)
• capabilities: optional expected tools/resources/prompts
• startup: optional health checks and boot timeout

Commands

The package installs two equivalent console scripts: ghostlab and rehearsal.

• ghostlab create — the end-to-end wizard described above.
• ghostlab init — advanced: scaffold a standalone ghostlab.yaml spec from a target JSON (see [spec vs job](#spec-vs-job) — most users want ghostlab create).
• ghostlab discover — inspect the job's target, lint its contract, refresh capabilities.
• ghostlab plan — generate (or curate) the coverage-driven test plan.
• ghostlab test — execute the test plan across the job's host adapters.
• ghostlab review — readiness report over discover + plan + test artifacts (release gate).
• ghostlab inspect — connect to a target MCP and capture what it exposes (no job needed).
• ghostlab profile — turn an inspect.json into a capability profile (codex).
• ghostlab generate-scenarios / generate-personas / generate-dataset — build reusable persona×scenario datasets outside the job model.
• ghostlab review-dataset / run-dataset — curate and run a standalone dataset.
• ghostlab run — run one dual-agent E2E scenario directly.
• ghostlab evaluate — score a run into a pass/fail verdict (codex judge).
• ghostlab critique — rate a run's tool ergonomics from the agent's perspective (codex).
• ghostlab scorecard — roll run verdicts and critiques into a summary scorecard.
• ghostlab compare — diff two dataset runs for regressions.
• ghostlab apps-probe / apps-render — probe/render MCP Apps ui:// widgets.
• ghostlab doctor — check codex and validate runner presets.
• ghostlab dashboard — build a standalone HTML dashboard for a ghostlab test run.
• ghostlab ui — launch the Streamlit pipeline UI.
• ghostlab db — manage the SQLite persistence database.

The UI: ghostlab ui

Run the whole pipeline from a browser instead of the CLI:

pip install 'ghostlab[ui]'       # installs streamlit
ghostlab ui                      # opens http://localhost:8501

The app mirrors the same job-based flow as ghostlab create: pick or create a job, discover its target, configure the agent-under-test host, generate and curate the test plan, run selected suites, and review colored results and full conversation traces — the same job.yaml/test-plan.yaml/results.json a CLI run of the same job would produce.

Understand a new MCP: inspect

Point it at a target and it introspects the server without any coding-agent credits or manual curl:

ghostlab inspect --target examples/target.json

This connects over the configured transport (stdio / streamable-HTTP / SSE), runs the initialize handshake, and pages through tools/list, resources/list, resources/templates/list, and prompts/list. It writes runs/-inspect/inspect.json (raw) and inspect.md (readable), and lints tool/resource descriptions for references to tools the server does not actually expose. This capability dump is the input to capability profiling and scenario generation.

Profile a new MCP: profile

Turn the raw inspect.json into a structured capability profile — the bridge between Understand and Generate. Deterministic structure (tool taxonomy by name family, read/write state surfaces, gaps) is computed locally; a domain summary and inferred multi-step workflows are generated by codex:

ghostlab profile \
  --inspect runs/-inspect/inspect.json

It writes capabilities.json + capabilities.md next to the inspect.json. Generated workflow steps are filtered to real tool names, so the profile never references hallucinated or non-exposed tools.

Generate scenarios: generate-scenarios

Generate grounded use-case scenarios the MCP supports, derived from the capability profile:

ghostlab generate-scenarios \
  --profile runs/-inspect/capabilities.json \
  --n 3 \
  --output-dir scenarios

Scenarios are spread across intents (happy_path / edge_case / adversarial) and each declares an exercises list of the tools it should drive the assistant to use. Tool references are filtered to real tool names.

Build a persona library: generate-personas

Personas are reusable user profiles decoupled from scenarios, so the same persona can be paired with many scenarios (the basis for the dataset matrix). Generate a domain-relevant library from a capability profile:

ghostlab generate-personas \
  --profile runs/-inspect/capabilities.json \
  --n 4 \
  --output-dir personas

Each persona has a summary, behavioral traits (terse, impatient, easily confused, non-native, ...), and a domain context map (nativelanguage, targetexam, level, ...). Pass one to a run with --persona:

ghostlab run ... --persona persona.json

Build a dataset: generate-dataset

A dataset is a persona x scenario matrix — different users, and different scenarios tailored to each of them. For every persona, codex generates persona-specific scenarios, and the pairs become runnable cases:

ghostlab generate-dataset \
  --profile runs/-inspect/capabilities.json \
  --personas 3 --scenarios-per-persona 3 --seed 7 \
  --name cortex

This writes a self-contained dataset directory:

datasets/cortex/
  dataset.json          manifest: mcp, seed, cases[]
  personas/.json
  scenarios/.json    persona-namespaced; inline `persona` is a situational note

Review & curate a dataset: review-dataset

Before spending agent credits, check that the dataset makes sense:

ghostlab review-dataset \
  --dataset datasets/cortex \
  --profile runs/-inspect/capabilities.json

This writes review.md + review.json with a tool-coverage matrix, per-case previews, and flags (near-duplicate cases, scenarios exercising non-exposed tools, personas with no scenarios). Curation is file-first — each case gets a status in dataset.json (pending / approved / rejected / needs-edit):

ghostlab review-dataset --dataset datasets/cortex \
  --approve case-a case-b --reject case-c

Run a dataset: run-dataset

ghostlab run-dataset \
  --dataset datasets/cortex \
  --target target.json \
  --aut-runner runners/codex-cortex-aut.json \
  --user-runner runners/codex-user-emulator.json \
  --limit 2

Each case runs through the orchestrator (with its persona) into its own run directory, and a dataset-level summary.md + results.json capture per-case status and turn counts.

Tool-call capture & output hygiene

Every run captures structured MCP tool calls from the agent host. The codex AUT runners set "parser": "codex-json" and run codex exec --json, so the orchestrator parses the JSONL stream and records each mcp_tool_call with its arguments, result, error, and status into events.jsonl, with a per-turn table in report.md. stdout and stderr are kept separate: only stdout (with known host noise redacted) becomes the conversational message handed to the other agent, while raw stderr is logged for debugging.

Evaluate a run: evaluate

ghostlab evaluate --run runs/ --capabilities runs/-inspect/capabilities.json

Combines deterministic checks over captured tool calls with a codex LLM-judge that scores each success_criterion/failure_signal from the transcript. Writes verdict.json + verdict.md; exits non-zero unless the verdict is pass.

Compare two runs: compare

ghostlab compare --base runs/-summary --candidate runs/-summary \
  --output comparison.md

Diffs case-by-case on verdict, listing regressions first, then fixes, then other changes. Exits non-zero when there are regressions, so it can gate CI.

MCP Apps: apps-probe / apps-render

Some MCPs ship **MC

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sajjadGG
• Source: sajjadGG/Ghostlab
• License: MIT
• Homepage: https://sajjadgg.github.io/Ghostlab/

Install and usage instructions live in the source repository linked above.