# Eval View > Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic. - **Type:** MCP server - **Install:** `agentstack add mcp-hidai25-eval-view` - **Verified:** Pending review - **Seller:** [hidai25](https://agentstack.voostack.com/s/hidai25) - **Installs:** 0 - **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [hidai25](https://github.com/hidai25) - **Source:** https://github.com/hidai25/eval-view - **Website:** https://evalview.com ## Install ```sh agentstack add mcp-hidai25-eval-view ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About The open-source behavior regression gate for AI agents. Think Playwright, but for tool-calling and multi-turn AI agents. --- Your agent can still return `200` and be wrong. A model or provider update can change tool choice, skip a clarification, or degrade output quality without changing your code or breaking a health check. **EvalView catches those silent regressions before users do — and gives you the loop to investigate them, grade the confidence, and broadcast the verdict to your team.** **You don't need frontier-lab resources to run a serious agent regression loop.** EvalView gives solo devs, startups, and small AI teams the same core discipline: snapshot behavior, detect drift, classify changes, and review or heal them safely. **Traditional tests tell you if your agent is up. EvalView tells you if it still behaves correctly.** It tracks drift across outputs, tools, model IDs, and runtime fingerprints with graded confidence — not a binary alarm — so you can tell "the provider changed" from "my system regressed." [](https://github.com/user-attachments/assets/96d8b5f7-3561-44a1-86a4-270fb0d1d8a6) **30-second live demo.** Most eval tools stop at detect and compare. EvalView helps you classify changes, inspect drift, and auto-heal the safe cases. - Catch silent regressions that normal tests miss - Separate provider/model drift from real system regressions - Auto-heal flaky failures with retries, review gates, and audit logs - Replay deterministically — cassettes capture real tool calls once so CI never re-hits live services Built for **frontier-lab rigor, startup-team practicality**: - targeted behavior runs instead of giant always-on eval suites - deterministic diffs first, LLM judgment where it adds signal - faster loops from change -> eval -> review -> ship [How we run EvalView with this operating model →](docs/OPERATING_MODEL.md) ``` ✓ login-flow PASSED ⚠ refund-request TOOLS_CHANGED - lookup_order → check_policy → process_refund + lookup_order → check_policy → process_refund → escalate_to_human ✗ billing-dispute REGRESSION -30 pts Score: 85 → 55 Output similarity: 35% ``` The money screen is the one-line **verdict** that lands under every check — a single ship/don't-ship decision derived from the diff, quarantine state, cost delta, and drift confidence: ``` ───────────────────────────────────────────── VERDICT: 🛑 BLOCK RELEASE ───────────────────────────────────────────── • 1 regression: billing-dispute • 1 test changed behavior: refund-request • Cost up 14% vs baseline Likely cause & next actions: 1. Rerun statistically to distinguish flake from real drift (high severity, high confidence) → evalview check --statistical 5 2. Review tool descriptions for: escalate_to_human (high severity, high confidence) Tool selection changed — usually a prompt edit nudged the model → evalview replay refund-request --trace → evalview golden update refund-request # if the new path is correct ``` Four tiers: `SAFE_TO_SHIP`, `SHIP_WITH_QUARANTINE`, `INVESTIGATE`, `BLOCK_RELEASE`. The verdict is part of `--json` output, the PR comment, and the cloud ship page — CLI, CI, and dashboard all tell the same story. ## Quick Start ```bash pip install evalview ``` ```bash evalview init # Detect agent, auto-configure profile + starter suite evalview snapshot # Save current behavior as baseline evalview check # Catch regressions after every change ``` That's it. Three commands to regression-test any AI agent. `init` auto-detects your agent type (chat, tool-use, multi-step, RAG, coding) and configures the right evaluators, thresholds, and assertions. **After `check`, the investigative loop:** ```bash evalview since # 2-second brief: what's changed since your last run evalview progress --since yesterday # delta report: what improved/regressed evalview drift # per-test sparklines + incident markers evalview slack-digest --webhook $SLACK # post the daily verdict to your team ``` These four commands turn a red ✗ into an answer — *is it real drift, a known flake, or a provider update?* — before anyone opens Slack. `since` is the habit anchor (daily open-the-terminal glance); `progress` answers "did my changes help?" with a worth-a-commit gate; `drift` visualizes the trend; `slack-digest` broadcasts the verdict. See [Daily Workflow →](#daily-workflow). ### Catch silent drift in closed models Worried that `claude-opus-4-5` might behave differently next week without warning? [`evalview model-check`](#model-drift-detection) runs a zero-judge canary suite directly against the provider and tells you exactly when the model has drifted — no agent required, no calibration. ```bash evalview model-check --model claude-opus-4-5-20251101 # first run saves baseline evalview model-check --model claude-opus-4-5-20251101 # next week, detects any change ``` [See the full model-check section →](#model-drift-detection) Other install methods ```bash curl -fsSL https://raw.githubusercontent.com/hidai25/eval-view/main/scripts/install.sh | bash ``` No agent yet? Try the demo ```bash evalview demo # See regression detection live (~30 seconds, no API key) ``` Or clone a real working agent with built-in tests: ```bash git clone https://github.com/hidai25/evalview-support-automation-template cd evalview-support-automation-template make run ``` More entry paths ```bash evalview generate --agent http://localhost:8000 # Generate tests from a live agent evalview capture --agent http://localhost:8000/invoke # Capture real user flows (runs assertion wizard after) evalview capture --agent http://localhost:8000/invoke --multi-turn # Multi-turn conversation as one test evalview generate --from-log traffic.jsonl # Generate from existing logs evalview init --profile rag # Override auto-detected agent profile ``` ## Daily Workflow Detection is only the first step. EvalView gives you the full investigative loop — so when a test goes red, you can answer *"is it real drift, a known flake, or a provider update?"* in four commands, before anyone opens Slack. **Open the terminal — `evalview since`** ``` ╭──────────────────────────── Since your last check ─────────────────────────────╮ │ │ │ 95% pass rate across 14 runs │ │ ⚠ 2 soft change(s) │ │ ✨ improved: search_cases, summarize_thread │ │ │ ╰────────────────────────────────────────────────────────────────────────────────╯ Drift sparklines (most-declining first) ▇▆▅▄▃▂▁▂ billing-dispute 🔥 Streak: 6 days of clean merges One thing to look at first: → evalview replay billing-dispute ``` `evalview since` is the **2-second habit brief** — one hero number, one concern, one action. It reuses Week 1's fingerprinted history, so the "since your last check" window is accurate whether you ran it 10 minutes ago or 4 days ago. Night owls at 2am and daily shippers at 9am see the same command; the label adapts. It's the command that goes in your `.zshrc` so `evalview since` fires when you open the terminal, before the espresso machine is warm. **Morning — `evalview progress --since yesterday`** ``` ✨ 3 test(s) now passing that weren't ⚠ 1 test(s) regressed Improved: + refund-flow + order-lookup (at a4f2e91) Regressed: − billing-dispute Output similarity: 85.20% → 87.50% ↑ +2.30% Worth a commit: ✓ refund-flow (high confidence) → evalview golden update refund-flow ``` A **"worth a commit" gate** (3+ consecutive passes) keeps you from celebrating flakes at 2am. **Triage — `evalview drift billing-dispute`** ``` Test │ Trend │ Samples │ Slope │ First → Last │ Status ─────────────────┼──────────────┼─────────┼────────┼──────────────┼──────── billing-dispute │ ! │ 20 │ -1.5% │ 90% → 80% │ declining │ ▇▆▅▄▃▂▁▂▁▂▁ │ │ │ │ Most concerning: billing-dispute — slope -1.50% per check over 20 samples → evalview replay billing-dispute --trace ``` Unicode sparklines + OLS slope + **incident markers** (`!`) show *when* the test flipped. Drift is graded `insufficient_history / stable / low / medium / high` — not a binary alarm. **Verdict — `evalview check --statistical 5`** When the verdict layer returns `INVESTIGATE`, a stability-replay recommendation is auto-injected at position #0 of the action list, surviving the severity sort so you never miss it. **Quarantine — `evalview quarantine list`** ``` Test │ Owner │ Age │ Flaky │ Trend │ Status │ Reason ──────────────────┼─────────┼──────┼───────┼───────┼───────────┼────────────── race-condition │ @hidai │ 12d │ 4 │ ↘ │ ⏸ active │ race condition db-timeout │ @jane │ 45d │ 8 │ ↗ │ ⏰ STALE │ db timeout ⏰ 1 entry stale — review overdue. Either fix the underlying flake or remove from quarantine: evalview quarantine remove db-timeout ``` Known-flaky tests **don't block CI** — but staleness tracking, owner tags, and a flaky-count trend glyph keep the list honest. Governance built in, not a dumping ground. **Broadcast — `evalview slack-digest --webhook $SLACK_WEBHOOK`** ``` 📊 EvalView digest — yesterday 🟢 95% pass rate across 47 runs Drift ▇▆▅ billing-dispute ⏰ Stale quarantine 1 overdue • db-timeout — @jane — 45d 🎯 Next: evalview check --fail-on REGRESSION ✓ Digest posted to Slack. ``` Stdlib-only Block Kit post (zero new deps). Fails soft on bad webhooks. Ends with **one actionable command** your team can copy-paste from the channel. --- **The loop closes:** detection → investigation → graded verdict → quarantine governance → broadcast. You wake up, run `progress`, triage with `drift`, confirm with `check --statistical`, and the team sees the digest before standup. That's the morning ritual — reach for it before the espresso machine warms up. ## Why EvalView? Most of these tools are built for a different job than EvalView, and several pair well with it. The short version of where each one focuses: | Tool | Built primarily for | |---|---| | [Langfuse](docs/VS_LANGFUSE.md) | Open-source observability and tracing | | [LangSmith](docs/VS_LANGSMITH.md) | Observability and evals, native to LangChain / LangGraph | | [Braintrust](docs/VS_BRAINTRUST.md) | Eval scoring, experiments, and production data loops | | [Promptfoo](docs/VS_PROMPTFOO.md) | Prompt and model comparison | | [DeepEval](docs/VS_DEEPEVAL.md) | Metric-based eval unit tests (pytest-style) | | **EvalView** | **Behavior-regression gating for tool-calling agents** | **Where EvalView puts its focus:** - Diffs the whole trajectory — tool calls, parameters, and order — not just the final output - Golden baselines with multi-variant support, so non-determinism doesn't make the gate flaky - Silent model / runtime change detection via fingerprinting - Auto-heal — retries flakes and proposes new variants instead of just failing - Hermetic record/replay cassettes so CI never re-hits live services - The deterministic tool + sequence diff runs without any API key Many teams run an observability tool (Langfuse, LangSmith) or an eval platform (Braintrust, DeepEval, Promptfoo) **and** EvalView — the first for visibility, EvalView for the merge-time regression gate. [Detailed comparisons →](docs/COMPARISONS.md) Comparison reflects each tool's primary positioning as of June 2026, based on public documentation; capabilities change over time. Spotted something inaccurate? [Open an issue or PR](https://github.com/hidai25/eval-view/issues). Product names are trademarks of their respective owners; EvalView is independent and not affiliated with or endorsed by them. ## What It Catches | Status | Meaning | Action | |--------|---------|--------| | ✅ **PASSED** | Behavior matches baseline | Ship with confidence | | ⚠️ **TOOLS_CHANGED** | Different tools called | Review the diff | | ⚠️ **OUTPUT_CHANGED** | Same tools, output shifted | Review the diff | | ❌ **REGRESSION** | Score dropped significantly | Fix before shipping | | 📉 **DRIFTING** | Trend sliding with graded confidence (low/med/high) | Run `evalview drift ` | | 🔎 **INVESTIGATE** | Verdict layer wants statistical replay | Run `evalview check --statistical 5` | | ⏳ **QUARANTINED** | Known-flaky, excluded from CI exit code | Fix underlying flake or remove | ### Model / Runtime Change Detection EvalView does more than compare `model_id`. - **Declared model change**: adapter-reported model changed from baseline - **Runtime fingerprint change**: observed model labels in the trace changed, even when the top-level model name is missing - **Coordinated drift**: multiple tests shift together in the same check run, which often points to a silent provider rollout or runtime change — now **graded `low / medium / high`** via `DriftTracker.classify_drift`, not a binary alarm When detected, `evalview check` surfaces a run-level signal with a classification (`declared` or `suspected`), confidence level, and evidence from fingerprints, retries, and affected tests. If the new behavior is correct, rerun `evalview snapshot` to accept the updated baseline. **Four scoring layers** — the first two are free and offline: | Layer | What it checks | Cost | |-------|---------------|------| | **Tool calls + sequence** | Exact tool names, order, parameters | Free | | **Code-based checks** | Regex, JSON schema, contains/not_contains | Free | | **Semantic similarity** | Output meaning via embeddings | ~$0.00004/test | | **LLM-as-judge** | Output quality scored by LLM (GPT, Claude, Gemini, DeepSeek, Ollama) | ~$0.01/test | ``` Score Breakdown Tools 100% ×30% Output 42/100 ×50% Sequence ✓ ×20% = 54/100 ↑ tools were fine ↑ this is the problem ``` ## CI/CD Integration Block broken agents in every PR. One step — PR comments, artifacts, and job summary are automatic. ```yaml # .github/workflows/evalview.yml — copy this, add your secret, done name: EvalView Agent Check on: [pull_request, push] jobs: agent-check: runs-on: ubuntu-latest permissions: pull-requests: write steps: - uses: actions/checkout@v4 - name: Check for agent regressions uses: hidai25/eval-view@v0.8.0 with: openai-api-key: ${{ secrets.OPENAI_API_KEY }} ``` What lands on your PR ``` ## ✅ EvalView: PASSED | Metric | Value | |--------|-------| | Tests | 5/5 unchanged (100%) | --- *Generated by EvalView* ``` When something breaks: ``` ## ❌ EvalView: REGRESSION > **Alerts** > - 💸 Cost spike: $0.02 → $0.08 (+300%) > - 🤖 Model changed: gpt-5.4 → gpt-5.4-mini | Metric | Value | |--------|-------| | Tests | 3/5 unchanged (60%) | | Regressions | 1 | | Tools Changed | 1 | ### Changes from Baseline - ❌ **search-flow**: score -15.0, 1 tool change(s) - ⚠️ **create-flow**: 1 tool change(s) ``` Common options: `strict: 'true'` | `fail-on: 'REGRESSION,TOOLS_CHANGED'` | `mode: 'run'` | `filter: 'my-test'` Also works with [pre-push hooks](docs/CI_CD.md) (`evalview install-hooks`) and [status badges](docs/CI_CD.md) (`evalview badge`). [Full CI/CD guide →](docs/CI_CD.md) ## Watch Mode Leave it running while you code. Every file save triggers a regression check. ```bash evalview watch # Watch current dir, check on change evalview watch --quick # No LLM judge — $0, sub-second evalview watch --test "refund-flow" # Only check one test ``` ``` ╭─────────────────────────── EvalView Watch ────────────────────────────╮ │ Watching . … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [hidai25](https://github.com/hidai25) - **Source:** [hidai25/eval-view](https://github.com/hidai25/eval-view) - **License:** Apache-2.0 - **Homepage:** https://evalview.com Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** yes - **Filesystem access:** no - **Shell / process execution:** yes - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: flagged — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-hidai25-eval-view - Seller: https://agentstack.voostack.com/s/hidai25 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.