Eval View

The open-source behavior regression gate for AI agents. Think Playwright, but for tool-calling and multi-turn AI agents.

---

Your agent can still return 200 and be wrong. A model or provider update can change tool choice, skip a clarification, or degrade output quality without changing your code or breaking a health check. EvalView catches those silent regressions before users do — and gives you the loop to investigate them, grade the confidence, and broadcast the verdict to your team.

You don't need frontier-lab resources to run a serious agent regression loop. EvalView gives solo devs, startups, and small AI teams the same core discipline: snapshot behavior, detect drift, classify changes, and review or heal them safely.

Traditional tests tell you if your agent is up. EvalView tells you if it still behaves correctly. It tracks drift across outputs, tools, model IDs, and runtime fingerprints with graded confidence — not a binary alarm — so you can tell "the provider changed" from "my system regressed."

[](https://github.com/user-attachments/assets/96d8b5f7-3561-44a1-86a4-270fb0d1d8a6)

30-second live demo.

Most eval tools stop at detect and compare. EvalView helps you classify changes, inspect drift, and auto-heal the safe cases.

• Catch silent regressions that normal tests miss
• Separate provider/model drift from real system regressions
• Auto-heal flaky failures with retries, review gates, and audit logs
• Replay deterministically — cassettes capture real tool calls once so CI never re-hits live services

Built for frontier-lab rigor, startup-team practicality:

• targeted behavior runs instead of giant always-on eval suites
• deterministic diffs first, LLM judgment where it adds signal
• faster loops from change -> eval -> review -> ship

[How we run EvalView with this operating model →](docs/OPERATING_MODEL.md)

  ✓ login-flow           PASSED
  ⚠ refund-request       TOOLS_CHANGED
      - lookup_order → check_policy → process_refund
      + lookup_order → check_policy → process_refund → escalate_to_human
  ✗ billing-dispute      REGRESSION  -30 pts
      Score: 85 → 55  Output similarity: 35%

The money screen is the one-line verdict that lands under every check — a single ship/don't-ship decision derived from the diff, quarantine state, cost delta, and drift confidence:

─────────────────────────────────────────────
 VERDICT: 🛑 BLOCK RELEASE
─────────────────────────────────────────────

  • 1 regression: billing-dispute
  • 1 test changed behavior: refund-request
  • Cost up 14% vs baseline

Likely cause & next actions:

  1. Rerun statistically to distinguish flake from real drift
     (high severity, high confidence)
     → evalview check --statistical 5

  2. Review tool descriptions for: escalate_to_human
     (high severity, high confidence)
     Tool selection changed — usually a prompt edit nudged the model
     → evalview replay refund-request --trace
     → evalview golden update refund-request   # if the new path is correct

Four tiers: SAFE_TO_SHIP, SHIP_WITH_QUARANTINE, INVESTIGATE, BLOCK_RELEASE. The verdict is part of --json output, the PR comment, and the cloud ship page — CLI, CI, and dashboard all tell the same story.

Quick Start

pip install evalview

evalview init        # Detect agent, auto-configure profile + starter suite
evalview snapshot    # Save current behavior as baseline
evalview check       # Catch regressions after every change

That's it. Three commands to regression-test any AI agent. init auto-detects your agent type (chat, tool-use, multi-step, RAG, coding) and configures the right evaluators, thresholds, and assertions.

After check, the investigative loop:

evalview since                             # 2-second brief: what's changed since your last run
evalview progress --since yesterday        # delta report: what improved/regressed
evalview drift                             # per-test sparklines + incident markers
evalview slack-digest --webhook $SLACK     # post the daily verdict to your team

These four commands turn a red ✗ into an answer — is it real drift, a known flake, or a provider update? — before anyone opens Slack. since is the habit anchor (daily open-the-terminal glance); progress answers "did my changes help?" with a worth-a-commit gate; drift visualizes the trend; slack-digest broadcasts the verdict. See [Daily Workflow →](#daily-workflow).

Catch silent drift in closed models

Worried that claude-opus-4-5 might behave differently next week without warning? [evalview model-check](#model-drift-detection) runs a zero-judge canary suite directly against the provider and tells you exactly when the model has drifted — no agent required, no calibration.

evalview model-check --model claude-opus-4-5-20251101   # first run saves baseline
evalview model-check --model claude-opus-4-5-20251101   # next week, detects any change

[See the full model-check section →](#model-drift-detection)

Other install methods

curl -fsSL https://raw.githubusercontent.com/hidai25/eval-view/main/scripts/install.sh | bash

No agent yet? Try the demo

evalview demo        # See regression detection live (~30 seconds, no API key)

Or clone a real working agent with built-in tests:

git clone https://github.com/hidai25/evalview-support-automation-template
cd evalview-support-automation-template
make run

More entry paths

evalview generate --agent http://localhost:8000           # Generate tests from a live agent
evalview capture --agent http://localhost:8000/invoke      # Capture real user flows (runs assertion wizard after)
evalview capture --agent http://localhost:8000/invoke --multi-turn  # Multi-turn conversation as one test
evalview generate --from-log traffic.jsonl                # Generate from existing logs
evalview init --profile rag                               # Override auto-detected agent profile

Daily Workflow

Detection is only the first step. EvalView gives you the full investigative loop — so when a test goes red, you can answer "is it real drift, a known flake, or a provider update?" in four commands, before anyone opens Slack.

Open the terminal — evalview since

╭──────────────────────────── Since your last check ─────────────────────────────╮
│                                                                                │
│   95%  pass rate across 14 runs                                                │
│   ⚠  2 soft change(s)                                                          │
│   ✨ improved: search_cases, summarize_thread                                  │
│                                                                                │
╰────────────────────────────────────────────────────────────────────────────────╯

Drift sparklines (most-declining first)
  ▇▆▅▄▃▂▁▂  billing-dispute

🔥 Streak: 6 days of clean merges

One thing to look at first:
  → evalview replay billing-dispute

evalview since is the 2-second habit brief — one hero number, one concern, one action. It reuses Week 1's fingerprinted history, so the "since your last check" window is accurate whether you ran it 10 minutes ago or 4 days ago. Night owls at 2am and daily shippers at 9am see the same command; the label adapts. It's the command that goes in your .zshrc so evalview since fires when you open the terminal, before the espresso machine is warm.

Morning — evalview progress --since yesterday

✨ 3 test(s) now passing that weren't
⚠  1 test(s) regressed

Improved:
  + refund-flow
  + order-lookup (at a4f2e91)

Regressed:
  − billing-dispute

Output similarity: 85.20% → 87.50% ↑ +2.30%

Worth a commit:
  ✓ refund-flow (high confidence)
    → evalview golden update refund-flow

A "worth a commit" gate (3+ consecutive passes) keeps you from celebrating flakes at 2am.

Triage — evalview drift billing-dispute

Test             │ Trend        │ Samples │ Slope  │ First → Last │ Status
─────────────────┼──────────────┼─────────┼────────┼──────────────┼────────
billing-dispute  │            ! │ 20      │ -1.5%  │ 90% → 80%    │ declining
                 │ ▇▆▅▄▃▂▁▂▁▂▁ │         │        │              │

Most concerning: billing-dispute — slope -1.50% per check over 20 samples
  → evalview replay billing-dispute --trace

Unicode sparklines + OLS slope + incident markers (!) show when the test flipped. Drift is graded insufficient_history / stable / low / medium / high — not a binary alarm.

Verdict — evalview check --statistical 5

When the verdict layer returns INVESTIGATE, a stability-replay recommendation is auto-injected at position #0 of the action list, surviving the severity sort so you never miss it.

Quarantine — evalview quarantine list

Test              │ Owner   │ Age  │ Flaky │ Trend │ Status    │ Reason
──────────────────┼─────────┼──────┼───────┼───────┼───────────┼──────────────
race-condition    │ @hidai  │ 12d  │ 4     │ ↘     │ ⏸ active │ race condition
db-timeout        │ @jane   │ 45d  │ 8     │ ↗     │ ⏰ STALE  │ db timeout

⏰ 1 entry stale — review overdue.
   Either fix the underlying flake or remove from quarantine:
   evalview quarantine remove db-timeout

Known-flaky tests don't block CI — but staleness tracking, owner tags, and a flaky-count trend glyph keep the list honest. Governance built in, not a dumping ground.

Broadcast — evalview slack-digest --webhook $SLACK_WEBHOOK

📊 EvalView digest — yesterday
🟢 95% pass rate across 47 runs

Drift
  ▇▆▅ billing-dispute

⏰ Stale quarantine
  1 overdue
  • db-timeout — @jane — 45d

🎯 Next: evalview check --fail-on REGRESSION

✓ Digest posted to Slack.

Stdlib-only Block Kit post (zero new deps). Fails soft on bad webhooks. Ends with one actionable command your team can copy-paste from the channel.

---

The loop closes: detection → investigation → graded verdict → quarantine governance → broadcast. You wake up, run progress, triage with drift, confirm with check --statistical, and the team sees the digest before standup. That's the morning ritual — reach for it before the espresso machine warms up.

Why EvalView?

Most of these tools are built for a different job than EvalView, and several pair well with it. The short version of where each one focuses:

| Tool | Built primarily for | |---|---| | [Langfuse](docs/VSLANGFUSE.md) | Open-source observability and tracing | | [LangSmith](docs/VSLANGSMITH.md) | Observability and evals, native to LangChain / LangGraph | | [Braintrust](docs/VSBRAINTRUST.md) | Eval scoring, experiments, and production data loops | | [Promptfoo](docs/VSPROMPTFOO.md) | Prompt and model comparison | | [DeepEval](docs/VS_DEEPEVAL.md) | Metric-based eval unit tests (pytest-style) | | EvalView | Behavior-regression gating for tool-calling agents |

Where EvalView puts its focus:

• Diffs the whole trajectory — tool calls, parameters, and order — not just the final output
• Golden baselines with multi-variant support, so non-determinism doesn't make the gate flaky
• Silent model / runtime change detection via fingerprinting
• Auto-heal — retries flakes and proposes new variants instead of just failing
• Hermetic record/replay cassettes so CI never re-hits live services
• The deterministic tool + sequence diff runs without any API key

Many teams run an observability tool (Langfuse, LangSmith) or an eval platform (Braintrust, DeepEval, Promptfoo) and EvalView — the first for visibility, EvalView for the merge-time regression gate.

[Detailed comparisons →](docs/COMPARISONS.md)

Comparison reflects each tool's primary positioning as of June 2026, based on public documentation; capabilities change over time. Spotted something inaccurate? Open an issue or PR. Product names are trademarks of their respective owners; EvalView is independent and not affiliated with or endorsed by them.

What It Catches

| Status | Meaning | Action | |--------|---------|--------| | ✅ PASSED | Behavior matches baseline | Ship with confidence | | ⚠️ TOOLSCHANGED | Different tools called | Review the diff | | ⚠️ OUTPUTCHANGED | Same tools, output shifted | Review the diff | | ❌ REGRESSION | Score dropped significantly | Fix before shipping | | 📉 DRIFTING | Trend sliding with graded confidence (low/med/high) | Run evalview drift | | 🔎 INVESTIGATE | Verdict layer wants statistical replay | Run evalview check --statistical 5 | | ⏳ QUARANTINED | Known-flaky, excluded from CI exit code | Fix underlying flake or remove |

Model / Runtime Change Detection

EvalView does more than compare model_id.

• Declared model change: adapter-reported model changed from baseline
• Runtime fingerprint change: observed model labels in the trace changed, even when the top-level model name is missing
• Coordinated drift: multiple tests shift together in the same check run, which often points to a silent provider rollout or runtime change — now graded low / medium / high via DriftTracker.classify_drift, not a binary alarm

When detected, evalview check surfaces a run-level signal with a classification (declared or suspected), confidence level, and evidence from fingerprints, retries, and affected tests.

If the new behavior is correct, rerun evalview snapshot to accept the updated baseline.

Four scoring layers — the first two are free and offline:

| Layer | What it checks | Cost | |-------|---------------|------| | Tool calls + sequence | Exact tool names, order, parameters | Free | | Code-based checks | Regex, JSON schema, contains/not_contains | Free | | Semantic similarity | Output meaning via embeddings | ~$0.00004/test | | LLM-as-judge | Output quality scored by LLM (GPT, Claude, Gemini, DeepSeek, Ollama) | ~$0.01/test |

Score Breakdown
  Tools 100% ×30%    Output 42/100 ×50%    Sequence ✓ ×20%    = 54/100
  ↑ tools were fine   ↑ this is the problem

CI/CD Integration

Block broken agents in every PR. One step — PR comments, artifacts, and job summary are automatic.

# .github/workflows/evalview.yml — copy this, add your secret, done
name: EvalView Agent Check
on: [pull_request, push]

jobs:
  agent-check:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Check for agent regressions
        uses: hidai25/eval-view@v0.8.0
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}

What lands on your PR

## ✅ EvalView: PASSED

| Metric | Value |
|--------|-------|
| Tests | 5/5 unchanged (100%) |

---
*Generated by EvalView*

When something breaks:

## ❌ EvalView: REGRESSION

> **Alerts**
> - 💸 Cost spike: $0.02 → $0.08 (+300%)
> - 🤖 Model changed: gpt-5.4 → gpt-5.4-mini

| Metric | Value |
|--------|-------|
| Tests | 3/5 unchanged (60%) |
| Regressions | 1 |
| Tools Changed | 1 |

### Changes from Baseline
- ❌ **search-flow**: score -15.0, 1 tool change(s)
- ⚠️ **create-flow**: 1 tool change(s)

Common options: strict: 'true' | fail-on: 'REGRESSION,TOOLS_CHANGED' | mode: 'run' | filter: 'my-test'

Also works with [pre-push hooks](docs/CICD.md) (evalview install-hooks) and [status badges](docs/CICD.md) (evalview badge).

[Full CI/CD guide →](docs/CI_CD.md)

Watch Mode

Leave it running while you code. Every file save triggers a regression check.

evalview watch                          # Watch current dir, check on change
evalview watch --quick                  # No LLM judge — $0, sub-second
evalview watch --test "refund-flow"     # Only check one test

╭─────────────────────────── EvalView Watch ────────────────────────────╮
│   Watching   .

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [hidai25](https://github.com/hidai25)
- **Source:** [hidai25/eval-view](https://github.com/hidai25/eval-view)
- **License:** Apache-2.0
- **Homepage:** https://evalview.com

Install and usage instructions live in the source repository linked above.