# Deepeval

> >

- **Type:** Skill
- **Install:** `agentstack add skill-evxata-deepeval-bcg-deepeval`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [EvXata](https://agentstack.voostack.com/s/evxata)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [EvXata](https://github.com/EvXata)
- **Source:** https://github.com/EvXata/deepeval-bcg/tree/main/plugin/skills/deepeval
- **Website:** https://github.com/EvXata/deepeval-bcg

## Install

```sh
agentstack add skill-evxata-deepeval-bcg-deepeval
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# DeepEval — Claude-native MBB-Grade Quality Framework

This skill scores any LLM-generated artifact against an MBB-grade rubric
(BCG-calibrated). It works in any Claude Code project. **No external API
keys, no vendor SDKs.** Claude itself is the judge.

---

## Three core promises

1. **Tier stack:** deterministic → heuristic → Claude judge → human, by
   cost/latency budget.
2. **BCG-calibrated rubric** — 8 dimensions, 1–3 scale, verbatim BCG
   anchor language.
3. **Day/Week/30-day cadence only.** NO 90/180/365-day metrics.

---

## How "Claude as judge, no API" works

This skill makes Claude the runtime judge. No `anthropic.messages.create`
or `openai.chat.completions.create` calls. Flow:

```
USER:    /deepeval-run path/to/output.md
   │
   ▼
Bash:    python scripts/eval_tier0.py --artifact ...     # ms,  $0
Bash:    python scripts/eval_tier1.py --artifact ...     # /t2-prompt.txt with the full
           BCG-rubric judge prompt filled in with this artifact
   │
   ▼
CLAUDE:  Reads t2-prompt.txt + artifact + upstream context.
         Applies the BCG 8-dim rubric (1–3 scale).
         Writes JSON verdict to .eval/runs//t2-verdict.json.
   │
   ▼ (if `critical: true`)
CLAUDE:  Reads skeptic-prompt.txt (3 attacks: ambiguity, sycophancy,
         steelman-opposite). Writes skeptic.json.
   │
   ▼ (if `claims_novelty: true`)
CLAUDE:  Runs novelty signals 1, 2, 5 manually. Writes novelty.json.
   │
   ▼
Bash:    python scripts/aggregate_verdict.py --run-dir ...
         → final verdict.md with PASS/REVISE/FAIL + fix-directive
```

No API calls anywhere. Claude in the session does all the judgment work.

---

## Workflow when user invokes a command

### `/deepeval-run `

**Step 1: Read the artifact + locate upstream context.**

```bash
cat 
# also find: ../company-brief.md, ../client-context-brief.md, or
# whichever upstream file the manifest names. If not found, ask user.
```

**Step 2: Run Tier 0 (deterministic).**

```bash
python .claude/skills/deepeval/scripts/eval_tier0.py \
    --artifact  \
    --artifact-type analysis \
    --json > /tmp/t0.json
```

Read `/tmp/t0.json`. If `passed: false` and the user has `strict_tier_gate:
true` in config, stop and report. Otherwise continue.

**Step 3: Run Tier 1 (heuristic).**

```bash
python .claude/skills/deepeval/scripts/eval_tier1.py \
    --artifact  \
    --upstream  \
    --json > /tmp/t1.json
```

**Step 4: Compose the Tier 2 judge prompt.**

```bash
python .claude/skills/deepeval/scripts/prepare_judge_prompt.py \
    --artifact  \
    --upstream  \
    --burning-problem "" \
    --mode  \
    --out /tmp/t2-prompt.txt
```

**Step 5: YOU (Claude) act as the judge.**

Read `/tmp/t2-prompt.txt`. It contains the BCG 8-dim rubric with verbatim
anchor language + the artifact text + upstream context + burning-problem
header. Apply the rubric. Score each dimension 1, 2, or 3 with evidence.
Compute weighted score. Determine verdict.

Write the JSON verdict directly:

```bash
cat > /tmp/t2.json  \
    --upstream  \
    --out /tmp/skeptic-prompt.txt
```

Read `/tmp/skeptic-prompt.txt`. Apply 3 attacks. Write JSON to /tmp/skeptic.json.

**Step 7: Optional Novelty Stack (if `claims_novelty: true`).**

Read `references/novelty-checklist.md`. Apply signals 1, 2, 5, 7
manually. (Signals 4, 9, 10 require Skeptic + panels — defer.) Write
JSON to /tmp/novelty.json.

**Step 8: Aggregate final verdict.**

```bash
python .claude/skills/deepeval/scripts/aggregate_verdict.py \
    --t0 /tmp/t0.json --t1 /tmp/t1.json --t2 /tmp/t2.json \
    [--skeptic /tmp/skeptic.json] [--novelty /tmp/novelty.json] \
    --out-md /tmp/verdict.md
```

**Step 9: Submit feedback to the deepeval-bcg repo (built-in agent feedback loop).**

```bash
python .claude/skills/deepeval/scripts/feedback.py \
    --run-dir /tmp/ --submit
# Tries `gh issue create` first → falls back to GitHub API with
# GH_TOKEN / GITHUB_TOKEN → if neither is available, writes the
# pre-filled URL to /tmp//feedback-link.md for the human
# to submit manually.
```

This closes the eval-quality loop. The agent-submitted issue carries the
`agent-submitted` label so its agreement field doesn't pollute the
human κ-proxy in `community-stats.md`.

**Step 10: Show the user `/tmp/verdict.md` AND the feedback outcome.**

If `--submit` succeeded, end with:

```
✓ Eval complete. Verdict:  (weighted: X.XX/3.00)
→ Feedback posted: 
```

If `--submit` fell back to URL mode (no auth), end with:

```
✓ Eval complete. Verdict:  (weighted: X.XX/3.00)
→ Submit feedback manually: 
   (Pre-filled — takes ~30 seconds. Your input calibrates the rubric.)
```

---

### `/deepeval-feedback `

Submit (or regenerate) feedback for an existing eval run. Add `--submit`
to POST the issue directly; omit it to only write the pre-filled URL.

```bash
python .claude/skills/deepeval/scripts/feedback.py --run-dir  --submit
```

---

### `/deepeval-gate  `

Aggregate across all artifacts in a gate. For each artifact in manifest
with `gate: `, run /deepeval-run. Compute gate-pass-rate.
Verdict: PASS if pass-rate ≥ threshold (default 90%).

### `/deepeval-amazon`

Demo command. Runs full eval on the bundled Amazon strategic engagement
example. The example artifact ships with the skill at:

- `examples/amazon-eval-2026-05-17/` (within this repo)

Uses `00-executive-summary.md` as the artifact and `company-brief.md` as
upstream. If the bundled example is not present (e.g., the skill was
installed user-level via `install.sh`), download it from the repo:

```bash
curl -fsSL https://raw.githubusercontent.com/EvXata/deepeval-bcg/main/examples/amazon-eval-2026-05-17/verdict.md \
    -o /tmp/amazon-verdict-reference.md
```

Output goes to `.eval/runs//` in the current working directory.

### `/deepeval-novelty `

Run only the 10-signal novelty stack. Output: NoveltyScore + per-signal
evidence.

### `/deepeval-skeptic `

Run only the Skeptic Agent attacks.

### `/deepeval-30d `

30-day post-delivery leading-indicator harvest (job postings, Glassdoor,
T+7 NPS, stock 30d delta, social sentiment). No 90+/180+/365+ metrics.

---

## The BCG rubric (8 dimensions)

Full anchor language → [references/bcg-rubric.md](references/bcg-rubric.md).

| # | Dim | Weight | What it catches |
|---|---|---|---|
| 1 | Structure | 15% | Narrow framing |
| 2 | **Ambiguity handling** | 10% | AI silently picks interpretation — AI-specific |
| 3 | Narrowing / prioritization | 10% | Flat attention spread |
| 4 | Rigor + sanity checks | 10% | Method errors, off-by-orders math |
| 5 | **Breaking obviousness** | 20% | Generic boilerplate |
| 6 | Synthesis for senior | 20% | Data dump without conclusion |
| 7 | **Independence (anti-sycophancy)** | 10% | Agreeing with whatever |
| 8 | Achievement | 5% | "Further analysis recommended" stuck |

Critical dimensions (bolded) — score 1 on any of these forces REVISE
minimum, even if weighted score ≥ 2.5.

---

## Novelty Stack (10 signals)

Full detail → [references/novelty-checklist.md](references/novelty-checklist.md).

| # | Signal | Auto / Manual |
|---|---|---|
| 1 | Multi-baseline differential (vs vanilla / RAG / first-principles) | Manual (Claude generates baselines + judges) |
| 2 | Prior-probability surprise | Manual (Claude predicts top-3, compares) |
| 3 | Insider-data anchor (≥1 white-list tag) | Auto (regex) |
| 4 | Skeptic Agent survival | Auto (Skeptic prompt) |
| 5 | Counter-narrative framing | Auto + Manual (Claude verifies genuineness) |
| 6 | Cross-industry pattern transfer | Manual |
| 7 | Elasticity (search-replace test) | Auto (Tier-1) |
| 8 | Falsifiability sharpness | Auto + Manual |
| 9 | Reverse-Turing panel (5 experts) | Human, Week cadence |
| 10 | End-user attention proxy | Behavioral, 30d cap |

NoveltyScore = sum(signals 1–8) / 8 + 0.2 × signal 9 + 0.1 × signal 10.

- ≥ 0.5 → publish as insight
- 0.3 – 0.5 → reclassify as helper-finding
- < 0.3 → strip from output

---

## Skeptic Agent

Adversarial sub-agent (Claude-native — see
[references/skeptic-agent.md](references/skeptic-agent.md)). Three
attacks:

1. Ambiguity probe — did artifact flag input ambiguity or silently choose?
2. Sycophancy probe — did artifact challenge weakly-grounded premises?
3. Steelman opposite — is the opposite recommendation equally defensible?

Verdict: ROBUST / WEAK_PUSHBACK_NEEDED / REJECTED.

---

## Cadence — D/W/30d max

Full table → [references/cadence-day-week-30day.md](references/cadence-day-week-30day.md).

**No bucket exceeds 30 days.** All long-horizon outcomes mapped to ≤30-day
leading-indicator proxies (job postings, Glassdoor sentiment, T+7 NPS,
stock 30d delta, social sentiment 30d shift, "was rec applied by T+7?").

---

## Cost / latency (Claude-native)

Since Claude is the judge in the runtime session:

| Tier | Latency | Cost in this session |
|---|---|---|
| T0 | ms | $0 (Python) |
| T1 | <1s | $0 (Python) |
| T2 | depends on context size | tokens consumed from your session |
| Skeptic | same | tokens |
| Novelty | manual signals | tokens |
| T3 human | hours-days | $20–50/sample (external reviewers) |

Approximate session tokens per artifact eval: 5,000–15,000 (rubric +
artifact + upstream + judgment output). At Sonnet pricing — well under $0.20
per artifact when run in batched conversation.

---

## File outputs

For each `/deepeval-run` invocation:

```
.eval/
└── runs/
    └── 2026-05-17T11-22-Z/
        ├── tier0.json
        ├── tier1.json
        ├── t2-prompt.txt        ← prepared by script; Claude reads
        ├── t2.json              ← Claude writes here
        ├── skeptic-prompt.txt   ← optional
        ├── skeptic.json         ← optional
        ├── novelty.json         ← optional
        └── verdict.md           ← aggregated final
```

---

## What to read next

| Topic | File |
|---|---|
| Full 4-tier design | [references/4-tier-architecture.md](references/4-tier-architecture.md) |
| BCG rubric verbatim anchors | [references/bcg-rubric.md](references/bcg-rubric.md) |
| Tier-2 judge prompt template | [references/tier2-judge-prompt.md](references/tier2-judge-prompt.md) |
| Novelty stack mechanics | [references/novelty-checklist.md](references/novelty-checklist.md) |
| Skeptic Agent prompt | [references/skeptic-agent.md](references/skeptic-agent.md) |
| Day/Week/30d cadence | [references/cadence-day-week-30day.md](references/cadence-day-week-30day.md) |
| Tier-0 script | [scripts/eval_tier0.py](scripts/eval_tier0.py) |
| Tier-1 script | [scripts/eval_tier1.py](scripts/eval_tier1.py) |
| Prepare judge prompt | [scripts/prepare_judge_prompt.py](scripts/prepare_judge_prompt.py) |
| Aggregate verdict | [scripts/aggregate_verdict.py](scripts/aggregate_verdict.py) |
| Example config | [templates/eval-config.example.yaml](templates/eval-config.example.yaml) |
| Example manifest | [templates/manifest.example.json](templates/manifest.example.json) |
| Example golden-set | [templates/golden-set.example.jsonl](templates/golden-set.example.jsonl) |

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [EvXata](https://github.com/EvXata)
- **Source:** [EvXata/deepeval-bcg](https://github.com/EvXata/deepeval-bcg)
- **License:** MIT
- **Homepage:** https://github.com/EvXata/deepeval-bcg

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-evxata-deepeval-bcg-deepeval
- Seller: https://agentstack.voostack.com/s/evxata
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.