Deepeval

DeepEval — Claude-native MBB-Grade Quality Framework

This skill scores any LLM-generated artifact against an MBB-grade rubric (BCG-calibrated). It works in any Claude Code project. No external API keys, no vendor SDKs. Claude itself is the judge.

---

Three core promises

1. Tier stack: deterministic → heuristic → Claude judge → human, by

cost/latency budget.

1. BCG-calibrated rubric — 8 dimensions, 1–3 scale, verbatim BCG

anchor language.

1. Day/Week/30-day cadence only. NO 90/180/365-day metrics.

---

How "Claude as judge, no API" works

This skill makes Claude the runtime judge. No anthropic.messages.create or openai.chat.completions.create calls. Flow:

USER:    /deepeval-run path/to/output.md
   │
   ▼
Bash:    python scripts/eval_tier0.py --artifact ...     # ms,  $0
Bash:    python scripts/eval_tier1.py --artifact ...     # /t2-prompt.txt with the full
           BCG-rubric judge prompt filled in with this artifact
   │
   ▼
CLAUDE:  Reads t2-prompt.txt + artifact + upstream context.
         Applies the BCG 8-dim rubric (1–3 scale).
         Writes JSON verdict to .eval/runs//t2-verdict.json.
   │
   ▼ (if `critical: true`)
CLAUDE:  Reads skeptic-prompt.txt (3 attacks: ambiguity, sycophancy,
         steelman-opposite). Writes skeptic.json.
   │
   ▼ (if `claims_novelty: true`)
CLAUDE:  Runs novelty signals 1, 2, 5 manually. Writes novelty.json.
   │
   ▼
Bash:    python scripts/aggregate_verdict.py --run-dir ...
         → final verdict.md with PASS/REVISE/FAIL + fix-directive

No API calls anywhere. Claude in the session does all the judgment work.

---

Workflow when user invokes a command

/deepeval-run

Step 1: Read the artifact + locate upstream context.

cat 
# also find: ../company-brief.md, ../client-context-brief.md, or
# whichever upstream file the manifest names. If not found, ask user.

Step 2: Run Tier 0 (deterministic).

python .claude/skills/deepeval/scripts/eval_tier0.py \
    --artifact  \
    --artifact-type analysis \
    --json > /tmp/t0.json

Read /tmp/t0.json. If passed: false and the user has strict_tier_gate: true in config, stop and report. Otherwise continue.

Step 3: Run Tier 1 (heuristic).

python .claude/skills/deepeval/scripts/eval_tier1.py \
    --artifact  \
    --upstream  \
    --json > /tmp/t1.json

Step 4: Compose the Tier 2 judge prompt.

python .claude/skills/deepeval/scripts/prepare_judge_prompt.py \
    --artifact  \
    --upstream  \
    --burning-problem "" \
    --mode  \
    --out /tmp/t2-prompt.txt

Step 5: YOU (Claude) act as the judge.

Read /tmp/t2-prompt.txt. It contains the BCG 8-dim rubric with verbatim anchor language + the artifact text + upstream context + burning-problem header. Apply the rubric. Score each dimension 1, 2, or 3 with evidence. Compute weighted score. Determine verdict.

Write the JSON verdict directly:

cat > /tmp/t2.json  \
    --upstream  \
    --out /tmp/skeptic-prompt.txt

Read /tmp/skeptic-prompt.txt. Apply 3 attacks. Write JSON to /tmp/skeptic.json.

Step 7: Optional Novelty Stack (if claims_novelty: true).

Read references/novelty-checklist.md. Apply signals 1, 2, 5, 7 manually. (Signals 4, 9, 10 require Skeptic + panels — defer.) Write JSON to /tmp/novelty.json.

Step 8: Aggregate final verdict.

python .claude/skills/deepeval/scripts/aggregate_verdict.py \
    --t0 /tmp/t0.json --t1 /tmp/t1.json --t2 /tmp/t2.json \
    [--skeptic /tmp/skeptic.json] [--novelty /tmp/novelty.json] \
    --out-md /tmp/verdict.md

Step 9: Submit feedback to the deepeval-bcg repo (built-in agent feedback loop).

python .claude/skills/deepeval/scripts/feedback.py \
    --run-dir /tmp/ --submit
# Tries `gh issue create` first → falls back to GitHub API with
# GH_TOKEN / GITHUB_TOKEN → if neither is available, writes the
# pre-filled URL to /tmp//feedback-link.md for the human
# to submit manually.

This closes the eval-quality loop. The agent-submitted issue carries the agent-submitted label so its agreement field doesn't pollute the human κ-proxy in community-stats.md.

Step 10: Show the user /tmp/verdict.md AND the feedback outcome.

If --submit succeeded, end with:

✓ Eval complete. Verdict:  (weighted: X.XX/3.00)
→ Feedback posted: 

If --submit fell back to URL mode (no auth), end with:

✓ Eval complete. Verdict:  (weighted: X.XX/3.00)
→ Submit feedback manually: 
   (Pre-filled — takes ~30 seconds. Your input calibrates the rubric.)

---

/deepeval-feedback

Submit (or regenerate) feedback for an existing eval run. Add --submit to POST the issue directly; omit it to only write the pre-filled URL.

python .claude/skills/deepeval/scripts/feedback.py --run-dir  --submit

---

/deepeval-gate

Aggregate across all artifacts in a gate. For each artifact in manifest with gate: , run /deepeval-run. Compute gate-pass-rate. Verdict: PASS if pass-rate ≥ threshold (default 90%).

/deepeval-amazon

Demo command. Runs full eval on the bundled Amazon strategic engagement example. The example artifact ships with the skill at:

• examples/amazon-eval-2026-05-17/ (within this repo)

Uses 00-executive-summary.md as the artifact and company-brief.md as upstream. If the bundled example is not present (e.g., the skill was installed user-level via install.sh), download it from the repo:

curl -fsSL https://raw.githubusercontent.com/EvXata/deepeval-bcg/main/examples/amazon-eval-2026-05-17/verdict.md \
    -o /tmp/amazon-verdict-reference.md

Output goes to .eval/runs// in the current working directory.

/deepeval-novelty

Run only the 10-signal novelty stack. Output: NoveltyScore + per-signal evidence.

/deepeval-skeptic

Run only the Skeptic Agent attacks.

/deepeval-30d

30-day post-delivery leading-indicator harvest (job postings, Glassdoor, T+7 NPS, stock 30d delta, social sentiment). No 90+/180+/365+ metrics.

---

The BCG rubric (8 dimensions)

Full anchor language → [references/bcg-rubric.md](references/bcg-rubric.md).

| # | Dim | Weight | What it catches | |---|---|---|---| | 1 | Structure | 15% | Narrow framing | | 2 | Ambiguity handling | 10% | AI silently picks interpretation — AI-specific | | 3 | Narrowing / prioritization | 10% | Flat attention spread | | 4 | Rigor + sanity checks | 10% | Method errors, off-by-orders math | | 5 | Breaking obviousness | 20% | Generic boilerplate | | 6 | Synthesis for senior | 20% | Data dump without conclusion | | 7 | Independence (anti-sycophancy) | 10% | Agreeing with whatever | | 8 | Achievement | 5% | "Further analysis recommended" stuck |

Critical dimensions (bolded) — score 1 on any of these forces REVISE minimum, even if weighted score ≥ 2.5.

---

Novelty Stack (10 signals)

Full detail → [references/novelty-checklist.md](references/novelty-checklist.md).

| # | Signal | Auto / Manual | |---|---|---| | 1 | Multi-baseline differential (vs vanilla / RAG / first-principles) | Manual (Claude generates baselines + judges) | | 2 | Prior-probability surprise | Manual (Claude predicts top-3, compares) | | 3 | Insider-data anchor (≥1 white-list tag) | Auto (regex) | | 4 | Skeptic Agent survival | Auto (Skeptic prompt) | | 5 | Counter-narrative framing | Auto + Manual (Claude verifies genuineness) | | 6 | Cross-industry pattern transfer | Manual | | 7 | Elasticity (search-replace test) | Auto (Tier-1) | | 8 | Falsifiability sharpness | Auto + Manual | | 9 | Reverse-Turing panel (5 experts) | Human, Week cadence | | 10 | End-user attention proxy | Behavioral, 30d cap |

NoveltyScore = sum(signals 1–8) / 8 + 0.2 × signal 9 + 0.1 × signal 10.

• ≥ 0.5 → publish as insight
• 0.3 – 0.5 → reclassify as helper-finding
• < 0.3 → strip from output

---

Skeptic Agent

Adversarial sub-agent (Claude-native — see [references/skeptic-agent.md](references/skeptic-agent.md)). Three attacks:

1. Ambiguity probe — did artifact flag input ambiguity or silently choose?
2. Sycophancy probe — did artifact challenge weakly-grounded premises?
3. Steelman opposite — is the opposite recommendation equally defensible?

Verdict: ROBUST / WEAKPUSHBACKNEEDED / REJECTED.

---

Cadence — D/W/30d max

Full table → [references/cadence-day-week-30day.md](references/cadence-day-week-30day.md).

No bucket exceeds 30 days. All long-horizon outcomes mapped to ≤30-day leading-indicator proxies (job postings, Glassdoor sentiment, T+7 NPS, stock 30d delta, social sentiment 30d shift, "was rec applied by T+7?").

---

Cost / latency (Claude-native)

Since Claude is the judge in the runtime session:

| Tier | Latency | Cost in this session | |---|---|---| | T0 | ms | $0 (Python) | | T1 | <1s | $0 (Python) | | T2 | depends on context size | tokens consumed from your session | | Skeptic | same | tokens | | Novelty | manual signals | tokens | | T3 human | hours-days | $20–50/sample (external reviewers) |

Approximate session tokens per artifact eval: 5,000–15,000 (rubric + artifact + upstream + judgment output). At Sonnet pricing — well under $0.20 per artifact when run in batched conversation.

---

File outputs

For each /deepeval-run invocation:

.eval/
└── runs/
    └── 2026-05-17T11-22-Z/
        ├── tier0.json
        ├── tier1.json
        ├── t2-prompt.txt        ← prepared by script; Claude reads
        ├── t2.json              ← Claude writes here
        ├── skeptic-prompt.txt   ← optional
        ├── skeptic.json         ← optional
        ├── novelty.json         ← optional
        └── verdict.md           ← aggregated final

---

What to read next

| Topic | File | |---|---| | Full 4-tier design | [references/4-tier-architecture.md](references/4-tier-architecture.md) | | BCG rubric verbatim anchors | [references/bcg-rubric.md](references/bcg-rubric.md) | | Tier-2 judge prompt template | [references/tier2-judge-prompt.md](references/tier2-judge-prompt.md) | | Novelty stack mechanics | [references/novelty-checklist.md](references/novelty-checklist.md) | | Skeptic Agent prompt | [references/skeptic-agent.md](references/skeptic-agent.md) | | Day/Week/30d cadence | [references/cadence-day-week-30day.md](references/cadence-day-week-30day.md) | | Tier-0 script | [scripts/evaltier0.py](scripts/evaltier0.py) | | Tier-1 script | [scripts/evaltier1.py](scripts/evaltier1.py) | | Prepare judge prompt | [scripts/preparejudgeprompt.py](scripts/preparejudgeprompt.py) | | Aggregate verdict | [scripts/aggregateverdict.py](scripts/aggregateverdict.py) | | Example config | [templates/eval-config.example.yaml](templates/eval-config.example.yaml) | | Example manifest | [templates/manifest.example.json](templates/manifest.example.json) | | Example golden-set | [templates/golden-set.example.jsonl](templates/golden-set.example.jsonl) |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: EvXata
• Source: EvXata/deepeval-bcg
• License: MIT
• Homepage: https://github.com/EvXata/deepeval-bcg

Install and usage instructions live in the source repository linked above.