Eval Guide

Eval Guide

Iron Law

NEVER generate eval tool code from memory. ALWAYS query Context7 MCP for the official API before writing any eval code. Every metric class name, constructor signature, and YAML provider ID must be verified against current official docs — these APIs change between minor versions.

Prefer LangChain-free eval paths. Ragas, Giskard, and Promptfoo all have LangChain-free paths — prefer them to reduce dependency surface and avoid version conflicts.

Consistent LLM judge. Best practice: use the same LLM provider as your main stack for LLM-as-judge to reduce vendor sprawl. For Gemini-based stacks: GeminiModel("gemini-2.5-flash") for DeepEval, Generator(model="google/gemini-3.1-flash") for Giskard, google:gemini-2.5-pro for Promptfoo.

Dispatch eval-reviewer agent after writing any eval code — same mandate as dispatching adk-reviewer after ADK agent code.

Documentation Sources — Query Context7 BEFORE Writing Any Tool Code

| Tool | Context7 query | Fallback | |------|---------------|---------| | DeepEval | deepeval | https://docs.confident-ai.com/docs | | Ragas | ragas | https://docs.ragas.io/en/latest | | Giskard OSS v3 | giskard | https://docs.giskard.ai/en/latest | | Promptfoo | promptfoo | https://www.promptfoo.dev/docs | | Langfuse | langfuse | https://langfuse.com/docs | | Arize Phoenix | arize-phoenix | https://docs.arize.com/phoenix | | ADK Eval | google-adk (adk-docs MCP) | https://google.github.io/adk-docs/evaluate | | Vertex GenAI Eval | google-cloud-aiplatform | https://cloud.google.com/vertex-ai/generative-ai/docs/evaluate | | pytest-asyncio | pytest-asyncio | https://pytest-asyncio.readthedocs.io |

Reference Files

| File | When to use | |------|-------------| | reference/deepeval-patterns.md | MCPUseMetric, GeminiModel, 15 confirmed metric classes, ArenaGEval A/B testing | | reference/ragas-patterns.md | ToolCallAccuracy, Faithfulness, ContextPrecision — LangChain-free path only | | reference/promptfoo-patterns.md | YAML config, google:gemini-2.5-pro provider, 70+ red-team plugins, MCP security suite | | reference/giskard-patterns.md | v3 Scenario/Suite API, LiteLLM Gemini setup, FHA check, RAGET v2-only warning | | reference/langfuse-prompts.md | PromptRegistry abstraction, prompt lifecycle, .compile(), emergency pack, drift detection | | reference/golden-dataset.md | 8-folder structure, dataset_manifest.yaml schema, per-agent case minimums | | reference/ci-tiers.md | R1-R4 tier config, pytest marks (@r1/@r2), path-routing rules, 9 CI blockers reference | | reference/per-agent-thresholds.md | Per-agent accuracy thresholds for all 14 agents, habitability 100% sub-threshold | | reference/mcp-eval-patterns.md | MCP contract suite, tenant isolation test pattern, audit-log verification | | reference/pytest-harness.md | asyncio_mode = "auto", conftest.py template, InMemoryRunner, parametrize-over-golden | | reference/failure-mode-taxonomy.md | 6 failure modes with symptom → eval tool routing table; fix patterns per mode |

Process — Before Writing Any Eval Code

1. Identify which tool(s) are needed
2. Query Context7 for that tool's current API — the reference files are starting points, NOT the final authority on API signatures
3. Read the relevant reference file for patterns and gotchas
4. Check reference/per-agent-thresholds.md for the target agent's required thresholds
5. Check reference/golden-dataset.md for dataset structure and minimum case counts
6. Mark every test with @pytest.mark.r1 (PR gate) or @pytest.mark.r2 (nightly) — never unmarked
7. After implementation: dispatch eval-reviewer agent

Make Targets Quick Reference

| Target | What it runs | When to use | |--------|-------------|-------------| | make eval-smoke | Lint + types + 1-2 eval cases for changed agent + prompt schema check | Every PR ( | Targeted Giskard scan → reports/giskard/.html | Security scan | | make mcp-eval-all | Contract + auth + behavior + security suites for MCP server | MCP eval | | make eval-all-local | All of the above in sequence | Full local validation | | make seed-prompts-local | Seeds prompts into local Langfuse with label="development" | Prompt registry setup | | make diff-prompts-staging | Detects Git ↔ Langfuse prompt drift | Pre-release check | | make phoenix-experiment AGENT= | Phoenix run_experiment() against golden dataset | Trace replay eval | | make mcp-inspect | Launches @modelcontextprotocol/inspector against local MCP | Interactive MCP debug | | make eval-multiturn AGENT= | ADK User Simulation multi-turn flows | Multi-turn eval | | make update-mcp-hashes | Regenerates mcp/.tool-surface-hashes.json` | After MCP tool changes |

Golden Dataset Minimum Requirements (Day-1)

| Agent/Suite | Minimum cases | Location | |-------------|-------------|---------| | Primary agent (highest-risk) | ≥ 8 Day-1 → ≥ 100 full target | tests/golden/agents// | | Secondary agents | ≥ 8 Day-1 → ≥ 20 full target | tests/golden/agents// | | Security suite | ≥ 1 each: promptinjection, tenantisolation, policy_bypass | tests/golden/security/ | | All other agents | ≥ 20 before agent PR merges | tests/golden/agents// | | RAG agents | ≥ 10 faithfulness cases | tests/golden/rag/ | | MCP contract | ≥ 1 per tool | tests/golden/mcp/ |

Related Skills

• adk-eval-guide — ADK-native eval only (8 ADK criteria, evalset schema, user simulation)
• google-adk — ADK agent construction patterns
• adk-observability-guide — Phoenix OTel integration, span inspection

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kumaran-is
• Source: kumaran-is/claude-code-onboarding
• License: MIT

Install and usage instructions live in the source repository linked above.