# Evals Implement

> Generate executable graders and configs from goldset. Generates Python graders / metrics and auto-runs unit tests to verify grader correctness.

- **Type:** Skill
- **Install:** `agentstack add skill-tikalk-adlc-team-skills-evals-implement`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [tikalk](https://agentstack.voostack.com/s/tikalk)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [tikalk](https://github.com/tikalk)
- **Source:** https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-implement
- **Website:** https://github.com/tikalk/agentic-sdlc-12-factors

## Install

```sh
agentstack add skill-tikalk-adlc-team-skills-evals-implement
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# evals-implement

## What this skill does

Generates the **complete executable evaluation implementation** following **EDD Principle VIII** (Close Production Loop) from the published goldset, with automated unit testing to verify evaluator correctness.

**Output**:
1. **Grader/Metric Implementation** - Python evaluators for each goldset criterion with binary pass/fail
   - PromptFoo: Python grader functions with JSON output in `evals/{system}/graders/`
   - DeepEval: Custom metric classes inheriting from `BaseMetric`
2. **Evaluator Unit Tests** - Automated tests (`evals/{system}/tests/test_check_*.py`) that run the goldset pass/fail examples against the generated graders to ensure the evaluator itself is accurate
3. **Evaluation Configuration** - Complete config file (`config.js` or `config.py`) with Tier 1 + Tier 2 evaluation structure
4. **Auto-handoff** to `/evals-validate` to run validation

**Key EDD Principles Applied**:
- **Principle VIII**: Close Production Loop - Failure type gates route to appropriate actions
- **Principle II**: Binary Pass/Fail - Ensure graders return strictly 1.0 (pass) or 0.0 (fail)
- **Principle IX**: Test Data as Code - Unit test generated code against dataset examples

## When to use

- **After `/evals-clarify`**: Convert accepted goldset criteria into executable code
- **Regenerating configs**: Re-build evaluator suite after adding new goldset criteria
- **Adding unit tests**: Hardening the evaluator itself against regression or bugs

## When NOT to use

- **Goldset not published**: Run `/evals-clarify` to generate `goldset.json` first
- **Running evaluations**: Use `/evals-validate` to run the suite against application outputs

## Process

### User Input
```text
$ARGUMENTS
```
- `--system SYSTEM` — Override active evaluation framework (`promptfoo` or `deepeval`)
- `--no-tests` — Skip automated unit test generation for graders (not recommended)

### Execution Steps

#### Phase 1: Trace-to-Grader Synthesis (Automated Eval Engineering)
- Reads `evals/{system}/goldset.json`.
- Maps rich evidence fields from the goldset criteria into grader logic (Trace-to-Grader Synthesis):
  - Uses `pass_condition` and `fail_condition` as the grader's core rubric.
  - Extracts pass/fail examples to act as raw data anchors and few-shot classification anchors inside the grader logic.
  - Injects `Root Cause Analysis` and `axial_coding` notes as contextual prompt guidelines or regex patterns to catch exact failure manifestations.
- For PromptFoo: Generates Python grader functions (`evals/{system}/graders/check_*.py`) containing specialized, dynamic LLM-judge templates or regex checks compiled from these goldset inputs.
- For DeepEval: Generates Custom Metric classes inheriting from `BaseMetric` compiled from these goldset inputs.
- All graders conform strictly to the binary pass/fail standard (returning only `1.0` or `0.0`, with zero Likert scale leakage).

#### Phase 2: Unit Test Generation
- Generates matching unit tests (`evals/{system}/tests/test_check_*.py`) for each grader.
- Unit tests verify the grader correctly identifies the goldset's training pass and fail examples.

#### Phase 2b: Closed-Loop Grader Self-Tuning
- Executes generated unit tests (`pytest evals/{system}/tests/`) to verify evaluator accuracy.
- **Grader Calibration Loop**:
  1. Inspects test results to detect any misclassifications (false positives/negatives) on the training cases.
  2. If any test fails, triggers a feedback edit step that parses the failure reasons and automatically adjusts the grader's internal prompt rubric, regex stubs, or score thresholds.
  3. Re-runs pytest to check accuracy.
  4. Repeats for up to **3 iterations** (the hard circuit-breaker limit).
- **Holdout Locking**: Ensure the holdout validation set (`holdout.json`) remains completely isolated and is never loaded or exposed to the self-tuning loop (to prevent overfitting).
- **Failure Escalation**: If the grader does not converge to 100% training accuracy within 3 iterations, the loop halts, surfaces the failing test case details, and raises an error rather than passing silently.

#### Phase 3: Config Generation
- Generates the unified framework configuration file (`config.js` or `config.py`).
- Configures separate Tier 1 (fast checks, <30s, deterministic) and Tier 2 (semantic checks, <5min, LLM-judge) pipelines.

#### Phase 4: Auto-Handoff
Trigger `/evals-validate` to run validation.

## Verification
- `evals/{system}/graders/` contains Python grader scripts for each criterion compiled dynamically from goldset pass/fail examples and root-cause analyses
- `evals/{system}/tests/` contains matching unit test files
- Framework config (`config.js` or `config.py`) successfully generated
- Grader calibration self-tuning loop ran and converged to 100% training accuracy within the 3-iteration cap (or raised explicit non-convergence errors)
- Holdout dataset protection confirmed (validation `holdout.json` remained completely isolated and untouched during tuning)
- All grader unit tests pass locally (`pytest evals/{system}/tests/`)
- Handover summary lists generated graders, self-tuning iterations, and test results

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [tikalk](https://github.com/tikalk)
- **Source:** [tikalk/adlc-team-skills](https://github.com/tikalk/adlc-team-skills)
- **License:** MIT
- **Homepage:** https://github.com/tikalk/agentic-sdlc-12-factors

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-tikalk-adlc-team-skills-evals-implement
- Seller: https://agentstack.voostack.com/s/tikalk
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.