Eval Harness

Evaluation Harness

Systematic evaluation framework for LLM-powered systems and code generation quality. Covers Pass@k metrics, golden datasets, LLM-as-judge evaluation, benchmark suites, regression detection, and eval-driven development. Use this skill when building, measuring, or improving evaluation pipelines for AI features.

When to Use

• Setting up evaluation infrastructure for LLM features
• Measuring code generation quality (Pass@k, accuracy)
• Building golden test datasets for RAG or agent systems
• Implementing LLM-as-judge evaluation pipelines
• Tracking regression across model or prompt changes
• Establishing eval-driven development (EDD) workflows
• Comparing model providers or configurations

Core Concepts

Pass@k Metrics

Pass@k measures the probability that at least one of k generated samples passes all test cases. It's the standard metric for code generation evaluation.

Pass@k = 1 - (C(n-c, k) / C(n, k))

Where:
  n = total number of generated samples per problem
  c = number of samples that pass all tests
  k = number of submissions allowed (typically 1, 10, 100)

Implementation:

import numpy as np
from scipy.special import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """
    Unbiased estimator for Pass@k.
    n: total samples generated per problem
    c: number of correct samples
    k: top-k submissions considered
    """
    if n - c  dict:
    """
    Evaluate a model's Pass@k on a set of coding problems.
    """
    results = []

    for problem in problems:
        problem_results = {"id": problem["id"], "samples": []}

        for i in range(num_samples):
            # Generate code from model
            generated = generate_code(model_name, problem["prompt"], temperature=0.8)

            # Write to temp file and run tests
            test_passed = run_unit_tests(problem["test_code"], generated)

            problem_results["samples"].append({
                "sample_id": i,
                "code": generated,
                "passed": test_passed,
            })

        correct_count = sum(1 for s in problem_results["samples"] if s["passed"])
        problem_results["pass@1"] = pass_at_k(num_samples, correct_count, 1)
        problem_results["pass@10"] = pass_at_k(num_samples, correct_count, 10)
        results.append(problem_results)

    # Aggregate
    avg_pass_1 = np.mean([r["pass@1"] for r in results])
    avg_pass_10 = np.mean([r["pass@10"] for r in results])

    return {
        "model": model_name,
        "num_problems": len(problems),
        "num_samples": num_samples,
        "pass@1": round(avg_pass_1, 4),
        "pass@10": round(avg_pass_10, 4),
        "per_problem": results,
    }

def generate_code(model: str, prompt: str, temperature: float) -> str:
    """Call the model API to generate code."""
    # Implementation depends on model provider
    pass

def run_unit_tests(test_code: str, solution_code: str) -> bool:
    """Execute test suite against generated solution."""
    combined = f"{solution_code}\n\n{test_code}"
    result = subprocess.run(
        ["python", "-m", "pytest", "-q", "--tb=no"],
        input=combined,
        text=True,
        capture_output=True,
        timeout=30,
    )
    return result.returncode == 0

Golden Datasets

Curated input-output pairs used as ground truth for evaluation.

Structure:

{
  "dataset_name": "api-code-generation",
  "version": "1.2.0",
  "created": "2025-01-15",
  "problems": [
    {
      "id": "prob-001",
      "category": "fastapi-endpoint",
      "difficulty": "medium",
      "prompt": "Create a FastAPI POST endpoint /users that validates input with Pydantic, creates a user record, and returns 201 with the user object.",
      "golden_solution": {
        "expected_files": ["routes/users.py"],
        "key_patterns": ["@router.post", "status_code=201", "BaseModel", "HTTPException"],
        "forbidden_patterns": ["global db", "print(", "except Exception"],
        "tests": ["test_create_user_valid", "test_create_user_missing_name", "test_create_user_duplicate_email"]
      },
      "acceptance_criteria": {
        "valid_request": { "status": 201, "response_keys": ["id", "name", "email"] },
        "missing_name": { "status": 422 },
        "duplicate_email": { "status": 409 },
        "invalid_email": { "status": 422 }
      }
    },
    {
      "id": "prob-002",
      "category": "react-component",
      "difficulty": "easy",
      "prompt": "Create a UserCard component that displays user name, email, and a delete button. Call onDelete callback when delete is clicked.",
      "golden_solution": {
        "expected_files": ["components/UserCard.tsx"],
        "key_patterns": ["export function UserCard", "onDelete", "prop-types or TypeScript interface"],
        "forbidden_patterns": ["any", "console.log"],
        "tests": ["renders user info", "calls onDelete"]
      }
    }
  ]
}

LLM-as-Judge Evaluation

Using a strong LLM to evaluate outputs against rubrics. Useful when exact-match comparison is insufficient.

import json
from openai import OpenAI

client = OpenAI()

def llm_judge_evaluate(
    prompt: str,
    candidate_output: str,
    golden_output: str,
    rubric: dict,
    judge_model: str = "gpt-4o",
) -> dict:
    """
    Evaluate candidate output using LLM-as-judge.
    Returns structured scores per rubric dimension.
    """
    rubric_str = json.dumps(rubric, indent=2)

    system_prompt = f"""You are an expert code evaluator. Evaluate the candidate code against the rubric.

Rubric dimensions (score 1-5 each):
{rubric_str}

Respond with ONLY a JSON object: {{"dimension_name": score, ...}}"""

    user_prompt = f"""Original prompt:
{prompt}

Golden reference:
{golden_output}

Candidate output:
{candidate_output}"""

    response = client.chat.completions.create(
        model=judge_model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        response_format={"type": "json_object"},
        temperature=0,
    )

    scores = json.loads(response.choices[0].message.content)
    return scores

# Example rubric
CORRECTNESS_RUBRIC = {
    "correctness": "Does the code produce the expected output?",
    "completeness": "Are all requirements addressed?",
    "error_handling": "Are edge cases and errors handled appropriately?",
    "code_quality": "Is the code clean, readable, and well-structured?",
    "security": "Does the code avoid common security vulnerabilities?",
    "efficiency": "Is the algorithmic complexity reasonable?",
}

# Usage
result = llm_judge_evaluate(
    prompt="Create a function that parses a CSV file and returns column statistics.",
    candidate_output=open("candidate.py").read(),
    golden_solution=open("golden.py").read(),
    rubric=CORRECTNESS_RUBRIC,
)
print(json.dumps(result, indent=2))

Benchmark Suites

Standardized test collections for consistent comparison across models and versions.

HumanEval-style benchmark config:

# evals/benchmarks/humaneval.yaml
name: humaneval
version: "1.0"
source: "https://github.com/openai/human-eval"
num_problems: 164
languages: [python]
metrics: [pass@1, pass@10, pass@100]
timeout_seconds: 30
max_tokens: 512

# Custom internal benchmark
evals/benchmarks/internal-api.yaml
name: internal-api-coding
version: "2.1"
num_problems: 50
categories:
  - fastapi-endpoints
  - pydantic-models
  - sqlalchemy-queries
  - react-components
difficulty_distribution:
  easy: 15
  medium: 25
  hard: 10
metrics: [pass@1, pass@5, llm_judge_score]

Eval-Driven Development (EDD) Workflow

EDD applies TDD principles to evaluation: define expected quality metrics first, then iterate until targets are met.

Step 1: Define Quality Targets

# evals/quality_targets.json
{
  "model": "gpt-4o",
  "targets": {
    "pass@1": 0.75,
    "pass@5": 0.90,
    "llm_judge_avg": 4.0,
    "zero_security_violations": true,
    "max_toxicity_score": 0.1
  },
  "regression_threshold": {
    "pass@1": -0.05,
    "pass@5": -0.03,
    "llm_judge_avg": -0.3
  }
}

Step 2: Build Eval Pipeline

# evals/pipeline.py
import json
import datetime
from pathlib import Path

class EvalPipeline:
    def __init__(self, config_path: str):
        self.config = json.loads(Path(config_path).read_text())
        self.results_dir = Path("evals/results")
        self.results_dir.mkdir(exist_ok=True)

    def run(self, model_name: str) -> dict:
        """Run full evaluation pipeline."""
        results = {
            "model": model_name,
            "timestamp": datetime.datetime.utcnow().isoformat(),
            "metrics": {},
        }

        # 1. Pass@k evaluation
        pass_k_results = self.run_pass_at_k(model_name)
        results["metrics"]["pass@1"] = pass_k_results["pass@1"]
        results["metrics"]["pass@5"] = pass_k_results["pass@5"]

        # 2. LLM-as-judge evaluation
        judge_results = self.run_llm_judge(model_name)
        results["metrics"]["llm_judge_avg"] = judge_results["average_score"]

        # 3. Security scan on generated code
        security_results = self.run_security_check(model_name)
        results["metrics"]["security_violations"] = security_results["violation_count"]

        # 4. Regression check
        self.check_regression(results["metrics"])

        # 5. Save results
        output_path = self.results_dir / f"{model_name}_{datetime.date.today()}.json"
        output_path.write_text(json.dumps(results, indent=2))

        return results

    def check_regression(self, current: dict) -> list[str]:
        """Compare against previous run and flag regressions."""
        previous = self.load_previous_results()
        if not previous:
            return []

        thresholds = self.config["targets"]["regression_threshold"]
        regressions = []

        for metric, threshold in thresholds.items():
            current_val = current.get(metric, 0)
            previous_val = previous["metrics"].get(metric, 0)
            delta = current_val - previous_val

            if delta  dict:
    """
    Compare current eval results against baseline.
    Returns regression report with severity levels.
    """
    regressions = {
        "critical": [],
        "warning": [],
        "info": [],
    }

    metrics_to_check = {
        "pass@1": {"critical": -0.10, "warning": -0.05},
        "pass@5": {"critical": -0.08, "warning": -0.03},
        "llm_judge_avg": {"critical": -0.5, "warning": -0.3},
    }

    for metric, thresholds in metrics_to_check.items():
        current = current_results.get(metric, 0)
        baseline = baseline_results.get(metric, 0)
        delta = current - baseline

        if delta  0:
            regressions["info"].append(
                f"{metric}: {baseline:.4f} → {current:.4f} (Δ {delta:+.4f}) ✓"
            )

    return regressions

# Example output
# {
#   "critical": ["pass@1: 0.7800 → 0.6200 (Δ -0.1600)"],
#   "warning": ["llm_judge_avg: 4.2000 → 3.8000 (Δ -0.4000)"],
#   "info": ["pass@5: 0.8900 → 0.9100 (Δ +0.0200) ✓"]
# }

Eval Config Examples

Full Eval Configuration

# evals/config.yaml
eval_name: api-codegen-eval
model_under_test: claude-sonnet-4-20250514
dataset: datasets/internal-api-v2.json

generation:
  temperature: 0.8
  top_p: 0.95
  max_tokens: 2048
  num_samples: 20
  stop_sequences: ["\n\n\n", "```"]

execution:
  timeout_seconds: 30
  memory_limit_mb: 512
  network_access: false
  allowed_imports: ["fastapi", "pydantic", "sqlalchemy", "pytest"]

evaluation:
  pass_at_k: [1, 5, 10]
  llm_judge:
    model: gpt-4o
    rubric: rubrics/code-correctness.json
    temperature: 0

regression:
  baseline: evals/results/baseline.json
  fail_on_critical: true
  alert_thresholds:
    pass@1: -0.05
    pass@5: -0.03

Scoring and Reporting

def generate_eval_report(results: dict) -> str:
    """Generate human-readable evaluation report."""
    report = f"""
# Evaluation Report: {results['model']}
Date: {results['timestamp']}

## Summary

| Metric | Score | Target | Status |
|---|---|---|---|
| Pass@1 | {results['metrics']['pass@1']:.4f} | 0.75 | {'✅' if results['metrics']['pass@1'] >= 0.75 else '❌'} |
| Pass@5 | {results['metrics']['pass@5']:.4f} | 0.90 | {'✅' if results['metrics']['pass@5'] >= 0.90 else '❌'} |
| LLM Judge Avg | {results['metrics']['llm_judge_avg']:.2f} | 4.0 | {'✅' if results['metrics']['llm_judge_avg'] >= 4.0 else '❌'} |
| Security Violations | {results['metrics']['security_violations']} | 0 | {'✅' if results['metrics']['security_violations'] == 0 else '❌'} |

## Per-Problem Breakdown

"""
    for problem in results.get("per_problem", []):
        status = "✅" if problem["pass@1"] >= 0.5 else "❌"
        report += f"- {problem['id']}: Pass@1 = {problem['pass@1']:.4f} {status}\n"

    return report

CI/CD Integration

# .github/workflows/evals.yml
name: Evaluation Pipeline
on:
  push:
    branches: [main]
    paths: ['prompts/**', 'evals/**']
  schedule:
    - cron: '0 4 * * 1'  # Weekly full eval run

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r evals/requirements.txt
      - name: Run eval pipeline
        run: |
          python -m evals.pipeline --model ${{ secrets.EVAL_MODEL }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - name: Check regressions
        run: python -m evals.check_regression
      - uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: evals/results/
          retention-days: 90

Common Pitfalls

| Pitfall | Consequence | Fix | |---|---|---| | Small dataset (10% Pass@1 drop should block merge

1. Version your datasets — tag releases, track what changed
2. Separate test/train — never evaluate on problems used during prompt engineering
3. Run full eval weekly — daily smoke tests, weekly deep eval
4. Report publicly — dashboards keep quality visible and accountable

Success Metrics

• [ ] Golden dataset has ≥50 problems across multiple difficulty levels
• [ ] Pass@1 meets or exceeds target threshold
• [ ] LLM-as-judge average score ≥ 4.0 / 5.0
• [ ] Zero security violations in generated code
• [ ] No critical regressions between consecutive runs
• [ ] Eval pipeline runs automatically in CI/CD
• [ ] Evaluation results visible in dashboard or report

---

Remember: If you can't measure it, you can't improve it. Evaluation is the feedback loop that turns guessing into engineering.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: noah-sheldon
• Source: noah-sheldon/ai-dev-kit
• License: MIT
• Homepage: https://noahsheldon.dev

Install and usage instructions live in the source repository linked above.