# Eval Designer

> >

- **Type:** Skill
- **Install:** `agentstack add skill-ravi2799-ai-agent-skills-eval-designer`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [ravi2799](https://agentstack.voostack.com/s/ravi2799)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [ravi2799](https://github.com/ravi2799)
- **Source:** https://github.com/ravi2799/ai-agent-skills/tree/main/skills/eval-designer
- **Website:** https://skills.sh

## Install

```sh
agentstack add skill-ravi2799-ai-agent-skills-eval-designer
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Eval Designer Skill

A skill that governs how to **design**, **implement**, or **analyze** evaluation frameworks for AI agent systems.

Identify which operation applies, then follow the corresponding section.

---

## Operation: DESIGN — Designing an Evaluation Framework

### Pre-Design Checklist

Before building any evaluation, verify:

- [ ] I have **defined success criteria** — what does "good" look like?
- [ ] Success criteria are **specific and measurable** (not "performs well")
- [ ] I know the **task type** (classification, generation, extraction, conversation, multi-step)
- [ ] I have **real examples** of expected inputs and outputs
- [ ] I know whether **automated or human** evaluation is appropriate

### Step 1 — Define Success Criteria

Use the SMART framework:

| Property | Bad | Good |
|---|---|---|
| **Specific** | "Good performance" | "Correctly classifies sentiment as positive/negative/neutral" |
| **Measurable** | "High quality" | "F1 score above 0.85 on test set" |
| **Achievable** | "100% accuracy" | "95% accuracy on well-formed inputs" |
| **Relevant** | "Fast response" (when accuracy matters) | "Accurate extraction" (for a data pipeline) |

### Step 2 — Choose Evaluation Categories

Select the categories relevant to your agent:

| Category | What It Measures | Example Metric |
|---|---|---|
| **Task fidelity** | Core accuracy on the primary task | Accuracy, F1, exact match |
| **Edge case handling** | Behavior on unusual inputs | Pass rate on adversarial set |
| **Consistency** | Same input produces similar output | Cosine similarity across runs |
| **Format compliance** | Output matches expected schema | JSON validation pass rate |
| **Tone and style** | Voice matches requirements | Likert scale (1-5) |
| **Context utilization** | Uses provided context effectively | Grounded answer rate |
| **Safety** | No harmful or leaking outputs | PII detection rate, refusal rate |
| **Latency** | Response speed | p50, p95 response time |
| **Cost** | Token/API usage | Cost per task completion |

### Step 3 — Choose Evaluation Method

Prefer methods higher in this hierarchy:

**1. Code-based (fastest, most reliable)**
- Exact string match — classification, yes/no answers
- Regex match — structured outputs with known patterns
- JSON schema validation — API responses, structured data
- Unit tests — deterministic transformations

**2. LLM-as-judge (flexible, verify reliability first)**
- Binary classification — "Did the response answer the question? yes/no"
- Likert scale — "Rate helpfulness 1-5"
- Ordinal comparison — "Which response is better, A or B?"
- Rubric-based — score against detailed criteria

**3. Human review (most flexible, use sparingly)**
- Expert evaluation — domain-specific quality
- User satisfaction — real-world acceptance
- Preference ranking — compare multiple outputs

---

## Operation: IMPLEMENT — Building Test Cases

### Minimum Requirements

- **At least 3 evaluation scenarios** per capability being tested
- Include **happy path**, **edge cases**, and **failure modes**
- Each test case has: input, expected output (or acceptance criteria), and evaluation method

### Test Case Template

```
## Test Case: [descriptive name]

**Input:**
[exact input to the agent]

**Expected Output:**
[expected response or acceptance criteria]

**Evaluation Method:**
[exact_match | regex | json_schema | llm_judge | human]

**Category:**
[task_fidelity | edge_case | consistency | format | safety]

**Notes:**
[why this test case matters, what it catches]
```

### Evaluation Patterns

**Pattern 1 — Exact Match**
Best for classification, entity extraction, yes/no questions.
```python
def eval_exact_match(output, expected):
    return output.strip().lower() == expected.strip().lower()
```

**Pattern 2 — JSON Schema Validation**
Best for structured outputs.
```python
def eval_schema(output, schema):
    try:
        data = json.loads(output)
        jsonschema.validate(data, schema)
        return True
    except (json.JSONDecodeError, jsonschema.ValidationError):
        return False
```

**Pattern 3 — LLM-as-Judge with Rubric**
Best for subjective quality. Always:
- Write a detailed rubric (not "rate quality")
- Request a numeric score (not prose)
- Use reasoning tags, then extract only the score
- Test grader reliability on 10+ examples before trusting it

```
Rate the following response on a scale of 1-5 for helpfulness.

1 - Does not address the question at all
2 - Partially addresses the question but misses key points
3 - Addresses the question but lacks depth or contains minor errors
4 - Addresses the question well with good detail
5 - Excellent response that fully addresses the question with insight

{{AGENT_OUTPUT}}

First reason about the response in  tags, then output
only a single integer 1-5 as your final answer.
```

**Pattern 4 — A/B Comparison**
Best for prompt iteration. Show both outputs to a judge without revealing which is A/B (randomize order).

### Building an Eval Suite

```
eval_suite/
  config.yaml          # eval settings, model, thresholds
  test_cases/
    task_fidelity.yaml  # core accuracy tests
    edge_cases.yaml     # unusual inputs
    safety.yaml         # adversarial inputs
  graders/
    rubric_helpful.md   # LLM judge rubric
  results/
    baseline.json       # scores before changes
    current.json        # scores after changes
```

---

## Operation: ANALYZE — Interpreting Results

### Analysis Steps

1. **Compare to baseline** — are scores better, worse, or same?
2. **Identify failure patterns** — do failures cluster by category, input type, or length?
3. **Rank improvements by impact** — fix the highest-frequency failures first
4. **Check for regressions** — did fixing one thing break another?
5. **Track over time** — maintain a score history across iterations

### Results Report Format

```
## Evaluation Report

### Summary
| Metric | Baseline | Current | Delta |
|--------|----------|---------|-------|
| Task accuracy | 78% | 85% | +7% |
| Edge case pass rate | 45% | 62% | +17% |
| Format compliance | 92% | 95% | +3% |
| Average latency | 2.1s | 2.3s | +0.2s |

### Failure Analysis
| Failure Type | Count | Example | Root Cause |
|---|---|---|---|
| Missing field | 12 | [example] | Schema not specified in prompt |
| Wrong format | 5 | [example] | Ambiguous format instruction |

### Recommended Actions (priority order)
1. [highest impact fix]
2. [next priority]
3. [...]
```

### Evaluation-Driven Development Loop

1. Run agent on real tasks **without** changes — document failures
2. Create eval scenarios **from those failures**
3. Establish **baseline scores**
4. Make **minimal changes** to address the top failures
5. Re-evaluate — compare to baseline
6. Repeat until target scores are met

### The Two-Agent Testing Pattern

- **Agent A** (designer) — writes or refines the prompt/skill
- **Agent B** (fresh instance) — tests it on real tasks with no prior context
- Observe Agent B's behavior, bring insights back to Agent A
- This prevents the designer from assuming context that won't exist in production

---

## What NOT To Do

- Evaluate without defining success criteria first
- Use only happy-path test cases (always include edge cases)
- Trust an LLM judge without testing its reliability
- Use human review when code-based evaluation would work
- Optimize for a metric that doesn't reflect real-world quality
- Skip the baseline — you cannot measure improvement without a starting point
- Create test cases that only pass with exact wording (brittle tests)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ravi2799](https://github.com/ravi2799)
- **Source:** [ravi2799/ai-agent-skills](https://github.com/ravi2799/ai-agent-skills)
- **License:** MIT
- **Homepage:** https://skills.sh

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-ravi2799-ai-agent-skills-eval-designer
- Seller: https://agentstack.voostack.com/s/ravi2799
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.