# Model Evaluation

> >

- **Type:** Skill
- **Install:** `agentstack add skill-scoobydont-666-shared-claude-skills-model-evaluation`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [scoobydont-666](https://agentstack.voostack.com/s/scoobydont-666)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [scoobydont-666](https://github.com/scoobydont-666)
- **Source:** https://github.com/scoobydont-666/shared-claude-skills/tree/main/skills/model-evaluation

## Install

```sh
agentstack add skill-scoobydont-666-shared-claude-skills-model-evaluation
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Model Evaluation — Systematic LLM Quality Assessment

## Evaluation Framework

### 5 Dimensions (Always Test All)

| Dimension | What to Measure | How |
|-----------|----------------|-----|
| **Factual Recall** | Specific numbers, dates, thresholds | Known-answer questions |
| **Calculation** | Step-by-step math, correct final answer | Problems with verifiable solutions |
| **Edge Cases** | Ambiguous scenarios requiring judgment | Tricky questions with nuanced answers |
| **Citation** | References real sources (IRC §, IRS Pub) | Check every citation exists |
| **Coherence** | Well-structured, complete, no degeneration | Length, structure, readability |

### Scoring

```
PASS:    Correct answer, well-reasoned, properly cited
PARTIAL: Correct concept, wrong number or missing citation
FAIL:    Wrong answer, hallucinated citation, or degenerated output
```

## Comparison Methodology

### Side-by-Side Evaluation
```python
# Run same question through both models
for question in eval_set:
    response_a = query(model_a, question)
    response_b = query(model_b, question)
    # Score independently on 5 dimensions
```

### Regression Detection
When fine-tuning iterations:
1. Run eval BEFORE training (baseline)
2. Run eval AFTER training (candidate)
3. Compare dimension-by-dimension
4. **Any dimension regression → investigate before deploying**

### Overfit Detection Signs
- Model outputs dots, repeating characters, or gibberish
- Responses are exact copies of training examples
- Model can't handle questions outside training distribution
- Response length dramatically changes (too short or infinite)

**Root cause:** Too many epochs on too little data.
**Fix:** Reduce epochs, add more diverse training data.

## Evaluation Set Design

### For Domain-Specific Models (e.g., Tax)

| Category | Count | Purpose |
|----------|-------|---------|
| Factual (known answers) | 10 | Tests memorization of current facts |
| Calculation | 5 | Tests reasoning + arithmetic |
| Scenario-based | 5 | Tests application of rules |
| Edge cases | 5 | Tests judgment under ambiguity |
| Out-of-domain | 3 | Tests guardrails (should refuse/deflect) |
| **Total** | **28** | Minimum viable eval set |

### Question Quality Rules
1. Every question must have a **verifiable correct answer**
2. Include the **year** in factual questions (tax rates change)
3. Calculation questions must have **worked solutions** for comparison
4. Edge cases should have **multiple valid perspectives**
5. Out-of-domain questions test that the model **doesn't hallucinate** expertise

## Evaluation Tooling

### evaluate.py Pattern
```python
def evaluate(models: list[str], questions: list[dict]) -> list[dict]:
    results = []
    for q in questions:
        row = {"question": q["text"], "expected": q["answer"], "responses": {}}
        for model in models:
            response = query_ollama(model, q["text"])
            row["responses"][model] = {
                "text": response,
                "tokens": token_count,
                "latency_ms": latency,
                "matches_expected": check_answer(response, q["answer"]),
            }
        results.append(row)
    return results
```

### Automated Scoring (Where Possible)
- **Factual:** Extract numbers, compare to expected
- **Calculation:** Extract final answer, compare to expected
- **Citation:** Regex for IRC §, IRS Pub — verify they exist
- **Coherence:** Check length > 100 chars, no repeated tokens
- **Edge cases:** Requires human review

## Training Data vs Model Quality Correlation

```
Quality = f(data_quality × data_quantity × model_size) / epochs²

More data → linear improvement
Better data → exponential improvement
More epochs → diminishing returns → overfit
Bigger model → better reasoning, same fact accuracy
```

## Production Deployment Checklist

Before deploying a fine-tuned model:
- [ ] Run full eval set (28+ questions)
- [ ] Compare against baseline on all 5 dimensions
- [ ] No dimension regression
- [ ] Overfit check: test 5 out-of-domain questions
- [ ] Coherence check: 10 random prompts, all produce structured output
- [ ] Latency check: <5s for typical responses
- [ ] VRAM check: fits in target GPU with room for inference

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [scoobydont-666](https://github.com/scoobydont-666)
- **Source:** [scoobydont-666/shared-claude-skills](https://github.com/scoobydont-666/shared-claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-scoobydont-666-shared-claude-skills-model-evaluation
- Seller: https://agentstack.voostack.com/s/scoobydont-666
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.