Model Evaluation

Model Evaluation — Systematic LLM Quality Assessment

Evaluation Framework

5 Dimensions (Always Test All)

| Dimension | What to Measure | How | |-----------|----------------|-----| | Factual Recall | Specific numbers, dates, thresholds | Known-answer questions | | Calculation | Step-by-step math, correct final answer | Problems with verifiable solutions | | Edge Cases | Ambiguous scenarios requiring judgment | Tricky questions with nuanced answers | | Citation | References real sources (IRC §, IRS Pub) | Check every citation exists | | Coherence | Well-structured, complete, no degeneration | Length, structure, readability |

Scoring

PASS:    Correct answer, well-reasoned, properly cited
PARTIAL: Correct concept, wrong number or missing citation
FAIL:    Wrong answer, hallucinated citation, or degenerated output

Comparison Methodology

Side-by-Side Evaluation

# Run same question through both models
for question in eval_set:
    response_a = query(model_a, question)
    response_b = query(model_b, question)
    # Score independently on 5 dimensions

Regression Detection

When fine-tuning iterations:

1. Run eval BEFORE training (baseline)
2. Run eval AFTER training (candidate)
3. Compare dimension-by-dimension
4. Any dimension regression → investigate before deploying

Overfit Detection Signs

• Model outputs dots, repeating characters, or gibberish
• Responses are exact copies of training examples
• Model can't handle questions outside training distribution
• Response length dramatically changes (too short or infinite)

Root cause: Too many epochs on too little data. Fix: Reduce epochs, add more diverse training data.

Evaluation Set Design

For Domain-Specific Models (e.g., Tax)

| Category | Count | Purpose | |----------|-------|---------| | Factual (known answers) | 10 | Tests memorization of current facts | | Calculation | 5 | Tests reasoning + arithmetic | | Scenario-based | 5 | Tests application of rules | | Edge cases | 5 | Tests judgment under ambiguity | | Out-of-domain | 3 | Tests guardrails (should refuse/deflect) | | Total | 28 | Minimum viable eval set |

Question Quality Rules

1. Every question must have a verifiable correct answer
2. Include the year in factual questions (tax rates change)
3. Calculation questions must have worked solutions for comparison
4. Edge cases should have multiple valid perspectives
5. Out-of-domain questions test that the model doesn't hallucinate expertise

Evaluation Tooling

evaluate.py Pattern

def evaluate(models: list[str], questions: list[dict]) -> list[dict]:
    results = []
    for q in questions:
        row = {"question": q["text"], "expected": q["answer"], "responses": {}}
        for model in models:
            response = query_ollama(model, q["text"])
            row["responses"][model] = {
                "text": response,
                "tokens": token_count,
                "latency_ms": latency,
                "matches_expected": check_answer(response, q["answer"]),
            }
        results.append(row)
    return results

Automated Scoring (Where Possible)

• Factual: Extract numbers, compare to expected
• Calculation: Extract final answer, compare to expected
• Citation: Regex for IRC §, IRS Pub — verify they exist
• Coherence: Check length > 100 chars, no repeated tokens
• Edge cases: Requires human review

Training Data vs Model Quality Correlation

Quality = f(data_quality × data_quantity × model_size) / epochs²

More data → linear improvement
Better data → exponential improvement
More epochs → diminishing returns → overfit
Bigger model → better reasoning, same fact accuracy

Production Deployment Checklist

Before deploying a fine-tuned model:

• [ ] Run full eval set (28+ questions)
• [ ] Compare against baseline on all 5 dimensions
• [ ] No dimension regression
• [ ] Overfit check: test 5 out-of-domain questions
• [ ] Coherence check: 10 random prompts, all produce structured output
• [ ] Latency check: <5s for typical responses
• [ ] VRAM check: fits in target GPU with room for inference

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: scoobydont-666
• Source: scoobydont-666/shared-claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.