Evaluating Llm Systems

Evaluating LLM Systems

Overview

LLM systems fail the same way training runs do: silently, with plausible output. The difference is that most teams never build the instrument that would show it.

Core principle: Look at your data, then build evals specific to the failure modes you actually find. Generic metrics measure nothing about your product.

Source: Hamel Husain and Shreya Shankar on applied LLM evals.

> "Don't rely on generic evaluation frameworks to measure the quality of your > AI. Instead, create an evaluation system specific to your problem."

Error Analysis Comes First

Before writing a single assertion:

1. Collect ~100 real traces (input, retrieved context, output, tools called).
2. Read them and write a free-form note on each failure — no fixed

categories yet.

1. Group the notes into recurring failure modes.
2. Count each mode. The counts set your priorities.

Categories invented before reading data describe your imagination, not your product.

> "You must remove all friction from the process of looking at data."

Build the trace viewer early. Anything that makes inspection annoying means it stops happening.

Three Levels

Run cheap ones constantly, expensive ones rarely.

| Level | What | Cadence | |---|---|---| | 1 — Assertions | Deterministic checks: valid JSON, required field, no placeholder, tool called, length bound | Every change, in CI | | 2 — LLM judge / human | Subjective quality on a curated set | Set cadence | | 3 — A/B in production | Real users, real outcomes | After significant changes |

Level 1 catches most regressions for almost no cost. Write assertions for the specific failure modes error analysis found — not generic ones.

A 100% pass rate is not the goal. Your target pass rate is a product decision.

Validating The Judge

An unvalidated LLM judge is a random number generator with good manners. Before trusting it:

1. Hand-label 25–50 examples binary — good/bad. Not 1–5; graded scales

produce inconsistent labels from humans and judges alike.

1. Run the judge on the same set.
2. Compare precision and recall separately, not raw agreement.
3. Iterate the judge prompt until it aligns, then re-measure on fresh examples.

> "Using agreement as a metric... can be misleading when classes are imbalanced. > Instead, you should typically measure precision and recall separately."

Re-validate whenever you change the judge prompt or the model behind it. A judge is a model, and it drifts.

Mapping The Training Recipe

The ml-superpowers:building-the-training-skeleton checks all have eval analogues:

| Check | LLM eval version | |---|---| | Verify loss @ init | Score a known-perfect answer (must be max) and an empty one (must be floor) | | Input-independent baseline | Answer with no retrieved context. If it matches the full system, retrieval contributes nothing | | Overfit one batch | Hand-grade 10 items; the harness must agree exactly | | Human baseline | Two humans label the same set — their disagreement is your noise floor and your ceiling | | Full test set | Grade every item, not a sample |

The human-disagreement number matters most: you cannot detect an improvement smaller than the gap between two human labelers.

Avoiding Eval Overfitting

Tuning prompts against a 20-item set until the number rises is overfitting with extra steps. Hold out a set you look at rarely and never tune against, exactly as with a test split.

Common Mistakes

| Mistake | Fix | |---|---| | Generic metrics (helpfulness, coherence) | Assertions specific to your observed failures | | Categories chosen before reading traces | Read first, categorize after | | Trusting an unvalidated judge | Measure precision/recall against human labels | | 1–5 quality scales | Binary. Graded scales are inconsistent. | | Only LLM-judge evals | Most regressions are caught by cheap assertions | | Tuning against the whole eval set | Hold one out |

Red Flags

• An eval suite written before anyone read production traces
• A judge whose agreement with humans has never been measured
• Prompt changes shipped because the output "looks better"
• No assertion-level tests in CI

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Umaraslam66
• Source: Umaraslam66/ml-superpowers
• License: MIT

Install and usage instructions live in the source repository linked above.