Dspy Evaluation Harness

DSPy Evaluation Harness (3.2.x)

The metric is usually more important than the program. For dspy.GEPA especially, the quality of textual feedback in your metric determines whether optimization converges.

Two rules

1. Return a dspy.Prediction(score=..., feedback=...), not a dict. dspy.Evaluate's parallel executor aggregates scores via sum, which breaks on dict outputs (TypeError: unsupported operand type(s) for +: 'int' and 'dict'). dspy.Prediction supports __float__/__add__ and is what GEPA's adapter natively unwraps. A bare float still works for pure dspy.Evaluate scoring, but GEPA needs the score+feedback pair.
2. Separate valset. Never optimize and evaluate on the same examples. Optimizers overfit fast.

Canonical rich-feedback metric

import dspy

def rich_metric(gold: dspy.Example, pred: dspy.Prediction, trace=None,
                pred_name: str | None = None, pred_trace=None):
    # 1. Compute sub-scores — multi-axis beats scalar
    correctness = 1.0 if _normalize(pred.answer) == _normalize(gold.answer) else 0.0
    cited = _has_citation(pred.answer, gold.sources) if hasattr(gold, "sources") else 1.0
    concise = 1.0 if len(pred.answer.split()) = 0.75, f"Regression: {result.score:.3f}"

Run offline in CI with a cached LM (dspy.LM(..., cache=True)) + pre-populated DSPY_CACHEDIR.

Tracing & observability

• track_usage=True on dspy.configure accumulates token counts on predictions (pred.get_lm_usage()).
• MLflow: import mlflow; mlflow.dspy.autolog() → traces every prediction.
• W&B: pass use_wandb=True to dspy.GEPA to log Pareto fronts.
• Always log eval results to a versioned JSON file (save_as_json=...) so you can diff runs.

Anti-patterns

• Scalar-only metrics (float but no feedback) when using GEPA — wasted signal.
• return {"score": s, "feedback": f} (dict) — crashes dspy.Evaluate's parallel aggregator. Use dspy.Prediction(score=s, feedback=f).
• Exact-match metrics on open-ended generation tasks — use semantic or LM-as-judge scoring.
• Evaluating on the trainset — optimistic by 10–30 points.
• Silently swallowing exceptions (provide_traceback=False) — you'll blame the LM for a KeyError.
• Changing the metric mid-experiment without re-baselining — prior numbers become incomparable.

Next

• Feed this metric into dspy-gepa-optimizer.
• Full harness pattern → [reference.md](reference.md).
• Runnable example → [examplemetric.py](examplemetric.py).

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: intertwine
• Source: intertwine/dspy-agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.