Test The Measuring Stick

Test the Measuring Stick

Before you trust what a score says about the work, test what the score says about garbage. A grader that rates a worthless answer highly is not measuring quality: it is measuring something else, and every "good" score it gave is unverifiable. This finds that out cheaply.

Use it whenever a number is about to drive a decision, especially when an AI assigned the number.

1. Build the laziest fake answer

Construct a response that does zero real work but has the right shape:

• Copy the question back as the "answer".
• Return an empty-but-correctly-formatted result.
• Give a single canned, content-free line.

The goal is something that should obviously score near the bottom if the grader works.

2. Run the dummy through the exact same scoring

Use the identical grader, rubric, prompt, or ranking, unchanged. No special-casing. Whatever scored the real work scores the dummy.

3. Read what the dummy score tells you

• Dummy scores low → the stick has teeth. The real scores mean something.
• Dummy scores high → the stick is broken. A presence/format/length signal is being rewarded instead of quality. The real scores are now unverified, not good. Treat every one of them as suspect.

A format or structure bonus is the usual culprit: it can float an empty answer to the top.

4. Check for a saturated ceiling

If almost everything (good, mediocre, dummy) clusters near the top, the test is too easy to separate quality at all. Reaching for a bigger, more expensive grader to push the winners higher is p-hacking, not science. Make the test harder, not the grader stronger.

5. Compare against the cheap floor, not zero

"Beats chance" or "beats nothing" is the wrong bar. Name the cheapest thing that already works (the free tool, a one-line baseline, copy-paste, the incumbent) and confirm the expensive option beats that. An expensive signal that loses to a free baseline is not an improvement, however high its raw score.

Output

Return:

• The dummy answer used and the score it received.
• The verdict: stick has teeth / task is trivially easy (saturated ceiling) / stick is broken.
• The cheap-floor comparison: what the cheapest working alternative scores, and whether the expensive option actually beats it.
• If the stick is broken, name the signal it is actually rewarding (presence, format, length) so it can be fixed.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ariaxhan
• Source: ariaxhan/the-agent-library
• License: MIT

Install and usage instructions live in the source repository linked above.