Test Builder

Test Builder

Build valid, fair, evidence-based assessments. Follow the working algorithm below; consult the referenced file at each step for the detailed rules.

This skill's files live at ${CLAUDE_PLUGIN_ROOT}/skills/test-builder/ (the "skill directory"). Reference paths like references/x.md are relative to it; run the bundled scripts by absolute path from the user's working directory so outputs land in the user's project, never in the plugin cache. § numbers cite the master guide the references derive from.

Entry points

• Build a test / items from scratch → Steps 0–7.
• Review an existing test or items → Steps 5–7 (reconstruct missing

passports as you go; report per references/review-checklist.md).

• Analyze response data after administration → Steps 8–10.
• A single quick item still gets: mode, Bloom tag, format rules,

anti-pattern screen, passport, and the human-review flag. No shortcuts.

Non-negotiable guardrails (apply at every step)

1. Human-in-the-loop. Every AI-drafted item is flagged

source: AI-generated, reviewed by in its passport and must pass human expert review against the checklist before use. Never present a generated test as ready-to-administer — deliver it as "draft pending human review."

1. Fact-check. Verify any factual claim in a stem, key, or distractor

against an authoritative source; fluent phrasing is not evidence. If you cannot verify a fact, say so in the passport and mark the item for expert verification.

1. Ethics and child safety by default. Screen out discriminatory,

sensitive, dangerous, and elitist contexts; apply crisis/war caution (references/anti-patterns.md §10.2) unless the teacher has deliberately prepared such a learning goal.

1. Scoring fairness. Any multiple-response or partial-credit scheme is

fixed and announced before administration. Refuse to leave scoring ambiguous — if the user won't choose, default to the guide's recommended scheme and state it explicitly.

1. Construct purity. Watch for construct-irrelevant variance (e.g.,

reading load in a math item) and construct underrepresentation (matrix gaps); name them when found.

The working algorithm

Step 0 — Decide the mode FIRST

Ask whether the test is formative (low-stakes), diagnostic, or summative (high-stakes), and identify subject, grade/level, topics, and outcome strands. The mode dictates difficulty, psychometric rigor, feedback, and ethics for everything downstream. In formative mode, never use negative grading; build in spaced repetition and, where relevant, a hinge question with misconception-mapped options. → read references/principles-and-modes.md

Step 1 — Write the specification + matrix

Fill assets/specification-template.md: topics and outcome strands with weights, formats and counts, max score, scoring scheme (announced in advance), administration time. Build the "topics × Bloom × outcome strands" matrix (assets/matrix-template.md); apply the summative difficulty distribution (≈20–25% easy / 50–60% optimal / 20–25% hard) when relevant. Scaffold the matrix with:

python3 "${CLAUDE_PLUGIN_ROOT}/skills/test-builder/scripts/new_matrix.py" --topics "Topic A,Topic B" --items 20 --weights "50,50"

→ read references/specification-and-matrix.md

Step 2 — Bloom-tag every planned item

Use behavioral action verbs; build a "question ladder" per topic to reach the higher levels. Keep difficulty and cognitive level distinct — each item carries both tags independently. → read references/taxonomy.md

Step 3 — Mine distractors from real errors

Where possible, pose the question in open form first, cluster the wrong answers, promote the most frequent to distractors. Document for every distractor the faulty mental model it embodies — this rationale goes in the passport and is validated. If no real error data exists, use documented, named misconceptions and treat the first administration as the mining round. → read references/writing-rules.md (§9.3)

Step 4 — Draft items by format rules

Respect per-format constraints: single-best-answer MC (3 strong options often beat 4 with a dead one; "all/none of the above" forbidden); multiple-response scoring schemes fixed and announced; True/False in series on a shared stimulus; matching with surplus responses; ordering unambiguous; short answer with a full creditable-variants list; extended response with a mandatory rubric. For competence-based curricula, include at least one PISA-style stimulus cluster. Randomize key positions across the test. → read references/item-formats.md and references/writing-rules.md; for subject-specific emphasis → references/subject-adaptations.md; for online delivery → references/online-and-ai.md

Step 5 — Screen against anti-patterns

Check every item, option by option, for the seven cue-giveaways, the ethics/fairness constraints, and the accessibility/cognitive-load rules. Cite violations by anchor (e.g., "violates §10.1.5"). → read references/anti-patterns.md

Step 6 — Complete a passport for every item

Copy assets/item-passport-template.yaml; fill id, subject, grade, topic, outcome strand, Bloom level, mode, format, stem, options, key, per-distractor rationale, scoring, source, reviewer. Flag AI-generated items explicitly in source. Validate:

python3 "${CLAUDE_PLUGIN_ROOT}/skills/test-builder/scripts/validate_passport.py" item-001.yaml item-002.yaml

A failing passport is a defective item — fix or discard before review.

Step 7 — Run the review checklist

Content, form, accessibility, ethics/fairness, and whole-test checks. A second pair of eyes is mandatory; for AI-generated items, a human expert. Discard unsound items — do not rescue them. Deliverable wording: "PASS (pending human sign-off)," never "ready to administer." → read references/review-checklist.md

Step 8 — After administration, compute statistics

Export responses to CSV (rows = takers, columns = items, plus a KEY row — or an already-scored 0/1/points matrix) and run:

python3 "${CLAUDE_PLUGIN_ROOT}/skills/test-builder/scripts/psychometrics.py" responses.csv --out report.md

Interpret P-value, D-index, Rit, and distractor uptake against the guide's interval tables and calibration anchors; give each item a verdict — bank, revise, or retire — and record it in its passport. → read references/psychometrics.md

Step 9 — Scale and report where needed

Raw score → scale → grade, with scale-type caveats (a 12-point grade is ordinal; state conversion rules before seeing results). → read references/psychometrics.md (§12)

Step 10 — Deliver feedback per the test's mode

Formative: immediate, descriptive, explaining every option, never negative grading. Diagnostic: by outcome strand, misconception map from distractor uptake. Summative: score plus item analysis where possible. → read references/principles-and-modes.md (§4)

Output conventions

• Deliver: the specification, the matrix, then items — each item followed by

its passport (YAML block). End with the review-checklist report and the human-review disclaimer.

• Scripts are Python 3, standard library only; they run offline and touch

nothing outside the paths given to them.

• When reviewing, quote the violated anchor (checklist ID and/or guide §) so

the teacher can look it up.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: OleksiiDotsenko
• Source: OleksiiDotsenko/test-builder
• License: MIT

Install and usage instructions live in the source repository linked above.