Nasde Benchmark Calibration

NASDE Benchmark Calibration

Close the loop between the LLM-as-a-Judge and a human reviewer. The judge scores trials against assessment_criteria.md (per task) and assessment_dimensions.json (benchmark-wide) — but an LLM judge is an imperfect grader, and how it reads the rubric may diverge from how a human grades the code. This skill publishes trial diffs + scores as Pull/Merge Requests for human review, pulls the comments back, and proposes concrete rubric edits.

This is the third skill in the benchmark lifecycle: nasde-benchmark-creator writes the rubric, nasde-benchmark-runner runs trials and scores them, and this skill calibrates the rubric against human judgment before the benchmark is frozen.

Prerequisites

• A sink repository that already exists (creation is out of scope). Configure it in nasde.toml:

``toml [calibration] repo = "https://github.com/Org/nasde-calibration" # full URL or owner/repo slug # platform = "gitlab" # only needed for a bare slug or a self-hosted host # base_branch = "main" # throttle_sec = 2.0 ``

• The platform CLI for that repo's host: gh (GitHub) or glab (GitLab), installed and logged in

(gh auth login / glab auth login). The platform is auto-detected from the repo URL host. nasde never handles tokens — the CLI's keyring does. See ADR-010.

• git on PATH.

Workflow

1. Decide which trials to calibrate

Calibration is about the criteria, not individual runs. Discuss with the user which trials to publish: a representative trial per (task, variant) is usually enough; publishing every repetition floods the sink with PRs. Selecting trials is the agent's judgment here — there is no --select flag yet (that logic may move into the CLI once the workflow settles).

You can pass job directories (all their trials) or individual trial directories — mixed is fine.

2. Publish

nasde calibrate publish jobs//__ -C 
# or a whole job:
nasde calibrate publish jobs/ -C 

Each trial becomes one PR/MR:

• An orphan base branch base/- holds the agent's start-state codebase (seeded once

per (repo, commit); git deduplicates shared blobs across bases).

• The feature branch calib/-/ carries the agent's diff applied as a real

commit, so the PR diff is exactly the agent's work — clean to review.

• The description renders the dominant evaluator cluster's per-dimension mean ± std, the

normalized score, and a "How to calibrate" note.

• A .calibration/ directory ships the review context: the task's instruction.md (what the

agent was asked to do), the assessment_criteria.md + assessment_dimensions.json the judge scored against, every assessment_eval_.json (full per-dimension reasoning), and metrics.json. This lets the reviewer judge whether a score is fair without leaving the PR.

Re-running is idempotent — a trial whose open PR/MR already exists is skipped. Closing a calibration round lets the same trials be re-published into a fresh round. Give the user the PR/MR URLs and ask them to review the diffs and comment inline where a score disagrees with their judgment. If a second person should review, the user adds them as a collaborator on the sink repo.

3. Pull the comments back

After the user has reviewed:

nasde calibrate pull-comments jobs//__ -C  --json

--json emits a machine-readable structure (issue-level + inline comments, each with path/line when inline) — consume it directly. Without --json it prints a summary table.

4. Diagnose the divergence per dimension

For each comment, line it up against the judge's score for that trial (read assessment_summary.json for the per-dimension mean ± std, and assessment_eval_*.json for the per-dimension reasoning). Inline comments carry a path/line — map that back to which dimension's rubric is implicated.

Classify each divergence:

• Judge too harsh — human says the code is fine, judge scored low → the rubric's threshold for

that dimension is too strict, or its description rewards the wrong thing.

• Judge too lenient — human flags a problem the judge missed → the rubric doesn't describe the

failure mode, or its scale lacks resolution there.

• Aligned — no action; record it as a positive signal.

A human "I think this score should be X" comment is a free-text gold label — interpret it; do not expect a rigid format.

5. Propose the rubric edit (for approval)

The rubric to edit lives at evals//tasks//assessment_criteria.md (per task) or evals//assessment_dimensions.json (benchmark-wide, when the divergence is about a dimension's scale/description, not one task's thresholds). The ` and come from the trial's result.json (source, task_name`).

Show the user a concrete diff of the rubric — the specific threshold/description change that would have moved the judge toward the human's score — and wait for approval before writing. Never edit the rubric silently.

After the edit, the loop repeats: re-run the trial through nasde-benchmark-runner (or re-eval an existing job with nasde eval) and re-publish to confirm the judge now aligns. This iterative measure → diagnose → fix → re-measure loop is the point of calibration.

Notes

• Trajectories are intentionally NOT published to the PR (they may contain secrets and clutter the

diff). The PR carries the diff, the assessment files, and metrics only.

• One trial = one PR. All of that trial's repeated evaluations are summarized in the description and

shipped as .calibration/assessment_eval_.json.

• The platform is inferred from the repo URL — GitHub and GitLab both work with no code change, only

the right CLI installed.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: NoesisVision
• Source: NoesisVision/nasde-toolkit
• License: MIT
• Homepage: https://noesis.vision/nasde/

Install and usage instructions live in the source repository linked above.