Adjudication Sheets

Human Adjudication Sheets from LLM-Pipeline Data

Problem

Adjudication sheets built from pipeline intermediates tend to carry truncated "excerpt" columns (previews made for machine diffing, not human judging). A human asked to rule "does this text show X?" on a fragment produces invalid rulings: the decisive sentence may sit past the cutoff. Separately, multi-line text in CSV cells renders as broken rows/empty cells in Excel, so the labeler reports "there is no data here" even when the column is populated.

Context / Trigger Conditions

• A labeler says the sheet has "no information" — check for embedded newlines in CSV fields first.
• Excerpt lengths cluster at an exact value (all 1,204 / 1,500 / 800 chars) or end with "[…]" — that is a hard cap, not natural length.
• The ruling question is "does the record/text show X?" — the ruling basis must be byte-identical to what the classifier/raters saw, or the ruling grades a different object.

Solution

1. Trace the true rating basis from the scoring code, not from intermediates. Find the line where text is passed to the model (e.g. prompt = ... + text) and reproduce that construction exactly. Verify: stored excerpt == full_text[:cap] (0 mismatches), or cache-key match against the LLM request cache.
2. Deliver .xlsx, not .csv, for any sheet with multi-line text: wraptext + frozen header + generous width on evidence columns; data-validation dropdown on the ruling column with the exact allowed codes (e.g. Epos/E_neg/unclear, or 0/1/2/unclear).
3. Handle the Excel cell cap (32,767 chars) explicitly: if full text exceeds it, inline the first ~30k ending with a loud marker ("[CONTINUES — see companion file]") AND write the complete text to a per-case .txt (header block with case metadata + full text); add a text_file path column. Write companion files for ALL cases regardless — they are the comfortable reading surface.
4. Keep the original CSV untouched as the machine-readable artifact; the xlsx is the human ruling surface; harvest rulings from the xlsx.

Distribution: text_file paths are project-relative and break the moment the sheet is emailed or copied out of the tree (the labeler asks "where are the attendant/companion files?"). If the labeler is not working inside the project folder, ship ONE zip containing the xlsx and the companion-texts folder side by side, with the instruction: extract together; the file for row X is \.txt next to the spreadsheet. Require the FILLED xlsx back and harvest from the returned copy, not the original.

1. Tell the labeler the epistemics: rule from the provided record only; silence = negative; outside research (Google) goes in ruling_notes, never the ruling — otherwise the labels leak information the classifier could never see.

Verification

• No excerpt column has >30% of rows at one exact length; no "[…]" markers remain.
• pd.read_excel round-trip shows full lengths (compare min/median/max vs the old excerpts).
• Dropdown rejects free-text entries; a saved test ruling survives reopen.

Example

Specialist Directors US, 2026-07-08: director sheet excerpts capped at ~1,200 chars (full dossiers up to 4,090); filing sheet capped at 1,500 chars while raters had scored the whole ~80k-char Item 1A — the PI caught both mid-sitting. Fix: src/director_v1/make_sitting_xlsx.py, tools/make_adjudication_v2_xlsx.py (full-text xlsx + 76 companion txts + dropdowns).

Notes

• Uniform-length clustering is the fastest tell; check it BEFORE handing any sheet to a human.
• Harvest gotcha: when comparing harvested rulings to prior labels, coerce BOTH sides to numeric first — Excel/pandas round-trips floats as "1.0" vs the dropdown's "1", so a string compare falsely flags every row as a disagreement.
• If the pipeline caches LLM requests content-addressed, verify the rebuilt basis against the cache rather than trusting a deterministic builder to have been stable.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kennethkhoocy
• Source: kennethkhoocy/applied-micro-skills
• License: MIT

Install and usage instructions live in the source repository linked above.