Trueskill Rank

TrueSkill Rank

Rank any collection of text items by quality using TrueSkill + LLM-as-judge.

Setup

pip install trueskill

• Python 3.11+ required
• agent-mux for parallel dispatch (optional -- falls back to direct OpenAI API if OPENAI_API_KEY is set)
• Claude Code: copy this skill folder into .claude/skills/trueskill-rank/
• Codex CLI: append this SKILL.md content to your project's root AGENTS.md

For the full installation walkthrough (prerequisites, verification, API fallback), see [references/installation-guide.md](references/installation-guide.md).

Staying Updated

This skill ships with an UPDATES.md changelog and UPDATE-GUIDE.md for your AI agent.

After installing, tell your agent: "Check UPDATES.md in the trueskill-rank skill for any new features or changes."

When updating, tell your agent: "Read UPDATE-GUIDE.md and apply the latest changes from UPDATES.md."

Follow UPDATE-GUIDE.md so customized local files are diffed before any overwrite.

---

Quick Start

PYTHON="python3"
CLI="~/.claude/skills/trueskill-rank/scripts/trueskill-rank.py"

# Full pipeline: prepare + dispatch + aggregate
$PYTHON $CLI run \
  --input items.json \
  --overlap 3 \
  --rubric ~/.claude/skills/trueskill-rank/rubrics/practitioner-signal.md \
  --output results.json

# Or step by step:
$PYTHON $CLI prepare --input items.json --overlap 3 \
  --rubric ~/.claude/skills/trueskill-rank/rubrics/practitioner-signal.md \
  --output-dir /tmp/ts-run/
# Dispatch is handled internally by trueskill-rank.py (no separate script needed)
$PYTHON $CLI aggregate --run-dir /tmp/ts-run/ --output results.json

Cost: each subset of 10 items produces C(10,2)=45 implicit pairwise comparisons. 100 items at overlap 3 = 30 subsets = 30 API calls = 1,350 implicit comparisons.

Decision Tree

Mode Selection

| Question | Answer | Mode | |----------|--------|------| | Ranking individual items (messages, posts)? | Yes | --mode batch (default) | | Comparing entities (channels, sources, candidates)? | Yes | --mode pairwise | | Items > 50? | Yes | --mode batch (far more efficient) | | Items < 20, need binary signal? | Yes | --mode pairwise |

Overlap Selection

| Overlap | When to use | Cost | |---------|-------------|------| | --overlap 2 | Quick scan, low stakes, small sets | Lowest | | --overlap 3 | Default. Good balance of speed and confidence | Medium | | --overlap 4 | High-stakes curation, final rankings | Highest |

Rubric Selection

| Rubric | Use for | |--------|---------| | practitioner-signal.md | General content quality. 6 criteria led by practitioner signal | | signal-serendipity-entropy.md | Content curation emphasizing surprise and cross-domain bridges | | Custom rubric via --rubric | Any domain -- create from example-template.md |

Input Format

{"items": [{"id": "item_001", "text": "...", "metadata": {...}}, ...]}

Source doesn't matter -- TG messages, HN posts, articles, papers, tweets. The id field is required, text is the content to rank, metadata is optional and passed through to output.

CLI Reference

prepare

Generate subsets and prompt files from input items.

$PYTHON $CLI prepare \
  --input items.json \         # Required. JSON with {"items": [...]}
  --mode batch \               # batch (default) or pairwise
  --overlap 3 \                # 2-4, controls statistical robustness
  --subset-size 10 \           # Items per subset (batch) or matchups per batch (pairwise)
  --rubric rubric.md \         # Required. Scoring criteria file
  --output-dir /tmp/ts-run/ \  # Where to write subsets.json and prompts/
  --seed 42 \                  # Random seed for reproducibility
  --text-cap 1500              # Max chars per item in prompts

Output: subsets.json + prompts/subset-NN.txt (batch) or prompts/batch-NN.txt (pairwise)

aggregate

Parse dispatch results and run TrueSkill rating.

$PYTHON $CLI aggregate \
  --run-dir /tmp/ts-run/ \     # Directory with subsets.json and results/
  --output results.json        # Final rankings JSON

run

Full pipeline: prepare + dispatch + aggregate.

$PYTHON $CLI run \
  --input items.json \
  --overlap 3 \
  --rubric rubric.md \
  --output results.json

Dispatch runs automatically via the built-in dispatch_workers() function. No external scripts required.

Output Format

{
  "rankings": [
    {"id": "item_001", "mu": 35.2, "sigma": 2.1, "conservative": 28.9,
     "rank": 1, "appearances": 3, "wins": 8, "losses": 2}
  ],
  "stats": {
    "total_items": 100, "subsets": 30, "results_parsed": 30,
    "parse_errors": 0, "coverage_gaps": 0, "mode": "batch",
    "overlap": 3, "rubric": "practitioner-signal"
  }
}

conservative = mu - 3*sigma. This is the ranking key. Penalizes items with few appearances (high uncertainty).

Dispatch

Built into trueskill-rank.py via dispatch_workers(). No external scripts.

Primary: agent-mux --engine codex --model gpt-5.3-codex-spark --reasoning low --effort low (free via GPT subscription). Resolves agent-mux via AGENT_MUX_PATH env var, then which agent-mux, then relative to skill directory. Runs 6 workers in parallel via concurrent.futures.ThreadPoolExecutor.

Fallback: If agent-mux not found, falls back to direct OpenAI API via urllib.request (stdlib, zero deps). Requires OPENAI_API_KEY env var. Uses gpt-4o-mini. Results written as {"success": true, "response": "..."} JSON.

Creating Custom Rubrics

Copy rubrics/example-template.md and fill in:

# Rubric Name

## Criteria (ordered by importance)
1. **CRITERION** -- Description
2. **CRITERION** -- Description

## Tiebreaker
How to break ties.

## Context
What kind of content this is for.

3-6 criteria recommended. Order matters -- most important first.

---

Anti-Patterns

| Do NOT | Do Instead | |--------|------------| | Use overlap 2 for high-stakes curation | Use overlap 3-4 for reliable rankings | | Use pairwise mode for 50+ items | Use batch ranking (far more efficient at scale) | | Skip the rubric file | Always specify a rubric via --rubric | | Run without trueskill installed | pip install trueskill first | | Parse result files manually | Use the aggregate subcommand |

Error Handling

| Problem | Solution | |---------|----------| | trueskill not installed | pip install trueskill | | agent-mux not found + no OPENAI_API_KEY | Install agent-mux or set OPENAI_API_KEY env var | | Parse errors in results | Check result files in results/ dir, retry failed subsets | | Coverage gaps in aggregate output | Increase overlap coefficient (--overlap 3 or --overlap 4) | | Empty items array error | Check input JSON format -- must be {"items": [...]} with at least one item |

---

Bundled Resources Index

| Path | What | When to Load | |------|------|-------------| | ./SKILL.md | Skill runbook (this file) | Always | | ./UPDATES.md | Structured changelog for AI agents | When checking for new features or updates | | ./UPDATE-GUIDE.md | Instructions for AI agents performing updates | When updating this skill | | ./scripts/trueskill-rank.py | Main CLI script -- prepare, dispatch, aggregate | Always (execution) | | ./references/algorithm.md | TrueSkill math, N-player mode, convergence, cost scaling | When tuning parameters or understanding ranking behavior | | ./references/prior-runs.md | Previous runs with statistics and lessons learned | When calibrating overlap, rubrics, or interpreting results | | ./references/installation-guide.md | Detailed install walkthrough for Claude Code and Codex CLI | First-time setup or environment repair | | ./rubrics/practitioner-signal.md | General content quality rubric (6 criteria) | Default rubric for most ranking tasks | | ./rubrics/signal-serendipity-entropy.md | Curation rubric emphasizing surprise and cross-domain bridges | Content discovery and curation | | ./rubrics/example-template.md | Template for creating custom rubrics | When creating a new domain-specific rubric |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: buildoak
• Source: buildoak/fieldwork-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.