Skill Audit Router

Skill audit and router

Two stages, in this order, because the second is worthless without the first. Scanning for SKILL.md and calling the result your catalog produces a number that has nothing to do with what the model can actually invoke.

Stage 1 — audit (always run this first)

python3 skill_audit.py --json catalog.json --fix-plan rewrites.md

Resolves the real load path and excludes what was never routable: cross-tool mirrors (.cursor/, .opencode/, .kiro/), marketplace clones, uninstalled plugins, dev checkouts outside the load path, byte duplicates, and name-shadowed skills. Plugin liveness comes from installed_plugins.json, the only authoritative source.

Then scores each description on static signals (length band, trigger phrasing, negative scoping, concrete anchors, filler penalties) plus three corpus-aware signals that are the ones that actually predict misrouting:

• Self-retrieval — query the index with a skill's own description. If it does

not rank itself first, no router will ever pick it correctly.

• Distinctiveness — mean idf of its top terms.
• Margin — lead over its nearest neighbour.

Stage 2 — route

python3 skill_router.py --catalog catalog.json "our cancel flow is leaking subscribers"
python3 skill_router.py --catalog catalog.json --eval evals.jsonl
python3 skill_router.py --catalog catalog.json --for-query "..."   # shortlist card

BM25 over a weighted bag (name ×3, triggers ×2, keywords ×2, description ×1), with hapax damping (a term in exactly one document is damped to 35%, so an incidental rare word cannot rocket a skill to the top) and negative triggers (terms a description disclaims are subtracted, not merely un-added).

The decision policy is the product. Four gates must pass before invoke: query mass, raw floor, coverage ≥0.30, and ≥2 matched terms. The floors are multiples of mean idf, not constants, so they scale with catalog size.

The fix loop

1. skill_audit.py --json catalog.json --fix-plan rewrites.md
2. Rewrite flagged descriptions in this shape:

what it produces. Use when `. Do NOT use for `.

1. Re-run the audit, confirm the score moved.
2. skill_router.py --eval evals.jsonl before and after.

Read the eval numbers correctly

top-1 accuracy : ranked first
auto-invoked   : ranked first AND confident   (rank ≠ fire)
top-3 recall   : reachable via shortlist
NONE (clean)   : returned NONE outright
NONE (safe)    : never auto-invoked           <- DEFEND THIS AT 100%

Auto-invoking on small talk hijacks the turn; returning ambiguous costs a few hundred tokens. Those failures differ in cost by an order of magnitude. Chasing clean-NONE to 100% pushes thresholds conservative enough to suppress real matches.

Hard-won cautions

• Descriptions may not be the problem. Check for an AUTO-GENERATED banner and

a sibling SKILL.md.tmpl or a second copy of the tree first — a stale copy looks exactly like bad writing. Recovering one recovered 16k chars in a real case.

• Enriching descriptions changes every other skill's score. BM25 floors are

multiples of mean idf. Re-run the eval after bulk edits; a real case saw safe-NONE drop 100%→86% from improvements alone. Fix over-triggering at the source with negative scoping, not by loosening constants.

• Never put a backup inside the skills tree — the audit (and the runtime) will

scan it as real skills.

• Use it as a shortlister, not a prompt card. --card renders the whole catalog;

--for-query retrieves first and describes only survivors, at ~3% of the tokens. Let the model make the final call — it is better at intent than any lexical scorer.

Full design notes, scoring detail and the list of traps already handled: REFERENCE.md in this directory.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: rushindrasinha
• Source: rushindrasinha/skill-audit-router
• License: MIT

Install and usage instructions live in the source repository linked above.