Skill Auditor

Skill Auditor

Portfolio-level skill routing analysis and optimization. Analyzes real session transcripts to find routing errors, attention competition, and coverage gaps, then generates an interactive HTML report.

Prerequisites

• tiktoken is optional — token counting falls back to character-based estimation.
• No external API keys required. Analysis uses the host agent's sub-agent capability: Claude Code's Agent tool or Codex's spawn_agent.

Workflow

Run all steps sequentially. The coordinator (you) manages data flow between scripts and sub-agents.

Step 0: Initial Questions

Before starting, ask the user three questions. Use AskUserQuestion in Claude Code, or normal user input in Codex:

1. Provider: "どの環境を分析しますか？" — Claude Code / Codex
2. Report language: "レポートの言語は？ (e.g. 日本語，English, 中文，...)" — Free text input. Default to the user's conversation language if not specified.
3. Scope: "分析範囲はどうしますか？" — Cross-project (all projects) / Current project only

Store these choices. Pass the language choice to all sub-agents as an instruction prefix: "Write all output text (healthassessment, detail, reason, suggestedfix, etc.) in [chosen language]."

Use --provider claude for Claude Code or --provider codex for Codex. For cross-project mode, use "all" as the project_path argument in Step 3. For current-project mode, use --cwd "$(pwd)".

Step 1: Detect Project

If cross-project mode was selected:

uv run python scripts/collect_transcripts.py all --provider  --days 14 \
  --output /transcripts.json --verbose

If current-project mode:

uv run python scripts/collect_transcripts.py --cwd "$(pwd)" --provider  --days 14 \
  --output /transcripts.json --verbose

If auto-detection fails, show the list and ask the user which project to audit.

For cross-project mode, base dir: ~/./skill-report/all/. For current-project mode, base dir: ~/./skill-report/projects//, where `` includes the nearest git root name and the audited path relative to that root with path separators changed to hyphens.

Step 2: Set Up Workspace

Each run gets a timestamped subdirectory so multiple runs never collide:

RUN_ID=$(date +%Y-%m-%dT%H-%M-%S)
WORKSPACE=/${RUN_ID}
mkdir -p ${WORKSPACE}

Use ${WORKSPACE} as ` in all subsequent steps. healthhistory.json stays at /healthhistory.json` (shared across runs — see Step 8).

Step 3: Collect Data

Run both scripts. They produce the input files for analysis.

# Transcripts already collected in Step 1

uv run python scripts/collect_skills.py --provider  \
  --output /skill_manifest.json --verbose

Report the collection summary to the user: "N sessions, M user turns, K skills found. Attention budget: T tokens total."

Step 4: Routing Audit (Sub-agents)

Spawn one or more routing-analyst sub-agents. Each sub-agent:

1. Reads agents/routing_analyst.md for its analysis rubric
2. Reads a filtered skill manifest (only skills visible to that batch)
3. Reads a batch of transcripts
4. Writes analysis to a batch JSON file

IMPORTANT — Project-aware batching: Projects with local skills must be batched separately. Projects with only global skills can be pooled together (they see the same skill set). When many projects have unique local skills, batches are capped at MAX_BATCHES (default 12). Excess groups are merged by greedy similarity — the group with the fewest extra skills is merged into the most similar existing batch. This adds a few extra skills to visible_skill_names but keeps sub-agent count bounded.

import json, math
from collections import defaultdict

data = json.load(open("/transcripts.json"))
manifest = json.load(open("/skill_manifest.json"))
sessions = data["sessions"]

# Identify global skills and project-local skills
global_skills = [s for s in manifest["skills"] if s["scope"] == "global"]
global_names = [s["name"] for s in global_skills]
project_local = defaultdict(list)  # project_path -> [skill dicts]
for s in manifest["skills"]:
    if s["scope"] == "project-local" and s.get("project_path"):
        project_local[s["project_path"]].append(s)

# Helper: does this encoded project_dir match a project_path with locals?
def find_local_skills(project_dir):
    for pp, skills in project_local.items():
        encoded = pp.replace("/", "-").replace(".", "-")
        if encoded.lstrip("-") in project_dir.lstrip("-"):
            return skills
    return []

# Separate sessions: projects with local skills vs global-only
global_only_indices = []  # can be pooled
local_project_groups = defaultdict(list)  # project_dir -> indices

for i, s in enumerate(sessions):
    pdir = s.get("project_dir", "unknown")
    locals = find_local_skills(pdir)
    if locals:
        local_project_groups[pdir].append(i)
    else:
        global_only_indices.append(i)

# Build batches
batch_size = 60
MAX_BATCHES = 12  # Cap total sub-agents to keep cost/time bounded
batches = []

# 1) Pool all global-only sessions together
for chunk_start in range(0, len(global_only_indices), batch_size):
    chunk = global_only_indices[chunk_start : chunk_start + batch_size]
    batches.append(
        {
            "session_indices": chunk,
            "label": "global-only (mixed projects)",
            "visible_skill_names": global_names,
        }
    )

# 2) Group projects with same local skill set, then batch together
by_skill_set = defaultdict(list)  # tuple of local names -> indices
for pdir, indices in local_project_groups.items():
    local_names = tuple(sorted(s["name"] for s in find_local_skills(pdir)))
    by_skill_set[local_names].extend(indices)

local_batches = []
for local_names, indices in by_skill_set.items():
    visible = global_names + list(local_names)
    for chunk_start in range(0, len(indices), batch_size):
        chunk = indices[chunk_start : chunk_start + batch_size]
        local_batches.append(
            {
                "session_indices": chunk,
                "label": f"local skills: {', '.join(local_names[:3])}{'...' if len(local_names) > 3 else ''}",
                "visible_skill_names": visible,
                "_local_set": set(local_names),
            }
        )

# 3) Merge if too many batches — greedily merge smallest into most similar
remaining_budget = MAX_BATCHES - len(batches)
while len(local_batches) > remaining_budget and len(local_batches) > 1:
    # Find the smallest batch
    smallest_idx = min(range(len(local_batches)), key=lambda i: len(local_batches[i]["session_indices"]))
    smallest = local_batches.pop(smallest_idx)
    # Find the most similar batch (fewest extra skills added)
    best_idx, best_extra = 0, float("inf")
    for j, b in enumerate(local_batches):
        extra = len(smallest["_local_set"] - b["_local_set"]) + len(b["_local_set"] - smallest["_local_set"])
        if extra  3 else ''}"

# Clean up internal field and add to batches
for b in local_batches:
    b.pop("_local_set", None)
    batches.append(b)

for i, b in enumerate(batches):
    print(f"Batch {i}: {len(b['session_indices'])} sessions, {len(b['visible_skill_names'])} skills — {b['label']}")

Before spawning, build a DMI list per batch from the manifest:

dmi_skills = {s["name"] for s in manifest["skills"] if s.get("disable_model_invocation")}
for b in batches:
    b["dmi_skill_names"] = sorted(set(b["visible_skill_names"]) & dmi_skills)

Spawn sub-agents in parallel — one per batch. In Claude Code, use the Agent tool. In Codex, use spawn_agent with a bounded task prompt and wait for the batch JSON files before merging.

For each batch i:
  Sub-agent:
    "Read agents/routing_analyst.md from the skill-auditor skill directory for
     your analysis instructions.
     Read /skill_manifest.json for skill definitions.
     Read /transcripts.json for session data.
     Only analyze sessions with these indices: [list from batch].
     Only evaluate against these skills: [visible_skill_names from batch].
     Ignore skills not in this list — they are not available in this
     project context.
     These skills have disable-model-invocation: true and NEVER auto-fire:
     [dmi_skill_names from batch]. Do NOT flag them as false_negative.
     Write your analysis as JSON to /batch_audit_.json
     following the exact schema in schemas/schemas.md (audit_report.json section)."

After all sub-agents complete, merge batch results:

• Union all skill_reports (combine incidents, recalculate stats per skill)
• Union all competition_pairs and coverage_gaps
• Recalculate meta totals (sum sessionsanalyzed, turnsanalyzed, etc.)

Write merged result to /audit_report.json.

Step 5: Portfolio Analysis (Sub-agent)

Spawn a portfolio-analyst sub-agent. In Claude Code, use the Agent tool. In Codex, use spawn_agent.

Sub-agent:
  "Read agents/portfolio_analyst.md from the skill-auditor skill directory.
   Read /skill_manifest.json for skill definitions and attention budget.
   Read /audit_report.json for the routing audit results.
   Write your portfolio analysis as JSON to /portfolio_analysis.json."

Step 6: Improvement Plan (Sub-agent)

Spawn an improvement_planner sub-agent. In Claude Code, use the Agent tool. In Codex, use spawn_agent.

Sub-agent:
  "Read agents/improvement_planner.md from the skill-auditor skill directory.
   Read /audit_report.json for routing audit results.
   Read /portfolio_analysis.json for portfolio analysis.
   Read /skill_manifest.json for current skill definitions.
   IMPORTANT: Write ALL output text in [chosen language] — this includes
   fixes_issues, changes_made, cascade_risk, expected_impact, rationale,
   suggested_description, and every other human-readable string field.
   Write your improvement proposals as JSON to /improvement_proposals.json.
   Also write individual patch files to /patches/ directory."

Step 7: Generate HTML Report

uv run python scripts/generate_report.py \
  --workspace 

Output: /skill_audit_report.html. Open the report in the browser. In Codex, prefer the Browser Use plugin for local files when available:

open /skill_audit_report.html

Step 8: Update Health History

Read /health_history.json (create if doesn't exist — start with empty array []). Append a new entry with the current run's summary:

{
  "timestamp": "",
  "sessions_analyzed": ,
  "turns_analyzed": ,
  "portfolio_health": "",
  "routing_accuracy_avg": ,
  "total_description_tokens": ,
  "competition_conflicts": ,
  "coverage_gaps": ,
  "skills_audited": ,
  "patches_proposed": 
}

If there's a previous entry, show the delta: "Accuracy changed from X to Y."

Step 9: Apply Patches (User Approval)

Show the user a summary from the HTML report. For each patch, show the before/after diff and cascade risk. Let the user approve or reject each.

For approved patches:

uv run python scripts/apply_patches.py \
  --patches /patches/ --confirm \
  --output /changelog.md

Step 10: Summary

Report what was done:

• How many sessions analyzed
• How many routing issues found
• Portfolio health score
• Patches proposed / approved / applied
• New skills suggested
• Link to the HTML report

Analysis Capabilities

Routing Accuracy

Per-skill fire count, accuracy, false positives/negatives, specific incidents with root cause analysis. See agents/routing_analyst.md for the rubric.

Attention Budget

Total description tokens across all skills. Per-skill token cost and efficiency rating. Identifies bloated descriptions that waste attention budget. See agents/portfolio_analyst.md.

Competition Matrix

Classifies skill-pair relationships: orthogonal / adjacent / overlapping / nested. Based on real transcript evidence, not just keyword overlap.

Portfolio-Aware Optimization

Patches consider the full skill set. Cascade checking is mandatory — each patch states what it fixes, what it might break, and the token budget impact. See agents/improvement_planner.md.

Error Taxonomy

| Verdict | Description | | --- | --- | | correct | Right skill loaded for the intent | | falsenegative | Skill should have loaded but didn't. High bar: task must be meaningfully worse without it | | falsepositive | Skill loaded but was irrelevant | | confused | Wrong skill loaded instead of the correct one | | noskillneeded | No skill was needed for this turn (most common) | | explicitinvocation | User explicitly called /skill-name — not a routing event, skip from accuracy calc | | coveragegap | User intent not covered by any existing skill |

Note on disable-model-invocation: true: Skills with this flag never auto-fire by design. They are excluded from false_negative analysis and listed separately in the report as "explicit-only" skills.

Workspace Structure

/                          # e.g. ~/.codex/skill-report/all/ or ~/.codex/skill-report/projects//
├── health_history.json              # shared across runs (append-only)
├── 2026-03-04T18-45-23/             # run 1
│   ├── transcripts.json
│   ├── skill_manifest.json
│   ├── batch_audit_*.json
│   ├── audit_report.json
│   ├── portfolio_analysis.json
│   ├── improvement_proposals.json
│   ├── patches/*.patch.json
│   ├── skill_audit_report.html
│   └── changelog.md
└── 2026-03-04T20-12-07/             # run 2
    └── ...

Troubleshooting

• "No project found": Run with --cwd pointing to the project root, or use --list to see available projects.
• tiktoken not installed: Token counts will use character-based approximation. Install with pip install tiktoken for accuracy.
• Large project (100+ sessions): Sessions are batched automatically. Multiple sub-agents run in parallel.
• Sub-agent produces invalid JSON: Re-run the specific sub-agent step. The rubric in agents/ includes exact schema specifications.
• Codex sub-agent hits a transient usage/approval limit while writing output: Re-run the same sub-agent prompt after the limit clears. If only one artifact is missing and rerun is blocked, the coordinator may generate that missing JSON from the same inputs, then continue with merge/report.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: furedea
• Source: furedea/agent-harness
• License: MIT

Install and usage instructions live in the source repository linked above.