Skill Conductor

Skill Conductor

Full lifecycle management for agent skills: draft → test → review → improve → repeat.

One skill to rule them all — from architecture to packaging. The core loop is always the same: write something, test it, see what fails, fix it, test again.

Runtime requirements (pre-flight)

Before any mode that touches scripts (CREATE, IMPROVE, VALIDATE, OPTIMIZE, PACKAGE), run the pre-flight block → references/runtime-setup.md (checks uv, sets UV_BIN/SKILL_CONDUCTOR_DIR, verifies LLM access). If uv is absent, stop and tell the user.

How to communicate

Read context cues. If the user is a skill author iterating on their own work, be direct and technical. If they're new to skills, explain the why behind each step — not just what to do, but why it matters. Default to conversational, not robotic.

• Explain trade-offs when there's a real choice to make
• Use concrete examples over abstract rules
• When something fails, explain the root cause, not just the fix
• Imperative voice in instructions: "Extract the data", not "You should extract"

Modes

Detect mode from context. If ambiguous, ask.

| Mode | When | What happens | | ----------- | ------------------------------------------------ | --------------------------------------------------------------- | | 1. CREATE | "build a skill", "new skill for..." | Full lifecycle: intent → architecture → scaffold → write → test | | 2. IMPROVE | "fix this skill", "it doesn't trigger" | Diagnose → eval loop → self-update loop → iterate | | 3. VALIDATE | "test this skill", "run evals" | Structural checks + trigger testing + BinEval scoring | | 4. REVIEW | "review this skill", third-party assessment | 11-point quality gate, quick and focused | | 5. OPTIMIZE | "improve triggering", "description optimization" | Automated description optimization with train/test split | | 6. PACKAGE | "package for distribution" | Validate + bundle into .skill file |

---

Mode 1: CREATE

Step 1: Capture Intent

Before writing anything, extract 2–3 concrete scenarios.

Ask:

• "What specific task should this skill handle?"
• "What would a user say to trigger it?"
• "What should NOT trigger it?"

Don't move on until you have a clear picture of what the skill does, for whom, and when. This prevents the most common failure: a skill that does something but triggers for the wrong things.

Step 2: Baseline (TDD RED)

Before writing the skill, verify the agent fails without it:

1. Take one scenario from Step 1
2. Run it in a clean session without the skill
3. Document what went wrong — what the agent guessed, what it missed

If the agent already handles it perfectly, the skill is unnecessary. This sounds obvious, but it's the most skipped step and the most valuable one.

Step 3: Architecture

Choose a primary pattern from references/patterns.md (can combine):

| Pattern | Use when | | ----------------------- | ---------------------------------------- | | Sequential workflow | clear step-by-step process | | Iterative refinement | output improves with cycles | | Context-aware selection | same goal, different tools by context | | Domain intelligence | specialized knowledge beyond tool access | | Multi-MCP coordination | workflow spans multiple services |

Choose degrees of freedom — this determines how much control vs. flexibility the skill gives the agent:

| Freedom | When | Example | | ------------------- | ------------------------------------------- | ----------------------- | | Low (scripts) | fragile, error-prone, must be exact | PDF rotation, API calls | | Medium (pseudocode) | preferred pattern exists, some variation ok | data processing | | High (text) | multiple valid approaches, judgment needed | design decisions |

Golden rule: read references/sop-practices.md before authoring or reviewing ANY skill. It holds the canonical 9 authoring principles (universal): pre-flight, no-process-in-description, MOC (SKILL.md = map, not prose), fresh-practitioner author, TWI "why", blind-agent test, inline checklists, one-term-per-concept, cut-the-fat (env/keys OUT of SKILL.md). For procedural skills (business process with branching: request, quote, onboarding, escalation) the same file also has the deep SOP methodology — format selection, 7-step process, procedural checklist.

Step 4: Scaffold

uv run scripts/init_skill.py  --path  [--resources scripts,references,assets]

Or create manually:

skill-name/
├── SKILL.md          # required — the brain
├── scripts/          # deterministic operations (executed, not loaded)
├── references/       # detailed docs (loaded on demand)
└── assets/           # templates, images for output (never loaded)

Step 5: Write SKILL.md

Frontmatter

---
name: kebab-case-name
description: >
  [Purpose in one sentence]. Use when [triggers].
  Do NOT use for [negative triggers].
---

The description is the single most important line. It determines whether the skill gets triggered at all. Rules:

• name: lowercase, digits, hyphens only. No consecutive hyphens. Matches folder name. Max 64 chars
• description: max 1024 chars. No angle brackets. No process/workflow steps
• Start with purpose, then "Use when...", then "Do NOT use for..."
• Don't put workflow in the description — tested: when the description lists process steps, the agent follows it and skips the body entirely

# GOOD: purpose + triggers, no process
description: Analyze Figma design files for developer handoff. Use when user uploads .fig files or asks for "design specs". Do NOT use for Sketch or Adobe XD.

# BAD: process in description (agent skips body)
description: Exports Figma assets, generates specs, creates Linear tasks, posts to Slack.

Body structure

# Skill Name

## Overview

What this enables. 1-2 sentences. Core principle.

## [Main sections]

Step-by-step with numbered sequences.
Concrete templates over prose.
Imperative voice throughout.

## Common Mistakes

What goes wrong + how to fix.

## Troubleshooting (if applicable)

Error: [message] → Cause: [why] → Fix: [how]

Writing rules

• One term per concept. Pick "template" and stick with it — not template/boilerplate/scaffold (Principle 8)
• SKILL.md = map, not prose. Body is a table-of-contents pointing to references; detail lives there (Principle 3)
• No secrets/env in SKILL.md. No keys, passwords, tokens, env values, or user-absolute paths (/home/, /Users/) — reference them, never inline (Principle 9a)
• Progressive disclosure. SKILL.md = brain ( rule (see references/sop-practices.md Principle 5, TWI)

Step 6: Test Cases & Eval Loop

This is the critical step — most failures hide here. Treat it as three sub-phases.

6a. Pre-flight (before spawning anything)

• [ ] evals/evals.json exists with 3–5 prompts (see references/schemas.md)
• [ ] Workspace dir created: -workspace/iteration-1/
• [ ] Each eval has a descriptive name (not just eval-0) and eval_metadata.json
• [ ] Anthropic key for executor subagents is set
• [ ] uv and eval-viewer/generate_review.py are reachable from current working dir

If any item fails — fix before proceeding. A missing workspace dir mid-run loses outputs.

6b. Run loop (do all in one turn)

| What | Key move | Why | |---|---|---| | Spawn with-skill runs | One subagent per eval, skill active, save outputs to iteration-N//with_skill/ | Parallel = same wall time as one run | | Spawn baseline runs in the same turn | Same prompt, no skill (or old version snapshot for IMPROVE), save to without_skill/ or old_skill/ | If you wait, baselines drift in time and aren't comparable | | Draft assertions while runs execute | Pull verifiable statements from eval prompts | Don't waste the 5–15 min of subagent time | | Capture timing on each notification | Save total_tokens, duration_ms to timing.json immediately | Notification is the only source — process per-arrival, don't batch |

6c. Post-run checklist

• [ ] All timing.json files written (one per run)
• [ ] Each run has a grading.json with fields text, passed, evidence (not name/met)
• [ ] benchmark.json aggregated: uv run scripts/aggregate_benchmark.py /iteration-N --skill-name
• [ ] Analyst pass done — see agents/analyzer.md for what to look for (non-discriminating assertions, high-variance evals, time/token tradeoffs)
• [ ] Eval viewer launched: uv run eval-viewer/generate_review.py --skill-name --benchmark
• In headless mode: --static and send file to user
• For iteration 2+: add --previous-workspace
• [ ] User saw the viewer before I started editing the skill

The last bullet is the trap. If you skip user review and "improve" based on your own reading of outputs, you optimize against your taste, not the user's.

Step 7: Verify & Refactor

1. Does the skill trigger automatically for the right queries?
2. Does the agent follow body instructions (not just description)?
3. Does the output meet use case requirements?
4. Does it NOT trigger on unrelated queries?

If any fail → iterate. Find how the agent rationalizes around the skill, plug loopholes, re-verify.

---

Mode 2: IMPROVE

Step 1: Diagnose

Read the existing SKILL.md completely. Identify the problem class:

| Problem | Signal | Fix | | ------------------- | --------------------------- | -------------------------------------------------------- | | Undertriggering | skill doesn't load | add keywords, trigger phrases, file types to description | | Overtriggering | loads for unrelated queries | add negative triggers, be more specific | | Skips body | follows description only | remove process/workflow from description | | Inconsistent output | varies across sessions | add explicit templates, reduce freedom, add scripts | | Too slow | large context | move detail to references/, cut body to `

• Headless/Cowork: use --static instead of live server

1. Review, provide feedback, iterate

Step 3: Self-Update Loop

Drive iteration off failing BinEval questions, not taste. Run the loop:

1. Generate questions and evaluate the skill (see Mode 3 Stage 3 + references/bineval-method.md) → collect failing[]
2. Spawn agents/analyzer.md as note-taker: turn the failing questions + their explanations into generalized, deduped lessons (not one-off patches for a single test case)
3. Apply targeted edits addressing those lessons
4. Re-evaluate. Revert any edit that introduces a NEW failing question.
5. Terminate when failing[] (or its critical subset) is empty, or after 3 iterations. Keep the best result by gate_passed first, then overall S.

Step 3b: Blind Comparison (optional, for major changes)

When you have two meaningfully different versions:

1. Run both versions on the same evals
2. Spawn agents/comparator.md — answers the SAME binary questions for outputs A and B without knowing which skill produced which
3. Comparator reports per-dimension yes-rate for each version; winner = higher overall yes-rate, tiebreak = critical-dimension yes-rate
4. Spawn agents/analyzer.md — unblinds results, analyzes WHY the winner won
5. Apply insights to improve the losing version

This prevents bias. The comparator judges output quality, not skill design.

---

Mode 3: VALIDATE

Three stages, run in order.

Stage 1: Structural Validation

uv run scripts/eval_skill.py 

Checks: frontmatter, naming, description quality, process leak detection, body size, structure, scripts. Target: 10/10, no warnings.

Stage 2: Discovery (trigger testing)

Generate 6 test prompts:

• 3 that SHOULD trigger the skill
• 3 that should NOT (similar-sounding but wrong domain)

Run each in clean session. Target: 6/6 correct.

For automated trigger testing at scale, use:

uv run scripts/run_eval.py --eval-set  --skill-path  --runs-per-query 3

Stage 3: BinEval Scoring

Evaluate with atomic binary yes/no questions across 5 dimensions — each answered 1/0 with evidence. See references/bineval-method.md for the method, references/quality-questions.md for the question bank, and agents/bineval.md for the evaluator that emits bineval.json.

The 5 dimensions: Discovery, Clarity, Structure, Robustness, Completeness.

Questions come from two sources:

• Deterministic — emitted by scripts/eval_skill.py --json (the sole emitter), e.g. DET-STRUCT-SKILLMD-EXISTS, DET-DISCOVERY-DESC-PRESENT. Some are flagged critical.
• Generated — per-skill binary questions via the two-step meta-prompt (summarize the skill into requirements → decompose each into ≥1 yes/no question with a violation example).

Aggregate: per-dimension dimension_scores S_d = mean of that dimension's answers; overall S = mean of all answers.

Display bands: S≥0.90 production-ready · 0.70–0.89 solid · 0.50–0.69 needs-work · \ --model claude-sonnet-4-20250514 \ --max-iterations 5 \ --holdout 0.4 \ --verbose

The loop:

- Splits queries into train (60%) and test (40%) to prevent overfitting
- Each iteration: evaluates current description → Claude proposes improvement → re-evaluates
- Improvement model sees only train results (blinded to test)
- Selects the best description by test score
- Opens live HTML report automatically

### Supporting scripts

| Script                           | Purpose                                    |
| -------------------------------- | ------------------------------------------ |
| `scripts/run_eval.py`            | Run trigger evaluation on a description    |
| `scripts/improve_description.py` | Claude proposes improved description       |
| `scripts/generate_report.py`     | HTML visualization of optimization history |
| `scripts/aggregate_benchmark.py` | Statistical aggregation of benchmark runs  |

---

## Mode 6: PACKAGE

1. Run REVIEW checklist (Mode 4)
2. Validate:

```bash
uv run scripts/quick_validate.py 

1. Package:

uv run scripts/package_skill.py  [output-dir]

Creates skill-name.skill (zip with .skill extension). Verify: unzip in temp dir, check structure intact.

---

Quick Reference

Skill categories

1. Document/Asset Creation — consistent output (docs, designs, code)
2. Workflow Automation — multi-step processes with methodology
3. MCP Enhancement — workflow guidance on top of tool access
4. Procedural / Process — business procedures with decision points and exceptions (handling a request, generating a quote, processing an invoice, onboarding, escalation). For these → read references/sop-practices.md

File purposes

| Directory | Loaded? | Purpose | | ----------- | -------------------- | ------------------------ | | SKILL.md | on trigger | brain — instructions | | references/ | on demand | detailed docs, schemas | | scripts/ | executed, not loaded | deterministic operations | | assets/ | never loaded | templates, images |

Progressive disclosure budget

| Level | When loaded | Budget | | ----------------- | ---------------------- | ---------- | | Frontmatter | always (system prompt) | ~100 words | | SKILL.md body | on trigger | <500 lines | | Bundled resources | on demand

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: smixs
• Source: smixs/skill-conductor
• License: MIT
• Homepage: https://t.me/aimastersme

Install and usage instructions live in the source repository linked above.