Schliff

Schliff — Skill Measurement & Iteration Framework

Constraint + clear metric + disciplined iteration = compounding gains. The composite score measures structural quality (file organization, keyword coverage, eval suite breadth) — not runtime effectiveness. Use --runtime to validate actual behavior.

Quick Start (Only 2 Inputs Required)

/schliff
Target: path/to/SKILL.md
Goal: Make the skill trigger correctly for deployment scenarios

Defaults: Metric=composite_score, Verify=score-skill.py, Iterations=30.

Core Loop (NEVER Pauses)

INPUT: Skill path + GOAL + PRIMARY METRIC + VERIFY method + time budget
SETUP: Read ALL files → Analyze → Generate eval suite → Baseline (#0)
LOOP (N iterations, continues until goal met or budget exhausted):
  Exp N: Review skill + results + git history
  → Pick ONE atomic change (based on gaps + history)
  → Edit SKILL.md or references
  → Commit: "schliff exp-N: [description]"
  → Run VERIFY, compute PRIMARY METRIC
  → Improved? Keep. Worse? Revert. Error? Fix or skip.
  → Append to history/ with diffs
  CONSTRAINT: Fixed iterations prevent infinite loops; autonomous mode =
  NO prompts between iterations, just continuous improvement.

When to Use

• Skill not triggering → Run /schliff on trigger-accuracy metric
• Wrong/incomplete outputs → Set goal, metric = binary eval pass rate
• Harden for edge cases → Focus on edge-coverage metric
• Skill too verbose → Optimize token-efficiency metric
• Don't know what's wrong → Run /schliff:analyze for auto-discovery
• Any custom goal → Define GOAL, pick/create METRIC, set VERIFY command

Do NOT use for creating new skills from scratch — use skill-creator first. Do NOT use for SQL query tuning or prompt template authoring.

Interface: GOAL + METRIC + VERIFY

/schliff
Target: .claude/skills/my-skill/SKILL.md
Goal: Fix skill to handle deployment scenarios correctly
Metric: Binary eval pass rate %
Verify: bash scripts/run-eval.sh
Time budget: 2 hours
Iterations: 30

Regression guards — prevent one dimension from regressing while improving another:

/schliff
Target: .claude/skills/deploy/SKILL.md
Goal: Maximize trigger accuracy
Metric: Trigger pass rate
Verify: python3 scripts/score-skill.py SKILL.md --json
Constraint: efficiency >= 80, composability >= 90

Quality Dimensions (Configurable via --weights)

| Dimension | Metric | How | Limitation | |-----------|--------|-----|------------| | Structure | Frontmatter lint score | score-skill.py | File quality, not instruction correctness | | Trigger accuracy | Keyword overlap | TF-IDF heuristic | Does not predict actual triggering | | Output quality | Eval assertion breadth | Test cases | Does not verify runtime output | | Edge coverage | Edge-case definitions | Edge test suite | Does not verify runtime handling | | Token efficiency | Signal/noise density | score-skill.py | Cannot assess content usefulness | | Composability | Scope boundaries | Static analysis | Cannot verify multi-skill interaction | | Clarity (default) | Contradiction + ambiguity | score-skill.py (--no-clarity to opt out) | Pattern-based, not semantic |

See references/metrics-catalog.md for rubrics.

Custom Metrics

Define any metric via a shell command returning a number:

Metric: "Time to first correct output (ms)"
Verify: time bash scripts/run-eval.sh | grep "passed"

Validate custom metrics by running once before the loop, for example by checking the return code.

Subcommands

| Command | Purpose | |---------|---------| | /schliff:init | Bootstrap eval-suite + baseline | | /schliff | Autonomous loop with GOAL + METRIC | | /schliff:auto | Self-driving auto-improve: deterministic patches in a loop | | /schliff:analyze | Skill analysis, gaps, anti-patterns, baseline | | /schliff:bench | Single evaluation run, current score | | /schliff:eval | Run eval suite, show results | | /schliff:report | Generate improvement summary + diffs | | /schliff:mesh | Scan skills for trigger overlap, broken handoffs, scope collisions | | /schliff:triage | Cluster logged failures, auto-generate fixes | | /schliff:log-failure | Log a skill failure for later triage |

Before the Loop (Setup Phase)

1. Read ALL files — SKILL.md + references + related skills.
2. Parse GOAL + METRIC + VERIFY from input. Use defaults if unspecified.
3. Run baseline — Execute VERIFY, record initial metric as exp #0.
4. Generate eval suite if none exists. Use SKILL.md examples as seeds.
5. Validate eval suite — Run once, verify assertions parse correctly.
6. Show gap analysis with estimated iterations. Start NEVER-PAUSE mode on confirm.

Autonomous Loop (Eight-Phase Protocol)

Per references/improvement-protocol.md. Immutable rules:

1. ONE change per experiment. Run git diff to verify scope, because atomic edits isolate causation.
2. Run VERIFY, check number, keep or discard. This prevents subjective drift.
3. Revert on regression: git revert HEAD. This ensures safe experimentation.
4. Re-read ALL files before each change. This prevents contradictions.
5. Descriptive commits: schliff exp-7: add deployment edge cases.
6. Stuck (5+ discards): re-read files, review history, try the opposite. This avoids local optima.
7. Never modify VERIFY during loop. Metric is fixed; skill is the variable.
8. Log everything to history/ — diffs, metrics, keep/discard status.
9. Plateau guard: Every 5 iterations, compare composite against 5-back. Delta 15 points. Fallback to sequential if worktree unavailable.

Noisy Metrics

When metrics fluctuate (>5%): run VERIFY 3x, use median, keep only if improvement > 2x noise floor. Revert to best checkpoint if composite dropped > 2 points despite individual keeps.

Cost Tracking

run-eval.sh --log records duration, tokens, delta, status per run. ROI = delta / iterations_spent. Stop when last 5 iterations gained 10 points:

1. Fix structural issues — Run python3 scripts/score-skill.py SKILL.md --json.
2. Expand triggers — Add synonyms, edge cases, negative boundaries.
3. Add input/output examples — Write 3+ concrete before/after pairs.
4. Add edge-case handling — Test with malformed input, missing context, empty files.
5. Optimize density — Remove redundancy, compress verbose phrasing.
6. Extract references — Move deep content to references/.
7. Verify composability — Check handoff points, run with adjacent skills.

See references/metrics-catalog.md for patterns per dimension.

Example Session

Goal: Trigger accuracy from 60% to 90%
Verify: bash scripts/run-eval.sh | grep "PASS" | wc -l

Exp 1: Add synonyms to description → 65% → Keep
Exp 2: Add negative trigger examples → 70% → Keep
Exp 3: Compress verbose setup section → 68% → Discard (revert)
Exp 4: Add edge case for partial audit → 75% → Keep

Parse history/results.jsonl between sessions. Compare keep rates to prioritize high-ROI changes next session.

Lineage

/skill-creator → v1 → /schliff → autonomous grinding → merge. Roll back via git log --oneline history/. For crashing skills: use systematic-debugging instead, then return to Schliff.

Requirements

Requires Python >= 3.9, Git >= 2.0, jq >= 1.6, Bash >= 4.0. Standard library only. All /schliff:* commands are namespaced. Deterministic scorer, safe to re-run. If scoring fails, returns structured error.

Files

Run ls -R in skill directory. Run python3 scripts/score-skill.py SKILL.md --json for scores. Key files:

• scripts/init-skill.py — Bootstrap eval-suite (--json --dry-run)
• scripts/generate-report.py — Shareable improvement report
• scripts/score-skill.py — Dimension scores incl. runtime (--diff --clarity --weights)
• scripts/text-gradient.py — Invert scorer issues into fix list (--json --top N --apply --dry-run)
• scripts/auto-improve.py — Autonomous loop (--max-iterations N --dry-run --resume)
• scripts/skill-mesh.py — Multi-skill conflict detection (--incremental)
• scripts/meta-report.py — Strategy predictor + auto-calibration
• scripts/episodic-store.py — Cross-session memory (--store --recall --synthesize)
• scripts/parallel-runner.py — Worktree parallel experimentation (--strategies --auto)
• scripts/runtime-evaluator.py — Invoke Claude with test prompts, check output
• scripts/analyze-skill.sh — Legacy linter (score-skill.py has this built-in)
• scripts/run-eval.sh — Run eval suite (--runtime auto-enabled if claude CLI available)
• scripts/progress.py — Convergence charts + strategy analysis (--emit-meta)
• hooks/session-injector.js — SessionStart hook: surfaces untriaged failures
• references/improvement-protocol.md — Full 9-phase loop spec
• references/metrics-catalog.md — Scoring rubrics + custom metrics
• templates/eval-suite-template.json — Eval skeleton for new skills

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Zandereins
• Source: Zandereins/schliff
• License: MIT
• Homepage: https://pypi.org/project/schliff/

Install and usage instructions live in the source repository linked above.