Skillscope Analyzer

SkillScope Analyzer

Purpose

Analyze the selected skill-use window only. Current chat or whole-machine history is out of scope unless an input file explicitly points to it.

SkillScope provides file paths for the selected skill, normalized skill units, normalized trace events, raw trace text, and output artifacts. Follow those files, not assumptions.

Required Workflow

1. Read all paths named in the launch prompt.
2. Read the selected SKILL.md and the normalized skill units.
3. Compile the skill into analysis IR: constraints, condition predicates, obligations, prohibitions, ordering rules, and output contracts.
4. Start from constraint-seed.json when present; review and refine it instead of re-deriving every line from scratch.
5. Write or update constraints.json as soon as the constraint review is complete.
6. Write or update skill-graph.json after extracting constraints and before judging the full trace.
7. Compile the trace into trace-facts.json: event facts, artifact facts, ordering facts, and final-output facts.
8. Read native-verifier.json when the launch prompt provides it. Treat it as deterministic evidence for final-output/artifact contracts only.
9. Run path-sensitive coverage over the skill graph using trace facts plus native verifier facts.
10. Update graph branch/path state after trace inspection.
11. Append concise progress notes to progress.md after each meaningful stage.
12. Write findings.json when you have coverage judgments worth surfacing.
13. Keep this pass focused on analysis artifacts; SkillScope launches skillscope-optimizer after findings.json is saved.
14. Finish with a human-readable report citing evidence event IDs and native verifier facts where used.

Program Analysis Model

Treat the selected SKILL.md as a small specification program and the selected trajectory as one execution trace of that program.

Use this compiler-style analysis pipeline:

1. Skill front-end

• Parse Markdown headings as modules / basic blocks.
• Parse bullets, numbered items, tables, examples, and output schemas into fine-grained constraints.
• Classify constraints into condition, obligation, prohibition, ordering, numeric_bound, command_contract, file_contract, evidence_requirement, and final_output_contract.
• Also classify each constraint target as final_output, artifact, process, tool_use, reporting, or unknown.
• Classify whether each constraint is a hard contract, soft recommendation, or implementation hint. Treat illustrative/reference code internals as hints unless surrounding prose says they are mandatory.
• Keep precise source spans. If one sentence contains two obligations or an obligation plus a condition, split it.

1. Skill CFG / control dependence

• Build a graph where headings contain constraints and conditions guard downstream obligations.
• Model "if/when/for X/unless/only when/before/after" as branch predicates.
• A condition may dominate multiple obligations. If the condition is not taken, dominated obligations should usually become not_applicable, not missed.
• A condition that is taken but whose dominated obligations are absent should create missed findings for those obligations.

1. Trace fact extraction

• Compile trace-events.json into facts before making judgments.
• Facts should be structural, not just keywords: commands executed, tools called, files read, files edited, paths mentioned, final JSON fields, output shape, user interruption boundaries, event ranges inspected, and discovered candidate artifacts.
• Treat lexical matches as retrieval hints only. A keyword hit may nominate an event to inspect, but it never proves coverage by itself.
• For each candidate event, read enough surrounding events, tool outputs, file/artifact facts, and final-response fields to decide whether the semantic contract was actually satisfied.
• Write these facts to trace-facts.json so the UI and developer can inspect the intermediate representation.

1. Path-sensitive coverage

• Evaluate branch predicates against task context and trace facts.
• Mark graph branch state as taken, not_taken, checked, or unknown.
• For each reachable obligation, look for satisfying facts after the branch was taken and before the skill-use window ended.
• For prohibitions, look for counterexample facts.
• For ordering rules, compare event indices, not text similarity.
• For numeric/output contracts, parse values where possible and compare exact counts/fields.
• For final-output or artifact contracts, use native verifier facts as hard evidence when available.
• Use native verifier pass/fail only for final-output/artifact contracts, not process-adherence constraints such as exact tool sequence, scoring order, or whether validation happened before writing.
• Treat "validate before writing", "run check before final answer", "call verifier before emitting", and other before/after validation-order rules as target: "process" or target: "tool_use", not target: "final_output". Native verifier pass can prove the final artifact is valid, but it cannot prove the required validation order occurred.

1. Evidence slicing

• Each finding should cite a minimal slice: branch evidence, satisfying evidence or counterexample evidence, and terminal/final evidence for ignored obligations.
• covered and violated need direct evidence event IDs.
• missed should cite the inspected range or terminal event when the absence is established.
• If reachability or facts are incomplete, use unknown.

1. Optimization handoff

• Leave skill text editing to the optimizer pass.
• Make violated and missed findings precise enough for skillscope-optimizer to turn them into a minimal patch.
• Include branch state, evidence event IDs, and suggested rewrite hints only when the trace supports them.

This means the analysis is not "read everything and summarize." It is closer to:

skill source -> constraint IR -> skill CFG
trace events -> trace facts -> observed path
CFG x facts -> followed / violated / ignored / not-taken / unknown

Prefer deterministic structural checks whenever facts make them possible. Use reasoning to design and apply the checks, not to replace the checks with vague impressions.

Constraint Extraction

Extract constraints from the skill, not from keywords alone. If SkillScope provides constraint-seed.json, treat it as a draft span index generated from the selected skill. Keep good seed entries, split or merge imprecise entries, and add missing constraints that matter for trace checking.

The extraction must be coverage-oriented, not summary-oriented. For a long skill, do not select only the most interesting constraints. Build a complete check plan for the selected skill-use window:

• Include every mandatory, prohibited, recommended, ordering, validation, evidence, tool/command, file/path, numeric, and final-response constraint that is observable in the trace.
• Include conditional rules when their trigger can be checked against the task or trace, even if the final status may become not_applicable or unknown.
• Keep examples only when they define an exact required pattern, forbidden pattern, numeric bound, command template, output shape, or file category.
• Treat code blocks labeled or framed as "reference", "example", "fragment", "simplest way", or "helper" as implementation hints by default. Extract their semantic contract, but do not make incidental details mandatory: local variable names, exact helper names, exception message text, arbitrary safety bounds, loop syntax, and tie-breaker order are not hard constraints unless the prose uses MUST/required/final contract language or native verifier evidence maps to that exact detail.
• If a code block contains both a semantic invariant and incidental implementation details, split them: keep the invariant as must, downgrade incidental details to should, may, or exclude them with a note in progress.md.
• Drop only purely explanatory prose, duplicate restatements, trigger text for using the skill, and text that cannot affect behavior in the selected window.
• If you merge seed entries, merge only true duplicates and preserve the narrowest useful source span.
• If you exclude many seed entries, record the exclusion reason and count in progress.md.

As a practical floor: after reading constraint-seed.json, constraints.json should usually retain most non-info seed entries. A result with only a few dozen constraints for a long, rule-heavy skill is suspicious unless the skill is mostly examples or prose. When unsure, keep the constraint and let the coverage status become unknown rather than dropping it.

Prefer precise constraints over sentence-level labels:

• Mandatory actions.
• Prohibited actions.
• Ordering rules.
• Required commands, tools, files, paths, arguments, or flags.
• Numeric thresholds and exact counts.
• Conditions such as "when X happens, do Y".
• Evidence requirements, validation requirements, and final-response requirements.

Mark vague, subjective, or unobservable text as low-confidence or unknown rather than forcing a judgment.

For UI highlighting, each constraint in constraints.json should include:

• unitId: the normalized unit ID from the input.
• text: the exact constraint phrase or the smallest meaningful span.
• kind: a short category such as action, prohibition, order, numeric, command, file_reference, evidence, condition, or other.
• severity: must, should, must_not, may, or info.
• sourceSpan: line and character span in the selected skill when available.
• target: final_output, artifact, process, tool_use, reporting, or unknown when you can classify it.
• rationale: why this should be checked.

This file is a working artifact for UI highlighting, not the final answer. Write it early even if later judgments are still unknown.

Skill Graph

Build a graph representation of the selected skill before doing full coverage judgment. The graph should expose structure that is hard to see in flat highlighting:

• Section/root nodes for major Markdown headings.
• Constraint/action/prohibition/order/numeric/command/evidence nodes for extracted constraints.
• Condition nodes for trigger-dependent rules, language-specific branches, task-shape branches, and "when X, do Y" logic.
• Edges for containment and control flow. Use contains, then, else, condition_true, condition_false, requires, or related.
• A trace path showing which graph nodes the selected trace actually followed.
• Skipped branches for conditions that did not trigger, instead of mixing them into the main highlighter as noisy failures.

Write skill-graph.json with this shape:

{
  "schemaVersion": "skilllens.skill-graph.v1",
  "requestId": "same request id when known",
  "nodes": [
    {
      "id": "graph.node.001",
      "unitId": "skill.unit.12",
      "constraintId": "optional constraint id",
      "parentId": "graph.node.parent",
      "kind": "condition",
      "label": "Language branch",
      "text": "If the changed code is JavaScript or TypeScript, inspect JS/TS sibling modules.",
      "predicate": "changed code is JavaScript or TypeScript",
      "sourceSpan": { "lineStart": 42, "lineEnd": 42, "charStart": 2, "charEnd": 91, "text": "..." },
      "branchState": "taken",
      "status": "covered",
      "confidence": 0.74,
      "evidenceEventIds": ["trace.event.25"],
      "rationale": "The trace selected a TypeScript file, so this branch was active."
    }
  ],
  "edges": [
    { "id": "graph.edge.001", "from": "graph.node.parent", "to": "graph.node.001", "kind": "condition_true", "taken": true }
  ],
  "paths": [
    { "id": "trace.path.main", "title": "Observed trace path", "nodeIds": ["graph.node.001"], "eventIds": ["trace.event.25"] }
  ]
}

Use branchState values:

• taken: the trace entered this condition/branch.
• not_taken: the condition did not trigger for this selected window.
• checked: the node was inspected but is not a conditional branch.
• unknown: the trace was insufficient to know whether the branch applied.

The graph is a working artifact. Update it as trace evidence changes, starting as soon as the skill structure is clear.

Native Verifier Evidence

When native-verifier.json is present, read it before final coverage judgment. This file may contain SkillsBench verifier results, CTRF pytest summaries, failed assertion names, and reward/pass metadata.

Use it with strict scope:

• If the native verifier fails, map failed assertion names and messages to final-output/artifact constraints before judging process constraints. Native failures are the strongest signal for optimization direction.
• If the native verifier passes, final-output and artifact constraints covered by that verifier should not be reported as output-level violated or missed merely because the trace did not show the internal check. Mark them covered with target: "final_output" or target: "artifact" and mention the native verifier evidence.
• If the native verifier fails, map failed tests to related final-output/artifact constraints when the assertion text is specific enough. This can support violated for final-output constraints.
• Native verifier evidence does not prove that the agent followed the specified process. If a skill explicitly requires "run validation before writing", "iterate keys in this order", or "score candidates with this tuple", judge that as process using trace/code evidence.
• If final artifacts pass but process evidence conflicts with the skill, report the process issue as target: "process" and make the rationale clear: final output passed, but process adherence differed.
• If final artifacts pass and the only conflict is with a reference-code helper detail or heuristic tie-breaker, prefer covered for the semantic contract and unknown or low-severity process only when the skill clearly intended exact helper fidelity. Keep helper fidelity gaps separate from primary non-compliance.
• A constraint about the final artifact's fields, schema, feasibility, or score is final_output/artifact. A constraint about when the agent checked those fields, which script it used, or whether it checked before writing is process/tool_use.
• Keep process gaps separate from final-output failures.

Trace Facts

Write trace-facts.json after reading the normalized trace and before final judgment. It is the trajectory IR used by the analysis.

Use this shape when possible:

{
  "schemaVersion": "skilllens.trace-facts.v1",
  "facts": [
    {
      "id": "fact.001",
      "kind": "command_executed",
      "eventIds": ["trace.event.12"],
      "subject": "rg --files packages/jest-config",
      "attributes": { "tool": "exec_command", "cwd": "/repo" },
      "confidence": 0.99
    }
  ],
  "indexes": {
    "commands": ["rg --files packages/jest-config"],
    "filesRead": ["packages/jest-config/src/ValidConfig.ts"],
    "filesEdited": [],
    "toolsCalled": ["exec_command"],
    "finalOutputShape": "json_object_with_functions_array",
    "inspectedEventRanges": ["trace.event.1-trace.event.263"]
  }
}

Useful fact kinds include:

• command_executed
• tool_called
• file_read
• file_edited
• file_mentioned
• search_result_seen
• candidate_discovered
• final_answer_emitted
• final_output_field
• output_shape
• ordering_relation
• human_intervention
• session_boundary
• absence_checked

For ignored obligations, an absence_checked fact is valuable: it records which event range was inspected and which required signal was absent. Use positive conflicting facts for violated; use absence facts for missed.

Trace Inspection

Inspect the selected trace window as an agent behavior record.

Use normalized event IDs when citing ev

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mathieu0905
• Source: mathieu0905/skilllens
• License: MIT

Install and usage instructions live in the source repository linked above.