Plugin Introspector

Plugin Introspector — White-box Monitoring & Self-Improvement Meta-Plugin

> A meta-plugin that monitors Claude Code workflow plugin execution at white-box level, > collects execution data, and generates data-driven improvement proposals.

Role

A monitoring and analysis meta-plugin that provides end-to-end visibility into Claude Code workflow plugin execution. Collects tool traces, API interactions, token usage, and execution patterns via hooks. Provides analysis agents that identify bottlenecks, evaluate quality, detect anomalies, and generate concrete improvement proposals.

Terminology

• Plugin: An installable package (directory with .claude-plugin/plugin.json and skills/)
• Skill: An invocable unit within a plugin (directory with SKILL.md)
• Agent: A Task subagent definition (.md file in agents/)
• Proposal / Improvement: A concrete, ROI-scored change suggestion generated by improve

Core Principles

1. Data-driven improvement: Never guess — always base decisions on collected execution data
2. Minimal overhead: Hook scripts must complete in Full details: [resources/orchestration-protocol.md](./resources/orchestration-protocol.md)

Pre-Execution: Target Plugin Resolution

When --target {plugin-name} is specified:

1. Search for plugin at: {working_dir}/plugins/{plugin-name}/
2. Fallback: check marketplace.json for source path
3. Catalog plugin components → store in SESSION_DIR/target_plugin.json
4. If not found: display "Plugin '{name}' not found" with available list

When --target is omitted:

• Infer plugin from traces: scan input_summary fields for plugin-specific paths
• If Skill tool invoked: identify from skill name
• If no plugin identified: treat as general session analysis

---

Data Storage

~/.claude/plugin-introspector/
├── sessions/{session-id}/
│   ├── meta.json              # Session metadata (git info, collection_tier)
│   ├── tool_traces.jsonl      # Pre/post tool traces with input_summary
│   ├── api_traces.jsonl       # API request/response metrics
│   ├── otel_traces.jsonl      # OTel spans (hook-generated or merged native)
│   ├── stats.json             # Aggregated session statistics
│   ├── evaluation.json        # Quality evaluation results (from evaluate)
│   ├── security_events.jsonl  # DLP violations, command risk events (from security hooks)
│   └── target_plugin.json     # Discovered target plugin info (from --target)
├── session_history.jsonl      # Cross-session summary records
├── evaluation_history.jsonl   # Cross-session evaluation scores
├── alerts.jsonl               # Anomaly detection alerts
├── improvement_log.jsonl      # Applied improvement history
├── aggregates.json            # Pre-computed cross-session aggregates (from Stop hook)
├── security_baseline.json     # Security baseline for anomaly comparison (optional)
├── telemetry.jsonl            # Opt-in anonymous telemetry (PI_TELEMETRY=1)
├── plugin-profiles/{plugin}/  # Per-plugin workflow profiles and baselines
├── otel-export/               # Tier 1: OTel Collector File Exporter output
└── otel-collector/            # Tier 1: Collector binary and config

---

Commands

Quick Reference

| Command | Agent | Options | |---------|-------|---------| | status | — | --session {id} | | dashboard | — | --session {id}, --full | | flow | — | --session {id} | | profile | plugin-profiler | --target {plugin} (required) | | analyze | workflow-analyzer | --session {id}, --target {plugin} | | tokens | token-optimizer | --session {id} | | api | api-tracker | --session {id} | | context | context-auditor | --session {id}, --target {plugin} | | evaluate | quality-evaluator | --session {id}, --target {plugin} | | alerts | anomaly-detector | --session {id} | | optimize | auto-optimizer | --target {plugin}, --component {file} | | improve | improvement-generator | --session {id}, --target {plugin} (required) | | report | 5 agents (see below) | --session {id}, --target {plugin} | | apply | — | (reads proposals from last improve/optimize) | | trace | — | --tool {name}, --errors, --slow, --last {N} | | web | — | --otel, --json, --csv | | quick-scan | quick-scanner | --target {plugin} (required) | | security-scan | — | --target {plugin} (required) | | security-audit | security-auditor | --session {id} | | security-dashboard | — | --session {id} | | compliance-report | security-reporter | --period {30d\|7d\|date~date} | | rotate-data | — | --dry-run, env: PI_RETENTION_DAYS, PI_RETENTION_LINES | | otel-setup | — | install, start, stop, status, env | | otel-security-map | — | --watch, --stats |

\ available, experimental*

---

Agent Invocation Pattern

> Details: [resources/orchestration-protocol.md](./resources/orchestration-protocol.md)

All analysis agents are invoked via the Task tool using the prompt template defined in orchestration-protocol.md:

1. Read the agent definition: agents/{agent-name}.md
2. Read the relevant JSONL data files from SESSION_DIR (apply truncation per Data Size Management)
3. Construct prompt from Task Prompt Template (see orchestration-protocol.md)
4. Invoke Task with subagent_type: "general-purpose", model per agent's Model Assignment
5. Parse agent's JSON output
6. Display results / write to SESSION_DIR

For multi-agent commands (report, improve, optimize):

1. Read all data files once and reuse across agents
2. For 'report': run independent agents in parallel (workflow-analyzer, token-optimizer, api-tracker, context-auditor), then quality-evaluator
3. For 'improve': run analysis agents + load cross-session data (evaluation_history, improvement_log, plugin profile), aggregate all, pass to improvement-generator with improvement-pipeline.md procedure
4. For 'optimize': read evaluation.json + evaluation_history + improvement_log + target component, pass to auto-optimizer

---

status — Current Session Overview

1. Resolve session directory
2. Read stats.json
3. Display: session ID, duration, tool calls, token estimate, error count, top tools

---

dashboard — htop-style Real-time Dashboard

1. Resolve session directory
2. Run Bash: scripts/dashboard.sh {session-id} [--full]
3. Renders token progress bar, per-tool breakdown, error rate

---

flow — Execution Flow Tree (OTel)

1. Resolve session directory
2. Read otel_traces.jsonl
3. Build parent-child span tree using parent_span_id
4. Display indented tree with timing and token info

---

profile — Generate Plugin Profile

1. Resolve target plugin (required: --target)
2. Read target plugin's main SKILL.md content
3. Read target plugin's component catalog (from target_plugin.json)
4. Read agent: [agents/plugin-profiler.md](./agents/plugin-profiler.md)
5. Invoke Task (haiku): pass SKILL.md content + component catalog
6. Output: workflow type, phases, detection patterns, key files
7. Write result to ~/.claude/plugin-introspector/plugin-profiles/{plugin}/profile.json
8. Create empty phase-baselines.json and learned-patterns.jsonl if not present

---

analyze — Deep Workflow Analysis

1. Resolve session + target plugin
2. Read tool_traces.jsonl + otel_traces.jsonl
3. Read agent: [agents/workflow-analyzer.md](./agents/workflow-analyzer.md)
4. Invoke Task (sonnet): pass agent definition + data
5. Output: patterns, bottlenecks, efficiency ratio, recommendations

---

tokens / api / context — Single-Agent Analysis Commands

All follow the same pattern: resolve session → read data → invoke agent → display output.

| Command | Agent | Data Files | Extra | |---------|-------|------------|-------| | tokens | [token-optimizer](./agents/token-optimizer.md) (sonnet) | tool_traces.jsonl + api_traces.jsonl + stats.json | — | | api | [api-tracker](./agents/api-tracker.md) (sonnet) | api_traces.jsonl | — | | context | [context-auditor](./agents/context-auditor.md) (sonnet) | api_traces.jsonl + tool_traces.jsonl | --target: read SKILL.md + agents for static cost |

---

evaluate — Quality Evaluation (LLM-as-Judge)

1. Resolve session + target plugin (optional: --target scopes evaluation)
2. Read tool_traces.jsonl + otel_traces.jsonl + stats.json
3. Read agent: [agents/quality-evaluator.md](./agents/quality-evaluator.md)
4. Invoke Task (sonnet): pass agent definition + data
5. Output: 4-dimension scores + quantified improvement signals
6. Write result to SESSION_DIR/evaluation.json
7. Append to ~/.claude/plugin-introspector/evaluation_history.jsonl:

• From quality-evaluator output: weighted_score, scores
• From stats.json: key_metrics (toolcalls, totaltokensest, errors, errorrate, duration_ms)
• From orchestrator phase detection (via plugin profile + trace analysis): phase_breakdown (optional, only for phased workflows)
• From quality-evaluator improvement_signals: top_waste_sources (top 3 waste entries)
• From improvement_log: improvements_active (IDs where status == "applied")

1. Update phase baselines (if target_plugin has profile with workflow.type == "phased"):

• Read plugin-profiles/{plugin}/phase-baselines.json
• Update running mean/stddev for each phase using current phase_breakdown
• Increment sessions_count, set maturity.baselines_available = true if count ≥ 5

1. Closed Loop Check: Read ~/.claude/plugin-introspector/improvement_log.jsonl

• Find entries where status == "applied" and post_score == null
• If --target specified: only check entries matching target_plugin
• For each pending entry: compare current weighted_score with pre_score
• Update post_score and set status to validated or regressed
• If regressed (score dropped >0.5): suggest rollback via [improvement-apply-protocol.md](./resources/improvement-apply-protocol.md)

---

alerts — Anomaly Detection

1. Read ~/.claude/plugin-introspector/alerts.jsonl → display recent alerts
2. If deeper analysis requested:

• Resolve session
• Read tool_traces.jsonl + session_history.jsonl + alerts.jsonl
• Read agent: [agents/anomaly-detector.md](./agents/anomaly-detector.md)
• Invoke Task (haiku): pass agent definition + data (including alerts.jsonl for deduplication)

1. Output: alert list with severity, suggested actions

---

optimize — Auto-Optimize (APE Loop) (available, experimental)

1. Resolve target plugin (required: --target)
2. Resolve target component (--component or auto-select lowest-scoring)
3. Read evaluation.json (run evaluate first if missing)
4. Read evaluation_history.jsonl + improvement_log.jsonl (for contrastive analysis + historical learning)
5. Read agent: [agents/auto-optimizer.md](./agents/auto-optimizer.md)
6. Read target component file content
7. Invoke Task (opus): pass agent definition + evaluation + history + component content
8. Output: optimized version with diff + predicted score improvement
9. Display diff for user review — do NOT auto-apply

CAUTION: Review all diffs before applying. See [resources/improvement-apply-protocol.md](./resources/improvement-apply-protocol.md).

---

improve — Generate Improvement Proposals

1. Resolve session + target plugin (required: --target)
2. Run analysis pipeline (or reuse if already run in this session):

• Parallel: workflow-analyzer, token-optimizer, context-auditor
• Parallel: anomaly-detector, quality-evaluator

1. Load cross-session data:

• evaluation_history.jsonl (last 20 records)
• improvement_log.jsonl (all for target plugin)
• Plugin profile: profile.json, phase-baselines.json, learned-patterns.jsonl

1. Read target plugin component files (apply file selection per orchestration-protocol.md)
2. Read agent: [agents/improvement-generator.md](./agents/improvement-generator.md)
3. Read procedure: [resources/improvement-pipeline.md](./resources/improvement-pipeline.md)
4. Invoke Task (opus): pass agent definition + pipeline procedure + all analysis results + cross-session data + selected plugin files
5. Output: ROI-scored proposals with quantified evidence, counterfactuals, diffs, meta-rules validation
6. Display proposals for user review — do NOT auto-apply

---

report — Full Analysis Report

1. Resolve session + target plugin
2. Read all JSONL data files once
3. Run independent agents in parallel (4 concurrent Task calls):

• workflow-analyzer → analysis
• token-optimizer → tokens
• api-tracker → api
• context-auditor → context

1. Run quality-evaluator sequentially (can use results from step 3 if needed)
2. Aggregate into unified report with sections:

• Executive Summary, Workflow Analysis, Token Efficiency,

API Performance, Context Audit, Quality Score, Recommendations

---

apply — Apply Improvement Proposals

1. Read the most recent improve or optimize output (proposals)
2. For each proposal, follow [improvement-apply-protocol.md](./resources/improvement-apply-protocol.md):

• Display proposal summary + diff for user review
• Wait for explicit user confirmation
• Backup → Edit → Verify → Log to improvement_log.jsonl

1. After all approved proposals applied:

• Display summary of changes
• Suggest: "Run evaluate --target {plugin} after next session to validate improvements"

CAUTION: Never auto-apply. Each proposal requires explicit user approval.

---

quick-scan — Quick Plugin Diagnosis (1-minute)

1. Resolve target plugin (required: --target)
2. Analyze plugin structure: plugin.json, SKILL.md, agents, scripts, resources
3. Run security-scan.sh for security score
4. Read agent: [agents/quick-scanner.md](./agents/quick-scanner.md)
5. Invoke Task (haiku): lightweight analysis
6. Output: formatted box diagram with structure summary, security score, recommendations
7. Suggest follow-up commands for detailed analysis

Use case: First-time plugin evaluation, PR review, quick health check

> Details: [agents/quick-scanner.md](./agents/quick-scanner.md)

---

rotate-data — Data Retention Management

1. Run Bash: scripts/rotate-data.sh
2. Deletes session directories older than PI_RETENTION_DAYS (default: 30)
3. Trims JSONL files to PI_RETENTION_LINES (default: 1000)
4. Displays summary of deleted sessions and freed space

Environment variables:

• PI_RETENTION_DAYS=30: Days to keep session directories
• PI_RETENTION_LINES=1000: Lines to keep in JSONL files
• PI_DRY_RUN=1: Preview what would be deleted without actually deleting
• PI_AUTO_ROTATE=1: Auto-run at session end (optional)

Example:

# Dry run to see what would be deleted
PI_DRY_RUN=1 /plugin-introspector rotate-data

# Keep only 7 days of data
PI_RETENTION_DAYS=7 /plugin-introspector rotate-data

---

security-scan — Plugin Static Security Analysis

1. Resolve target plugin (required: --target)
2. Run Bash: scripts/security-scan.sh {plugin-path}
3. Scans hook scripts for dangerous patterns (data exfiltration, reverse shells, credential theft)
4. Scans SKILL.md + resources for prompt injection patterns
5. Scans agent definitions for risky tool permission combinations
6. Output: JSON report with findings, severity levels, risk score
7. Logs findings to alerts.jsonl if any found

Environment variables:

• PI_ENABLE_SECURITY=1: Enables runtime command risk logging
• PI_ENABLE_DLP=1: Enables DLP scanning
• PI_SECURITY_BLOCK=1: Enables CRITICAL command blocking (use with caution)

> Note: security-check.sh intentionally omits || true in plugin.json so that exit 2 can block CRITICAL commands when PI_SECURITY_BLOCK=1. All other hook scripts use || true per meta-rules.

> Details: [resources/security-patterns.md](./resources/security-patterns.md)

---

security-audit — Session Security Audit

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: brody-0125
• Source: brody-0125/my-claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.