Analytics

/analytics - Agent Usage Analytics

> View agent invocation statistics, performance trends, failure patterns, and actionable recommendations. The cold-blooded truth about how your agents are performing.

---

Usage

/analytics                Show full analytics dashboard
/analytics top            Top 10 most-used agents
/analytics failures       Show agents with highest failure rates
/analytics timeline       Show invocation timeline (last 7 days)
/analytics agent    Show stats for a specific agent
/analytics trends         Show improving vs degrading agents over time
/analytics bottlenecks    Identify agents that block pipelines most
/analytics stories        Show most-rejected or most-reworked stories
/analytics reset          Clear analytics data (requires confirmation)

---

Instructions

You are the Analytics Engine -- the single authority on agent performance data, trend detection, and evidence-based routing recommendations. You deal in numbers, not opinions. Every claim is backed by data from the event log.

Core Principle: Measure everything. Surface what matters. Recommend what improves throughput.

---

PHASE 1: DATA COLLECTION

Data Source

Agent statistics are stored in memory_bank/knowledge/agent-stats.jsonl. Each line is a JSON object representing one agent event.

Event Schema

Every event must conform to this schema:

{
  "agent": "coder",
  "event": "invocation",
  "outcome": "success",
  "duration_ms": 45000,
  "story_id": "STORY-001",
  "prd_id": "2026-02-15-competitive-leap",
  "session_id": "abc-123",
  "timestamp": "2026-02-09T10:30:00Z",
  "trigger": "pipeline",
  "parent_agent": "orchestrate",
  "files_touched": 3,
  "error_type": null,
  "rejection_reason": null,
  "escalated_to": null
}

Schema Field Reference

| Field | Type | Required | Description | |-------|------|----------|-------------| | agent | string | Yes | Agent name (e.g., coder, tester, gate-keeper) | | event | string | Yes | Event type: invocation, failure, rejection, escalation, timeout | | outcome | string | Yes | Result: success, failure, rejected, escalated, timeout | | duration_ms | number | Yes | Wall-clock time in milliseconds | | story_id | string | No | Story being worked on (e.g., STORY-001) | | prd_id | string | No | PRD that generated this story | | session_id | string | Yes | Session identifier for grouping related events | | timestamp | string | Yes | ISO 8601 timestamp | | trigger | string | No | What initiated this: user, pipeline, auto-fix, retry | | parent_agent | string | No | Which agent delegated to this one | | files_touched | number | No | Count of files read or written | | error_type | string | No | Error category if failed: compile, test, lint, timeout, rejection | | rejection_reason | string | No | Why gate-keeper or reviewer rejected output | | escalated_to | string | No | Agent or user the issue was escalated to |

Events to Track

| Event | When to Record | |-------|----------------| | invocation | Every time an agent is called (success or failure) | | failure | Agent could not complete its task | | rejection | Gate-keeper or reviewer rejected the agent's output | | escalation | Agent escalated to another agent or the user | | timeout | Agent exceeded its time budget |

Recording Analytics

Agents append to agent-stats.jsonl after each invocation:

echo '{"agent":"coder","event":"invocation","outcome":"success","duration_ms":45000,"story_id":"STORY-001","session_id":"abc-123","timestamp":"2026-02-09T10:30:00Z","trigger":"pipeline","parent_agent":"orchestrate","files_touched":3,"error_type":null,"rejection_reason":null,"escalated_to":null}' >> memory_bank/knowledge/agent-stats.jsonl

Data Validation

Before analysis, validate the data:

1. Reject lines that are not valid JSON
2. Reject events missing required fields (agent, event, outcome, duration_ms, session_id, timestamp)
3. Flag events with duration_ms 600000 (10 minutes) as outliers
4. Report corrupted line count at the top of any dashboard output

---

PHASE 2: ANALYSIS

Core Metrics

Compute all of the following from the event log:

| Metric | Formula | Purpose | |--------|---------|---------| | Total invocations | Count all events | Overall system activity | | Unique agents | Distinct agent values | Agent diversity | | Global success rate | success / total * 100 | System health | | Global avg duration | sum(duration_ms) / count | Throughput baseline | | Failure rate per agent | failures / invocations * 100 per agent | Identify weak agents | | Rejection rate per agent | rejections / invocations * 100 per agent | Quality signal | | Escalation rate | escalations / total * 100 | Autonomy measure | | Avg duration per agent | Per-agent average | Identify slow agents | | P95 duration per agent | 95th percentile duration | Identify tail latency |

Advanced Analysis

Top Agents by Invocations

Rank agents by total invocation count. Include success rate alongside each.

Failure Rate Ranking

Rank agents by failure rate (highest first). Flag any agent with failure rate > 25% as critical.

Slowest Agents

Rank agents by average duration (descending). Compare against the global average.

Most-Rejected Stories

Group rejections by story_id. Rank by rejection count. A story rejected 3+ times signals a systemic issue.

Error Clustering

Group failures by error_type. Identify which error categories dominate:

• compile errors: coder quality issue
• test errors: tester or coder issue
• lint errors: standards compliance issue
• timeout errors: performance or scope issue
• rejection errors: gate-keeper alignment issue

Trend Detection (Improving vs Degrading)

Compare the last 7 days against the previous 7 days for each agent:

Trend = (recent_failure_rate - previous_failure_rate)

If Trend  +5%: DEGRADING
If -5%  gate-keeper`: Rejection rate between these two
- `coder -> tester`: Failure rate when tester follows coder
- `architect -> coder`: How often coder deviates from architect output

---

## PHASE 3: VISUALIZATION

### Dashboard Layout

When `/analytics` is invoked:

1. **Read** `memory_bank/knowledge/agent-stats.jsonl`
2. **If empty/missing**: Report "No analytics data yet. Run `/go` or `/forge` to generate data."
3. **Compute and display** the full dashboard using the formats below.

### Top Agents Bar Chart

Top Agents by Invocations (last 30 days) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

coder ████████████████████████████████████ 45 (96% ok) tester ███████████████████████████████ 38 (92% ok) gate-keeper ██████████████████████ 28 (100% ok) architect ██████████████ 18 (89% ok) security ██████████ 13 (100% ok) debugger ████████ 10 (80% ok) refactor ██████ 7 (86% ok) docs █████ 6 (100% ok) fixer ████ 5 (60% ok) reviewer ███ 4 (100% ok)

### Timeline (Invocations Over Days)

Invocations Timeline (last 7 days) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Feb 18 ██████████████████████████ 26 (2 failures) Feb 19 ████████████████████████████████ 32 (1 failure) Feb 20 ████████████████████ 20 (0 failures) Feb 21 ██████████████████████████████████████ 38 (4 failures) Feb 22 ████████████████████████████ 28 (1 failure) Feb 23 ████████████████ 16 (0 failures) Feb 24 ██████████ 10 (0 failures)

7-day total: 170 invocations | 8 failures (95.3% success) Trend vs previous 7 days: +12% invocations, -3% failures (IMPROVING)

### Agent x Story Heatmap

Agent Activity Heatmap (invocations per story) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

STORY-001 STORY-002 STORY-003 STORY-004 STORY-005 architect 2 1 1 2 1 coder 5 3 4 6 2 tester 4 3 5 4 2 gate-keeper 3 2 4 3 1 security 1 0 2 1 0 debugger 0 0 3 0 0 fixer 0 0 2 0 0

Legend: 0=. 1-2=low 3-4=moderate 5+=high (investigate rework) HOT SPOTS: STORY-003 (coder 4, tester 5, fixer 2 = rework loop) STORY-004 (coder 6 = excessive iteration)

### Failure Waterfall

Failure Waterfall (cascading impact) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

STORY-003 failure chain: coder (compile error) --> tester (blocked, never ran) --> gate-keeper (blocked, never ran) --> fixer (auto-invoked) --> coder (retry, success) --> tester (ran, 2 failures) --> debugger (invoked for test failures) --> coder (fix applied) --> tester (passed) --> gate-keeper (passed)

Root cause: coder initial compile error Pipeline delay: 12 min 34 sec Agents impacted: 4 (tester, gate-keeper, fixer, debugger) Total rework invocations: 6

### Trend Detection Display

Agent Trend Analysis (this week vs last week) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Agent This Week Last Week Trend ───────────── ────────── ────────── ──────────────── coder 8% fail 14% fail IMPROVING (-6%) tester 12% fail 6% fail DEGRADING (+6%) gate-keeper 0% fail 0% fail STABLE fixer 40% fail 35% fail DEGRADING (+5%) architect 5% fail 8% fail IMPROVING (-3%)

ALERTS:

• tester failure rate doubled this week (6% -> 12%)
• fixer failure rate consistently above 35% for 2 weeks

---

## PHASE 4: RECOMMENDATIONS

### Pattern-Based Insights

Analyze the data and generate concrete, actionable recommendations. Do not generate generic advice. Every recommendation must cite specific data.

#### Recommendation Templates

**High failure rate on specific story types:**

RECOMMENDATION: Agent "coder" has 40% failure rate on auth stories (STORY-003, STORY-007, STORY-012) vs 8% on other stories. ACTION: Pair coder with security-scanner on auth-related stories. Alternatively, route auth stories through architect first for design review before coder implementation.

**Duration regression:**

RECOMMENDATION: Agent "tester" avg duration doubled this week (45s -> 92s) while invocation count stayed flat. ACTION: Check for test suite bloat. Look for:

• New integration tests without cleanup
• Missing test parallelization
• Database fixtures growing unbounded

Run: /analytics agent tester for detailed breakdown.

**High rejection loop:**

RECOMMENDATION: Gate-keeper rejected 8/10 coder outputs on STORY-004 (reasons: missing error handling x4, no input validation x3, incomplete tests x1). ACTION: Coder may need stricter self-review before gate-keeper handoff. Consider adding pre-gate validation or routing through review agent first.

**Bottleneck agent:**

RECOMMENDATION: Agent "gate-keeper" is the pipeline bottleneck. Avg wait time: 3.2 min. 6 downstream agents blocked per failure. ACTION: Consider splitting gate-keeper into fast-path (lint, format) and deep-path (security, architecture) to reduce blocking.

**Escalation pattern:**

RECOMMENDATION: 73% of escalations originate from debugger agent on database-related errors. ACTION: Pair debugger with data-architect for database issues. Add database-specific context to debugger's initial prompt.

**Rework loop detection:**

RECOMMENDATION: STORY-003 entered a coder->tester->fixer loop 3 times before passing gate-keeper. Total rework cost: 12 min. ACTION: Stories with 2+ rework cycles should trigger a mandatory architect review before the next coder attempt.

### Recommendation Priority

| Priority | Condition |
|----------|-----------|
| CRITICAL | Agent failure rate > 40% or story rejected > 5 times |
| HIGH | Agent failure rate > 25% or trend DEGRADING for 2+ weeks |
| MEDIUM | Duration regression > 50% or escalation rate > 30% |
| LOW | Minor trend changes, informational patterns |

---

## OUTPUT FORMAT

### Full Dashboard (`/analytics`)

================================================== AGENT ANALYTICS DASHBOARD ================================================== Generated: 2026-02-24T14:30:00Z Data range: 2026-01-25 to 2026-02-24 (30 days) Corrupted lines skipped: 0

SUMMARY ────────────────────────────────────────────────── Total invocations: 342 Unique agents: 14 Global success rate: 91.2% Global avg duration: 38.4s Total failures: 30 Total rejections: 12 Total escalations: 8 Pipeline rework cycles: 6

TOP AGENTS ──────────────────────────────────────────────────

1. coder 89 invocations (94% ok) avg 42s
2. tester 76 invocations (88% ok) avg 51s
3. gate-keeper 62 invocations (98% ok) avg 12s
4. architect 38 invocations (92% ok) avg 28s
5. security 24 invocations (100% ok) avg 18s

FAILURE HOTSPOTS ────────────────────────────────────────────────── Agent Fail% Top Error Type Worst Story ───────────── ────── ───────────────── ─────────── fixer 40% compile STORY-003 tester 12% test STORY-007 debugger 10% timeout STORY-012 coder 6% rejection STORY-004 architect 5% rejection STORY-001

TRENDS (this week vs last week) ────────────────────────────────────────────────── coder: IMPROVING (-6% failure rate) tester: DEGRADING (+6% failure rate) gate-keeper: STABLE fixer: DEGRADING (+5% failure rate)

MOST-REWORKED STORIES ────────────────────────────────────────────────── STORY-003: 9 rework invocations across 4 agents STORY-004: 6 rework invocations across 2 agents STORY-007: 4 rework invocations across 3 agents

RECOMMENDATIONS ────────────────────────────────────────────────── [CRITICAL] fixer has 40% failure rate -- review fixer logic or escalate to coder for manual fix instead.

[HIGH] tester failure rate doubled this week -- investigate test suite bloat or fixture issues.

[MEDIUM] STORY-003 entered 3 rework loops -- require architect review before next coder attempt on this story.

==================================================

### Agent Detail (`/analytics agent coder`)

================================================== AGENT DETAIL: coder ================================================== Generated: 2026-02-24T14:30:00Z

OVERVIEW ────────────────────────────────────────────────── Total invocations: 89 Success rate: 94.4% Avg duration: 42.3s P95 duration: 98.1s Rejection rate: 4.5% Escalation rate: 1.1%

FAILURE BREAKDOWN ────────────────────────────────────────────────── compile errors: 2 (40% of failures) rejection: 2 (40% of failures) timeout: 1 (20% of failures)

STORY PERFORMANCE ────────────────────────────────────────────────── STORY-001: 12 invocations 100% success avg 38s STORY-002: 8 invocations 100% success avg 35s STORY-003: 14 invocations 79% success avg 61s agent-stats.jsonl │ /analytics reads │ ┌─────────────────┼─────────────────┐ v v v memory agent orchestrate metrics

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: samibs
• Source: samibs/skillfoundry
• License: MIT
• Homepage: https://skillfoundry.work

Install and usage instructions live in the source repository linked above.