# Evals Analyze

> Analyze evaluation results and close the loop. Specification failures create local CDRs to fix agent rules; generalization failures go to evaluator backlog.

- **Type:** Skill
- **Install:** `agentstack add skill-tikalk-adlc-team-skills-evals-analyze`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [tikalk](https://agentstack.voostack.com/s/tikalk)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [tikalk](https://github.com/tikalk)
- **Source:** https://github.com/tikalk/adlc-team-skills/tree/main/skills/evals/evals-analyze
- **Website:** https://github.com/tikalk/agentic-sdlc-12-factors

## Install

```sh
agentstack add skill-tikalk-adlc-team-skills-evals-analyze
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# evals-analyze

## What this skill does

Provides **cross-functional team elevation** and **closed-loop feedback** following **EDD Principle VIII** (Close the Production Loop) by deep-analyzing trajectory failure traces and routing them to correct resolution pathways.

**Output**:
1. **Trajectory Analysis** - Full multi-turn trace analysis with tool calls and context preservation (EDD Principle V)
2. **Failure Routing**:
   - **Specification Failures** (agent logic missing/ambiguous) → Automatically triggers a local call to `levelup-specify` to propose new context rules in `.adlc/drafts/cdr/` to fix agent behavior.
   - **Generalization Failures** (grader flawed or lacks edge-case coverage) → Appends evaluator backlog items to the project backlog for ongoing monitoring.
3. **Cross-Functional PR** - Creates a team-ai-directives PR with insights and rule updates (EDD Principle X)

**Key EDD Principles Applied**:
- **Principle VIII**: Close Production Loop - Spec failures → fix directives; Gen failures → evaluator backlog
- **Principle V**: Trajectory Observability - Full multi-turn traces, not just outputs
- **Principle X**: Cross-Functional Observability - PMs, domain experts, and AI engineers collaborate

## When to use

- **After `/evals-validate`**: Analyze failures and resolve them
- **Closing a development loop**: Translate evaluation failure insights into rule or evaluator fixes
- **Reporting to stakeholders**: Generate readable summaries for PMs and domain experts

## When NOT to use

- **Evals not yet executed**: Run `/evals-validate` first to generate results in `evals/results/`
- **Trivial tasks**: Closed-loop analysis is overhead for simple features

## Process

### User Input
```text
$ARGUMENTS
```
- `--focus AREA` — Focus analysis on specific areas (e.g., security, quality, performance)
- `--dry-run` — Analyze results and print report, but skip PR creation and local skill triggers

### Execution Steps

#### Phase 1: Load Evaluation Results
- Reads results JSON from `evals/results/`.
- Extracts failure cases and full multi-turn conversation traces (including tool calls).

#### Phase 2: Failure Classification
Categorizes each failure trace:
- **Specification Failure**: The agent was correct relative to its context, but the rule/directive was missing, ambiguous, or incorrect.
- **Generalization Failure**: The rule was correct, but the agent made a mistake anyway (hallucinated, missed a constraint, or grader lacked edge-case coverage).

#### Phase 3: Action Routing (Close the Loop)
- **For Specification Failures**: Automatically triggers local skill `/levelup-specify` with the failure trace as input. This creates new rule/persona/example CDRs in `.adlc/drafts/cdr/` to fix the agent's behavior.
- **For Generalization Failures**: Appends an evaluator backlog item to `evals/results/evaluator_backlog.md` detailing the needed grader edge-case updates.

#### Phase 4: Cross-Functional Insights & PR
- Generates a stakeholder-specific report in `evals/results/team_insights.md` (tailored for PMs, domain experts, and AI engineers).
- If git remote and gh CLI are available, commits rule/eval changes in `team-ai-directives` and opens a draft PR (uses `levelup-publish` logic under the hood).

## Verification
- Trajectory failure traces analyzed and classified
- Specification failures successfully routed to `/levelup-specify` (proposes CDRs in `.adlc/drafts/cdr/`)
- Generalization failures written to `evals/results/evaluator_backlog.md`
- Stakeholder report `evals/results/team_insights.md` generated
- Draft PR created in team-ai-directives (if applicable)
- Final report summary presented with PR link and backlog details

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [tikalk](https://github.com/tikalk)
- **Source:** [tikalk/adlc-team-skills](https://github.com/tikalk/adlc-team-skills)
- **License:** MIT
- **Homepage:** https://github.com/tikalk/agentic-sdlc-12-factors

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-tikalk-adlc-team-skills-evals-analyze
- Seller: https://agentstack.voostack.com/s/tikalk
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.