Agentv Trace Analyst

AgentV Trace Analyst

Analyze evaluation traces headlessly using agentv inspect primitives and jq.

Primitives

# List result files (most recent first)
agentv inspect list [--limit N] [--format json|table]

# Show results with trace details
agentv inspect show  [--test-id ] [--tree] [--format json|table]

# Percentile statistics
agentv inspect stats  [--group-by target|suite|test-id] [--format json|table]

# A/B comparison between runs
agentv compare   [--threshold 0.1] [--format json|table]

Analysis Workflow

1. Discover results

agentv inspect list

Pick the result file to analyze. Most recent is first.

2. Get overview

agentv inspect stats 

Read the percentile table. Key signals:

• score p50 30s: Performance bottleneck
• cost p99 spike: Outlier cost tests to investigate
• tool_calls p90 >> p50: Some tests are much chattier

3. Investigate failures

agentv inspect show  --format json | jq '[.[] | select(.score  --test-id 

# Tree view (if output messages available)
agentv inspect show  --test-id  --tree

The tree view shows the agent's execution path — LLM calls interspersed with tool invocations. Look for:

• Excessive tool calls: Agent looping or exploring unnecessarily
• Missing tools: Expected tool not called
• Long durations: Specific tool calls that are slow

5. Compare runs

agentv compare  

Look for:

• Wins vs losses: Net improvement or regression?
• Mean delta: Overall direction of change
• Per-test deltas: Which tests regressed?

6. Group analysis

# By target provider
agentv inspect stats  --group-by target

# By suite
agentv inspect stats  --group-by suite

Compare providers side-by-side: which is cheaper, faster, more accurate?

Advanced Queries with jq

All commands support --format json for piping to jq:

# Top 3 most expensive tests
agentv inspect show  --format json \
  | jq 'sort_by(-.cost_usd) | .[0:3] | .[] | {test_id, cost: .cost_usd, score}'

# Tests where token usage exceeds 10k
agentv inspect show  --format json \
  | jq '[.[] | select(.token_usage.input + .token_usage.output > 10000) | {test_id, tokens: (.token_usage.input + .token_usage.output)}]'

# Score distribution by suite
agentv inspect show  --format json \
  | jq 'group_by(.suite) | .[] | {suite: .[0].suite, count: length, avg_score: ([.[].score] | add / length)}'

# Tool usage frequency across all tests
agentv inspect show  --format json \
  | jq '[.[].trace.tool_calls // {} | to_entries[]] | group_by(.key) | .[] | {tool: .[0].key, total_calls: ([.[].value] | add)}'

# Find regressions > 0.1 between two runs
agentv compare baseline.jsonl candidate.jsonl --format json \
  | jq '.matched[] | select(.delta  0 is good, but check individual test regressions — a few large losses can hide behind many small wins.

## Accessing reference files

To load a specific reference without pulling the entire skill into context:

```bash
agentv skills get agentv-trace-analyst --ref 

Or resolve the skill directory and read files directly:

cat $(agentv skills path agentv-trace-analyst)/references/.md

Use --full to retrieve every file in the skill at once.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: EntityProcess
• Source: EntityProcess/agentv
• License: MIT
• Homepage: https://agentv.dev

Install and usage instructions live in the source repository linked above.