Agent Harness Review

Agent Harness Engineering Design Review

To audit any codebase that wraps LLM agents, apply the AI Agent Harness Engineering framework. The harness is the runtime infrastructure — state machines, retry logic, context management, output validation, and AI-specific UX — that makes LLM systems reliable in production.

> Key finding (2026): Two teams using the same LLM see 60% vs 98% task completion rates based purely on harness quality. Token usage explains 80% of performance variance between agent implementations. Model selection matters far less than harness engineering.

> Definition: Agent = Model + Harness. The harness is everything except the model itself.

---

When to Invoke

• Before shipping an AI-powered feature
• After adding a new LLM provider or tool
• During post-incident review of AI failures
• When AI responses feel slow, inconsistent, or opaque to users
• Periodic fitness check for any agentic system

---

The 6 Dimensions

All 42 named patterns with scoring rubrics are in [references/patterns.md](references/patterns.md).

| Dimension | Patterns | What It Covers | |-----------|----------|----------------| | Resilience | R1–R7 | Retry, circuit breakers, fallbacks, timeouts, multi-provider | | State Management | S1–S7 | FSMs, slot filling, checkpointing, rainbow deployments | | Context Engineering | C1–C6 | Token management, memory layers, observation masking, note-taking | | Output Quality | Q1–Q5 | Schema validation, guardrails, LLM-as-judge, failure-to-test | | UX Patterns | U1–U8 | Streaming, TTFT, error UX, streaming/schema incompatibility | | Architecture | A1–A9 | Routing, multi-provider, config-driven models, feedforward/feedback |

---

How to Run This Review

Step 1: Scope

If an argument was provided (path or module), scope all phases to that subtree. Otherwise default to the full codebase.

Step 2: Explore the Codebase

For each dimension in [references/patterns.md](references/patterns.md), search for each pattern's grep targets. Note: present, absent, or partial.

Key grep targets by dimension:

• Resilience: retry, backoff, sleep, delay, circuit breaker, fallback, provider routing, timeout config
• State: state enum, FSM transition functions, slot/parameter tracking, checkpoint/restore
• Context: history truncation, token limit, summary, tool result handling, memory layers
• Output Quality: JSON schema, structured output, strict, validation, guardrail, fallback parse
• UX: streaming config, TTFT, error message strings, typing indicator, skeleton/placeholder
• Architecture: LLM client/router class, model selection config, provider switching, cost tracking

Step 3: Score Each Dimension

Use the 0-10 rubric from [references/patterns.md](references/patterns.md). Be conservative; award partial credit only when a pattern is meaningfully implemented.

Step 4: Check the Anti-Patterns Checklist

Flag any of these as Critical:

• [ ] Determinism expectation — Code assumes same prompt always returns same output (no retry, no validation, no fallback)
• [ ] Context stuffing — Full conversation history sent every turn without filtering (degrades quality after turn 9)
• [ ] Blocking UI on LLM — UI thread blocked waiting for LLM response without timeout or cancel
• [ ] Single-provider dependency — No failover path; one provider outage = full feature outage
• [ ] Silent gray failures — LLM returns plausible but incorrect output (narration instead of tool call) with no detection
• [ ] Generic error messages — All LLM failures show same "Something went wrong" regardless of error category
• [ ] Hidden system state — Users cannot tell if AI is waiting, processing, streaming, or failed
• [ ] Hardcoded model names — Provider/model selection in code not config; re-deployment required to switch models

Step 5: Classify Findings by UX Leverage

• HIGH: Directly visible to users (latency, error messages, dead ends, missing affordances)
• MEDIUM: Affects reliability in edge cases users will eventually hit
• LOW: Code quality, maintainability, future-proofing

Step 6: Write the Report

Use the template below.

---

Output Report Template

# Agent Harness Engineering Review — [Project Name]
Date: [date]
Reviewed scope: [path or "full codebase"]

## Dimension Scores
| Dimension           | Score | Key Gap |
|---------------------|-------|---------|
| Resilience          |  /10  |         |
| State Management    |  /10  |         |
| Context Engineering |  /10  |         |
| Output Quality      |  /10  |         |
| UX Patterns         |  /10  |         |
| Architecture        |  /10  |         |
| **Overall**         |  /10  |         |

## Anti-Patterns Found
- [ ] List any anti-patterns detected

## Strengths (Preserve These)
| Pattern | Where | Why It Matters |
|---------|-------|----------------|
| ...     | ...   | ...            |

## Findings by UX Leverage

### HIGH (Fix First)
1. [Pattern ID + name] — [description] — [file:line if applicable]

### MEDIUM
...

### LOW
...

## Recommended Next Steps (Sprint-sized)
1. [Actionable item, scoped to ≤1 week]

---

Overall Health Brackets

| Score | Meaning | |-------|---------| | ≥8.0 | Production-hardened harness | | 6.0–7.9 | Solid foundation, specific gaps to address | | 4.0–5.9 | Significant gaps, reliability risk in production | | model selection | | Token usage explains 80% of performance variance | Optimize token spend before switching models | | 90.2% improvement: multi-agent over single-agent, same model | Consider agent decomposition before model upgrades | | 39% LLM perf drop in multi-turn conversations around turn 9 | Apply context compression at turn boundary | | Multi-provider adoption: 23% → 40% in one year; 114 provider incidents/90 days | Single provider is an existential reliability risk | | Synchronous guardrail stacking adds 500–1,300ms | Use risk-based routing; run async validation for low-risk paths | | TTFT <500ms perceived as "instant" by users | TTFT Bridge is high-value UX improvement when streaming is blocked | | Observation masking cuts costs ~50% vs full tool outputs in history | Strip raw tool responses after extracting key data | | 100k+ token contexts can degrade reasoning quality | Apply C5 (Minimum Viable Context) before hitting window limits |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: microstone88
• Source: microstone88/claude-harness-skills
• License: MIT

Install and usage instructions live in the source repository linked above.