Harness Engineering

⚙️ Harness Engineering: The Emerging Discipline of Making AI Agents Reliable

A comprehensive guide to the practice of shaping the environment around AI agents so they can work dependably — based on references from the Awesome Harness Engineering collection.

---

Table of Contents

1. [🏗️ What Is Harness Engineering?](#1-️-what-is-harness-engineering)
2. [⚡ Why It Matters Now](#2--why-it-matters-now)
3. [🧮 The Core Equation: Agent = Model + Harness](#3--the-core-equation-agent--model--harness)
4. [🧠 Foundations & Key Mental Models](#4--foundations--key-mental-models)
5. [💭 Context Engineering: The Working Memory Budget](#5--context-engineering-the-working-memory-budget)
6. [🛡️ Constraints, Guardrails & Safe Autonomy](#6-️-constraints-guardrails--safe-autonomy)
7. [📋 Specs, Agent Files & Workflow Design](#7--specs-agent-files--workflow-design)
8. [📊 Evals & Observability](#8--evals--observability)
9. [🚀 Runtimes, Harnesses & Reference Implementations](#9--runtimes-harnesses--reference-implementations)
10. [🎯 Benchmarks: Measuring Harness Quality](#10--benchmarks-measuring-harness-quality)
11. [🛠️ Practical Playbook: Engineering Your Own Harness](#11-️-practical-playbook-engineering-your-own-harness)
12. [🔮 The Future of Harness Engineering](#12--the-future-of-harness-engineering)
13. [✅ Conclusion](#13--conclusion)
14. [📚 References & Further Reading](#14--references--further-reading)

---

1. 🏗️ What Is Harness Engineering?

Harness engineering is the practice of designing, building, and iterating on the environment, tooling, constraints, and feedback loops that surround an AI agent — everything that isn't the model itself. The term gained widespread traction in early 2026, popularized by field reports from OpenAI, Anthropic, LangChain, Thoughtworks, and HumanLayer, all converging on the same insight: the reliability of an AI agent depends less on the model and more on the system wrapped around it.

As LangChain's Vivek Trivedy crystallized it:

> Agent = Model + Harness. If you're not the model, you're the harness.

A harness includes:

• System prompts — the instructions that shape the agent's persona and constraints
• Tools, skills, and MCP servers — capabilities the agent can invoke
• Bundled infrastructure — filesystem, sandboxes, browsers, observability stacks
• Orchestration logic — sub-agent spawning, handoffs, model routing
• Hooks and middleware — deterministic control flow for compaction, continuation, lint checks, and verification
• Memory and state management — progress files, git history, structured knowledge bases

A raw model is not an agent. It becomes one only when a harness gives it state, tool execution, feedback loops, and enforceable constraints. Harness engineering is the discipline of making all of that work well.

---

2. ⚡ Why It Matters Now

🔍 The "Skill Issue" Realization

As HumanLayer argued, teams that blame weak agent results on model limitations are usually wrong. After hundreds of agent sessions across dozens of projects, the pattern is consistent:

> It's not a model problem. It's a configuration problem.

Every time a team instinctively says "GPT-6 will fix it" or "we just need better instruction-following," the real fix is almost always in the harness — better context management, smarter tool selection, proper verification loops, or cleaner handoff artifacts.

🤖 The OpenAI Proof Point

OpenAI's flagship field report provided dramatic evidence. A three-person engineering team built and shipped an internal product with zero manually-written code — roughly a million lines across application logic, tests, CI, documentation, and tooling — all generated by Codex agents. The team averaged 3.5 merged PRs per engineer per day, and Codex runs regularly worked autonomously for six hours or more.

The key insight was that early progress was slower than expected — not because Codex was incapable, but because the environment was underspecified. The primary job of human engineers became enabling agents to do useful work: building the harness.

> "Because the only way to make progress was to get Codex to do the work, human engineers always stepped into the task and asked: 'what capability is missing, and how do we make it both legible and enforceable for the agent?'"

📈 Harness Changes Move Benchmarks

LangChain demonstrated that harness changes alone can significantly improve benchmark performance — moving their coding agent from Top 30 to Top 5 on Terminal-Bench 2.0 by only changing the harness, not the model. Anthropic showed that infrastructure configuration can move coding benchmark scores by more than many leaderboard gaps. The implication is profound: benchmarks often measure harness quality as much as — or more than — model quality.

---

3. 🧮 The Core Equation: Agent = Model + Harness

LangChain's Anatomy of an Agent Harness provides the clearest decomposition. Working backwards from what models cannot do natively reveals why each harness component exists:

| What We Want | What Models Can't Do Natively | Harness Solution | |---|---|---| | Persistent memory | Maintain durable state across interactions | Filesystem, git, progress files, AGENTS.md | | Autonomous problem-solving | Execute arbitrary code | Bash tool, code execution sandboxes | | Real-time knowledge | Access information beyond training cutoff | Web search, MCP tools, Context7 | | Safe operation | Understand risk boundaries | Sandboxes, allow-lists, network isolation | | Long-horizon coherence | Work across multiple context windows | Compaction, Ralph Loops, planning files | | Self-verification | Know if their work is correct | Test runners, browser automation, linters |

The filesystem emerges as the most foundational harness primitive because it unlocks everything else: agents get a workspace, work can be incrementally persisted, and multiple agents can coordinate through shared files. Git adds versioning so agents can track work, rollback errors, and branch experiments.

---

4. 🧠 Foundations & Key Mental Models

4.1 🔄 Feedforward and Feedback (Thoughtworks)

Birgitta Böckeler's framework at Thoughtworks provides the most rigorous mental model for harness engineering. She frames it through two control mechanisms:

• Guides (feedforward controls) — anticipate the agent's behavior and steer it before it acts. They increase the probability of good results on the first attempt. Examples: AGENTS.md files, architecture documentation, skills, coding conventions, reference applications.

• Sensors (feedback controls) — observe after the agent acts and help it self-correct. Most powerful when they produce signals optimized for LLM consumption. Examples: linters with custom error messages, test suites, code review agents, browser screenshots.

Without guides, the agent keeps repeating mistakes. Without sensors, the agent encodes rules but never finds out whether they worked. A good harness requires both.

4.2 💻 Computational vs. Inferential

Each control can be either:

• Computational — deterministic and fast, run by the CPU. Tests, linters, type checkers, structural analysis. Milliseconds to seconds; results are reliable.
• Inferential — semantic analysis, AI code review, "LLM as judge." Slower, more expensive, non-deterministic — but capable of richer judgment.

| Control | Direction | Type | Example | |---|---|---|---| | Coding conventions | Feedforward | Inferential | AGENTS.md, Skills | | Structural tests | Feedback | Computational | ArchUnit tests checking module boundaries | | Code review agent | Feedback | Inferential | A review skill using a strong model | | Bootstrap scripts | Feedforward | Both | Skill with instructions and a bootstrap script | | Code mods | Feedforward | Computational | OpenRewrite recipes |

4.3 ⚙️ The Cybernetic Governor

The harness acts as a cybernetic governor — combining feedforward and feedback to regulate the codebase toward its desired state. Böckeler identifies three regulation dimensions:

1. Maintainability harness — internal code quality (linters, complexity checks, coverage). The most mature category with extensive pre-existing tooling.
2. Architecture fitness harness — system characteristics (performance, observability, security). Essentially architectural fitness functions.
3. Behaviour harness — functional correctness. The hardest category: how do we verify that the application does what we need? This remains the elephant in the room.

4.4 🏛️ The Three Pillars (Thoughtworks)

Thoughtworks frames harness work into three pillars:

1. Context engineering — managing what the agent knows and when
2. Architectural constraints — enforcing invariants mechanically
3. Garbage collection — fighting entropy and drift continuously

4.5 🚗 Control–Agency–Runtime (CAR) Decomposition

An academic position paper proposes treating the harness layer as a first-class research object with three dimensions:

• Control — constraints, guardrails, permissions
• Agency — planning, decision-making, self-evaluation
• Runtime — execution environment, tools, infrastructure

---

5. 💭 Context Engineering: The Working Memory Budget

Context engineering is the practice of managing the agent's context window as a working memory budget rather than a dumping ground. It is arguably the most critical aspect of harness engineering.

5.1 🚫 The One Big File Anti-Pattern

OpenAI learned the hard way that a monolithic AGENTS.md doesn't scale:

> - Context is a scarce resource. A giant instruction file crowds out the task, the code, and the relevant docs. > - Too much guidance becomes non-guidance. When everything is "important," nothing is. > - It rots instantly. A monolithic manual turns into a graveyard of stale rules. > - It's hard to verify. A single blob doesn't lend itself to mechanical checks.

Their solution: treat AGENTS.md as a table of contents (~100 lines) that points to deeper sources of truth in a structured docs/ directory. This enables progressive disclosure — agents start with a small, stable entry point and are taught where to look next.

5.2 📖 Progressive Disclosure

Progressive disclosure is the principle that agents should only receive specific instructions, knowledge, or tools when they actually need them. Loading everything upfront pushes the agent into what HumanLayer calls "the dumb zone" — where context window fill degrades performance even on simple tasks.

Chroma's research on context rot provides empirical backing: models perform measurably worse at longer context lengths, and degradation is steeper when there's low semantic similarity between the query and the relevant information in context.

Skills solve this: they're activated on demand, bringing in focused knowledge only when needed.

5.3 🔧 Context-Efficient Backpressure

HumanLayer's backpressure philosophy is essential: verification mechanisms must be context-efficient. Running a full test suite after every change floods the context window with thousands of lines of passing tests. The agent loses track of its actual task.

The rule: success is silent, only failures produce output. Swallow the output of passing checks and only surface errors.

5.4 🧱 Sub-Agents as Context Firewalls

Sub-agents provide context isolation — each gets a fresh, small, high-relevance context window for its task, and only the condensed result flows back to the parent. This is far more powerful than simply making context windows bigger:

> "A bigger context window doesn't make the model better at finding the needle — it just makes the haystack bigger."

Effective sub-agent tasks: codebase exploration, grep/search operations, tracing information flow, research tasks — anything with a straightforward question and simple answer that requires many intermediate tool calls.

5.5 📝 Lessons from Manus

Manus' playbook contributed specific techniques: KV-cache locality optimization, tool masking, filesystem memory, and keeping useful failures in-context while discarding noise.

5.6 🗜️ OpenHands Context Condensation

OpenHands' approach to bounded conversation memory preserves goals, progress, critical files, and failing tests while condensing everything else — keeping long-running coding sessions efficient without losing essential state.

---

6. 🛡️ Constraints, Guardrails & Safe Autonomy

6.1 ⚖️ Enforcing Invariants, Not Micromanaging

OpenAI's approach is instructive: enforce boundaries centrally, allow autonomy locally. They require Codex to parse data shapes at the boundary (parse, don't validate), but don't prescribe how. Each business domain follows a fixed layered architecture (Types → Config → Repo → Service → Runtime → UI) with strictly validated dependency directions enforced by custom linters and structural tests — all Codex-generated.

> "This is the kind of architecture you usually postpone until you have hundreds of engineers. With coding agents, it's an early prerequisite: the constraints are what allows speed without decay."

Custom linter error messages are written to inject remediation instructions into agent context — a positive form of prompt injection that guides self-correction.

6.2 📦 Sandboxing and Controlled Execution

Anthropic's work on sandboxing focuses on reducing approval friction without losing control. Rather than prompting humans for every action, better sandboxing and policy design allow agents to work more autonomously while staying within safe boundaries.

MCP-based code execution gives agents controlled execution power through explicit, inspectable tool boundaries — making it clear what the agent can and cannot do.

6.3 🔒 Tool Design for Safety

Anthropic's guidance on writing tools for agents emphasizes that tool interfaces should be easy for models to call correctly and safely. Poorly designed tools lead to misuse; well-designed tools guide the agent toward correct behavior.

6.4 💉 Prompt Injection Defense

OpenHands' practical guide covers confirmation mode, analyzers, sandboxing, and hard policies for reducing prompt-injection risk. This is especially important given that MCP server tool descriptions are injected into system prompts — never connect to one you don't trust.

6.5 ✔️ Quality Checks in the Loop

Rather than relying on after-the-fact manual review, Thoughtworks advocates moving quality

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: truongpx396
• Source: truongpx396/harness-engineering
• License: MIT

Install and usage instructions live in the source repository linked above.