Harness Engineering Audit

Harness Engineering Audit

Use this skill to turn a repository into a production-ready, agent-legible engineering environment optimized for Codex, OMX, and long-running AI-assisted development.

This skill is audit-first, AGENTS-first, and low-risk auto-approved. It may generate reports and planning artifacts before edits, then low-risk recommendations from the audit are approved for follow-up execution without another user approval. Medium/high-risk changes still require explicit approval.

Core operating model

1. Inspect the repo and collect evidence.
2. Score the repo against the harness-engineering rubric.
3. Produce a strict report with low/medium/high-risk recommendations.
4. Generate AGENTS.md priority and OMX handoff artifacts.
5. Auto-approve low-risk fixes, with root AGENTS.md as P0 when it is missing, oversized, stale, or lacks docs/validation pointers.
6. Classify lifecycle as greenfield-bootstrap, brownfield-cleanup, or mature-audit.
7. Use OMX planning/execution or explicit setup modes for auto-approved low-risk fixes; defer medium/high-risk changes until explicit approval.

The default report location is:

.codex/reports/harness-engineering-audit/

Each new run replaces the previous generated report directory at that path to avoid stale audit confusion. The included scripts only overwrite the known report directory or an explicitly provided output directory that is marked as a harness-engineering audit report directory.

When to use

Use this skill when a user asks to audit or improve:

• harness engineering
• agentic development readiness
• Codex readiness
• OMX workflow quality
• AGENTS.md scope and quality
• .codex/config.toml
• MCP configuration
• skills
• hooks and rules
• subagent/team workflow
• docs authority
• validation commands
• command registries
• generated artifact policy
• stack/tooling upgrade recommendations
• scaffolding/legacy entropy
• repo legibility for AI agents
• production-readiness for agent-first development
• docs-to-production app readiness
• visual product spec packs from generated PNG/SVG targets, UI packs, stories, token packs, and component contracts
• frontend parity and product usability harnesses
• runtime evidence / local product done gates
• prompt-injection and instruction-trust readiness

Authoritative principles

Treat OpenAI/Codex guidance as normative:

• AGENTS.md should be a concise map, not an encyclopedia.
• Repo-local docs should be the system of record.
• Validation should be executable and evidence-backed.
• Skills should encode repeatable workflows.
• Domain vocabulary should be discoverable and progressively disclosed, so agents use project language consistently without bloating hot-path instructions.
• Docs should be gardened as living knowledge: raw sources, synthesized docs, indexes, logs, and health checks need explicit boundaries and maintenance workflows.
• MCP should solve real external/context access problems.
• Hooks/rules should be deterministic, scoped, and safe.
• Subagents/team workflows should have clear delegation, path scope, and review boundaries.
• Generated artifacts and scaffolding require lifecycle policies.
• Generated visual targets are design-intent evidence, not executable specifications; every priority target needs an adjacent screen contract.
• Implied behavior must be classified as explicit, inferred, or unresolved; unresolved and sensitive inferred behavior stays approval-gated.
• Production app readiness requires traceability from user stories to visual targets, screen contracts, routes, APIs, components, tests, screenshots, accessibility evidence, and production deltas.

Use Claude/Cursor/Karpathy/third-party resources only as comparison material, not as authority over Codex behavior. The vocabulary/domain-language audit surface is informed by Matt Pocock's Dictionary of AI Coding as comparison material: shared terms, handoff artifacts, progressive disclosure, and the distinction between deterministic checks and review are useful harness concepts, but this skill should not copy that repo's prose. The doc-gardening audit surface is informed by Andrej Karpathy's LLM Wiki gist as comparison material: persistent markdown knowledge bases, source/wiki/schema layering, ingest/query/lint workflows, indexes/logs, and wiki health checks are useful harness concepts, but this skill should not copy that gist's prose.

Required evidence to inspect

Always inventory these surfaces when present:

• root AGENTS.md
• nested AGENTS.md and AGENTS.override.md
• .codex/config.toml
• .codex/hooks.json
• .codex/rules/**
• .codex/prompts/**
• .agents/skills/**
• .omx/**
• .github/workflows/**
• root README and docs indexes
• docs/**
• canonical product docs such as docs/product/**, docs/stories/**, docs/screens/**, docs/design/**, docs/architecture/**, docs/contracts/**, and docs/quality/**
• generated/imported UI targets such as docs/screens/**/target*.png, docs/screens/**/target*.svg, docs/ui/targets/**, and equivalent Figma/design-export packs
• screen and behavior contracts such as screen.contract.yaml, interactions.yaml, motion.yaml, data-requirements.yaml, route maps, state machines, acceptance matrices, and traceability matrices
• design token and component packs such as tokens.json, docs/design/tokens/**, docs/design/components/**, Style Dictionary/Tokens Studio config, component recipes, and Storybook stories
• visual QA surfaces such as .storybook/**, *.stories.*, playwright.config.*, screenshot baselines, visual-regression policy, accessibility policy, and viewport/theme/reduced-motion matrices
• glossary / terminology / domain-language docs such as docs/GLOSSARY.md, docs/TERMINOLOGY.md, internal/CONTEXT.md, internal/domain.md, and dictionary/**
• ADR / decision-record surfaces such as docs/adr/**, docs/ADR/**, and internal/adr/**
• doc gardening / wiki / knowledge-base surfaces such as raw sources, generated/synthesized docs, index.md, log.md, ingest/query/lint workflows, and stale/contradiction/orphan/broken-link checks
• validation scripts and package scripts
• test/build/lint/typecheck/smoke commands
• generated artifact directories
• scaffold/legacy/stage/demo/preview markers
• cross-agent files: CLAUDE.md, .claude/**, .cursor/**, .windsurf/**, .cline/**, .gemini/**, .github/copilot-instructions.md

Docs-to-production harness audit

The audit now includes a stack-neutral Docs-to-Production App Readiness family. It detects repository capabilities and evaluates whether the repo exposes the harness layers needed for Codex/OMX to build a locally usable product from docs and design targets:

• Visual Product Spec Pack
• Visual Target Contracts
• User Story & Acceptance Pack
• Frontend Parity System
• Design Token & Component Pack
• Component Workbench / Storybook Harness
• Visual Regression Gate
• Product Usability Contract
• Runtime Evidence Harness
• Coverage & Traceability Harness
• Local Provider Simulator Harness
• Brownfield State Inventory Harness
• State Machine Harness
• Negative Path Harness
• Role Walkthrough Harness
• Runtime Observability Harness
• Local Product Done Gate
• Repo-scoped Codex skills
• Instruction Trust / Prompt Injection Audit

For UI-led work, the skill applies these tenets:

• a generated screenshot/mockup is an input artifact, not the contract;
• screen.contract.yaml or an equivalent artifact must state route, viewport, theme, visual state, interactions, data/API assumptions, motion, accessibility, and approval-gated inferences;
• the repo should keep a canonical docs set for product intent, user stories, screens, design tokens/components, architecture, contracts, and quality gates;
• a traceability matrix should connect each priority story to visual targets, screen contracts, routes, components, APIs, fixtures, tests, and evidence;
• visual completion should be proven through a workbench/Storybook surface, E2E/runtime evidence, accessibility checks, and human-approved visual baselines rather than raw prompt assertions.

The audit writes pre-mortem and readiness artifacts under .codex/reports/harness-engineering-audit/, including premortem.md, failure-mode-register.yaml, harness-gap-matrix.yaml, docs-to-production-readiness.md, capability-adapter-report.yaml, instruction-trust-report.md, and reusable OMX prompt files under recommended-omx-prompts/.

product-harness-setup can create low-risk harness templates and repo-scoped skill templates for visual-target ingest, design-token/component contracts, frontend parity, product usability, runtime evidence, traceability, visual QA, and final review. It must not implement application features, enable live providers, mutate security-sensitive configuration, or use real data.

Run the audit script

Before running the audit in an interactive session, prompt the user for the audit level and proceed with their selection. Offer these choices:

| Choice | Side effects | Approval posture | | --- | --- | --- | | minimal (default) / audit | Report-only audit; no target source mutation. | Safe default. | | safe-setup | Create missing low-risk docs/templates only; never creates .codex/agents. | Explicit setup selection. | | force-ideal-harness | Stronger low-risk docs/template consolidation; no deletes or CI/config/hooks/security changes. | Explicit setup selection. | | symphony-repo-local | Repo-local Symphony contracts/templates and inert handoff text. | Explicit setup selection; live installs still approval-gated. | | symphony-live-handoff | Approval-gated handoff text in the report directory only; no target source or install/config mutation. | Explicit handoff selection. | | full-orchestration | Lane-pack orchestration contracts and harness custom-agent TOML; never runs agents or live install/config commands. | Explicit opt-in only; never the default. | | product-harness-setup | Docs-to-production harness templates, local-product-done docs, and repo-scoped Codex skill templates; no app implementation. | Explicit setup selection. |

The Python script also prompts for this level when run from an interactive TTY without --mode. Non-interactive runs default to minimal/audit so automation remains stable.

From the repo root after installing this skill, run the script from the installed location.

Project-scoped install:

python3 .agents/skills/harness-engineering-audit/scripts/run_audit.py .

User-scoped install:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py .

The script writes:

.codex/reports/harness-engineering-audit/
  inventory.json
  scorecard.json
  stack-inventory.json
  tool-inventory.json
  upgrade-recommendations.json
  upgrade-recommendations.md
  web-verification-queue.json
  source-trust-policy.md
  update-status.json
  report.md
  findings.md
  recommended-fixes.md
  agents-priority.md
  omx-handoff.md
  next-step.md
  next-step.json
  docs-to-production-readiness.md
  premortem.md
  premortem.json
  failure-mode-register.yaml
  harness-gap-matrix.yaml
  capability-adapter-report.yaml
  instruction-trust-report.md
  recommended-omx-prompts/
  prompts/
    deep-interview.md
    ralplan.md
    team.md
    ralph.md
    symphony-adoption.md
    tool-upgrade-ralplan.md

Recommendation defaults are intentionally conservative:

• recommend tools: true
• web verification requested: true
• human approval gate: required
• install/config mutation: false
• write reports: true

Local Python scripts cannot browse. They therefore write a web verification queue and keep web_verified: false until a browsing-capable human/agent records evidence. Generated install/config/validation/rollback commands are inert handoff text and must not be executed without explicit approval.

Primary artifacts now include a “What happened / What did not happen / What needs approval” section plus machine-readable mode_summary, approval_state, tool_recommendation_state, and current_step_explanation fields in next-step.json.

Use a custom output directory only when needed:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --out /tmp/harness-engineering-audit

Execution modes

The default prompt choice remains report-only. Use minimal or audit explicitly when you want to skip the prompt while preserving report-only behavior:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --mode minimal
python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --mode audit

Additional explicit modes create only low-risk harness artifacts with provenance markers and setup-rollback-manifest.json:

AUDIT_SCRIPT=~/.codex/skills/harness-engineering-audit/scripts/run_audit.py
# For a project-scoped install, use:
# AUDIT_SCRIPT=.agents/skills/harness-engineering-audit/scripts/run_audit.py

python3 "$AUDIT_SCRIPT" . --mode safe-setup
python3 "$AUDIT_SCRIPT" . --mode force-ideal-harness
python3 "$AUDIT_SCRIPT" . --mode symphony-repo-local
python3 "$AUDIT_SCRIPT" . --mode symphony-live-handoff
python3 "$AUDIT_SCRIPT" . --mode full-orchestration
python3 "$AUDIT_SCRIPT" . --mode product-harness-setup

--force keeps its existing output-overwrite meaning; it is not the force-ideal-harness mode.

Mode boundaries:

• audit: reports only, no target source mutation.
• safe-setup: creates missing low-risk docs/templates only.
• force-ideal-harness: explicit stronger low-risk consolidation; no deletes or CI/config/hooks/security changes.
• symphony-repo-local: repo-local Symphony contracts/templates and inert handoff text only.
• symphony-live-handoff: approval-gated handoff text only; no install/config mutation.
• full-orchestration: explicit opt-in for lane-pack orchestration contracts and project custom-agent TOML. It is never the default, never runs agents, and never executes live install/config commands.
• product-harness-setup: explicit opt-in for docs-to-production harness docs and repo-scoped skill templates, including visual product spec pack templates; it never implements app code or enables live providers.

Lane-pack boundaries:

• safe-setup creates lane-pack Markdown docs under docs/harness/** only and must not create .codex/agents.
• Stack-detected lane packs must report activation confidence, matched evidence, and recommendation policy. Weak signals remain advisory candidates instead of default missing-lane work.
• .codex/agents/*.toml custom-agent definitions are generated only by full-orchestration, use harness-* names, and are invoked only when the user explicitly asks Codex to use subagents.
• The lane-pack registry uses universal core lanes plus stack-detected lanes such as UI/UX, backend/API, data, security, performance, infra, AI/ML/CV, docs gardening, and QA.

Skill update behavior

Normal audit runs check this skill's update status by default and write update-status.json plus a Skill update status section in report.md. This check is report-only and never updates skill files. Disable it with:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --no-check-update

Self-update is explicit and approval-gated. To update this one user-scoped skill installation, run:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --self-update --update-scope user

The self-update path runs only this command and exits immediately after success:

gh skill install ryne2010/harness-engineering-audit skills/harness-engineering-audit --agent codex --scope user --force

Project-scoped installs should generally be updated intentionally through the repository and reviewed in a PR. Avoid gh skill update --all for this flow because system or manually installed skills may lack GitHub metadata and because this skill should upd

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ryne2010
• Source: ryne2010/harness-engineering-audit
• License: MIT

Install and usage instructions live in the source repository linked above.