Context Engineer

Context Engineer

Predict, prevent, and mitigate context window failures before they happen.

The Victory Condition

Before reading anything else, know the test:

> If you can /clear the entire conversation, read your state file, and continue the workflow without any degradation — your architecture is correct.

Every design decision in this skill serves that single criterion. If your workflow passes it, proceed with confidence. If it doesn't, you have hidden state in conversation memory that the next compaction will silently destroy.

Core Problem

Context windows are finite (200K tokens for Opus 4.6). Complex workflows — iterative audits, multi-agent pipelines, large refactors — can exceed this limit. When they do, two bad things happen:

1. Hard failure: Context cannot be compacted, session crashes, work is lost.
2. Soft failure: Context IS compacted, but the summary silently drops information the workflow needs later. This is worse than a crash because you don't know what you lost.

Compaction is lossy compression. A 150K-token conversation summarized to 10K tokens has lost 93% of its information. The question is never "will information be lost?" — it's "will the RIGHT information survive?"

When to Invoke This Skill

Use /context-engineer BEFORE starting any workflow that:

• Will spawn 5+ subagents
• Involves iterative loops (audit → fix → re-audit)
• Will read more than 10 files
• Produces intermediate artifacts that later steps depend on
• Runs for more than 30 minutes
• Has been attempted before and crashed or produced degraded output

Information Taxonomy

Not all context tokens are equal. Every piece of information in a context window falls into one of five categories, each with different fidelity requirements.

The categories form a dependency chain:

RELATIONAL depends on → REASONING depends on → FACTUAL depends on → IMPERATIVE

Losing a foundation layer cascades upward and destroys everything that depends on it. If you lose FACTUAL information (scores, measurements), all REASONING built on those facts becomes ungrounded, and all RELATIONAL connections between those reasoning chains become meaningless. Preserve from the bottom up.

Category 1: IMPERATIVE (instructions, goals, constraints)

• Examples: "Reach 97/100 on 3 consecutive runs", "Never push without audit", style guides
• Fidelity requirement: LOSSLESS — must survive every compaction
• Preservation strategy: Belongs in CLAUDE.md, skills, or rules files. Never rely on conversation memory for imperatives.
• Risk if lost: Workflow drifts from goal, constraints violated silently

Category 2: FACTUAL (discovered facts, measurements, data)

• Examples: "Run 2 scored 80/100", "File X has 235 lines", "3 tests failed"
• Fidelity requirement: LOSSLESS for key facts, lossy acceptable for supporting details
• Preservation strategy: Write to structured state files (JSON). Facts in conversation memory WILL be corrupted or lost during compaction.
• Risk if lost: Decisions made on incorrect premises, regression, repeated work

Category 3: REASONING (chains of thought, decision rationale, causal analysis)

• Examples: "We chose approach X because Y would break Z", "The score dropped because of the FActScore lineage fix"
• Fidelity requirement: HIGH — expensive to reconstruct, often impossible to reconstruct identically
• Preservation strategy: Write decision logs to disk. Include the WHY, not just the WHAT.
• Risk if lost: Same mistakes repeated, contradictory decisions across runs, inability to explain choices

Category 4: EPHEMERAL (raw tool output, intermediate computation, verbose logs)

• Examples: Full git diff output, complete test suite results, raw web fetch content
• Fidelity requirement: NONE after extraction — safe to discard once key facts are extracted
• Preservation strategy: Extract facts immediately, then let compaction discard the raw data.
• Risk if lost: None, if facts were extracted first. Total, if facts were NOT extracted.

Category 5: RELATIONAL (connections between facts, dependency graphs, cross-reference maps)

• Examples: "Finding F015 caused Fix X which caused regression in Run 3", "Agent B6 contradicted Agent C3 on claim Y"
• Fidelity requirement: CRITICAL — this is the category most damaged by compaction
• Preservation strategy: Explicit relationship files on disk. Summaries preserve individual facts but destroy the relationships between them.
• Risk if lost: The most dangerous failure mode. You retain isolated facts but lose the web of connections that gives them meaning. Decisions become incoherent across runs.

Phase 1: Workflow Analysis

When invoked, first understand the planned workflow by asking:

1. What is the workflow? (iterative audit, multi-file refactor, research synthesis, pipeline, etc.)
2. How many steps/iterations? (fixed or convergence-based?)
3. What does each step produce? (reports, code changes, data, analysis?)
4. What does each step consume from previous steps?
5. What is the success criterion? (score target, all tests pass, user approval?)

Then estimate the token budget:

Token Budget Estimation

For each step in the workflow, estimate:

Step N:
  Inputs read:     [list files/data read, estimate tokens]
  Processing:      [reasoning, tool calls, discussion — estimate tokens]
  Outputs written: [files written, reports generated — estimate tokens]
  Carry-forward:   [what MUST survive to Step N+1 — estimate tokens]
  Discardable:     [what can be safely forgotten — estimate tokens]

  Step total:      [sum of above]
  Cumulative:      [running total across all steps]
  Headroom:        [200K - cumulative - system overhead (~20K)]

System overhead (always present, cannot be reduced):

• System prompt + CLAUDE.md: ~8-12K tokens
• MCP tool definitions: ~5-15K tokens (varies by server count)
• Skill definition (if loaded): ~2-6K tokens
• Extended thinking budget: ~2-32K+ tokens per turn (adaptive; can be higher with interleaved thinking during tool use)

Available working context: ~145-170K tokens depending on configuration.

Accumulation Patterns

Identify which pattern the workflow follows:

Linear accumulation (research, single-pass analysis):

Tokens: ────────────────────────────▶ overflow
Each step adds ~constant tokens, no discarding
Risk: Predictable. Overflow at step = headroom / tokens_per_step

Sawtooth (iterative with compaction):

Tokens: /\/\/\/\/\/\─── overflow if compaction insufficient
Each cycle: accumulate → compact → accumulate → compact
Risk: Each compaction is lossy. Fidelity degrades with each cycle.
After N cycles, you're working from an N-times-compressed summary.

Exponential (multi-agent with cross-referencing):

Tokens: ──────────╱ overflow (sudden)
Agents produce findings, findings reference each other,
cross-references multiply token count non-linearly
Risk: Hardest to predict. Often feels fine until sudden blowup.

Plateau (disk-as-memory, well-designed):

Tokens: ───────────────────────────── stable
Each step externalizes to disk, compacts, starts clean
Risk: Lowest. But requires upfront architecture work.

Phase 2: Chokepoint Prediction

A chokepoint is any step where:

• Cumulative tokens will exceed 80% of context window (150K+)
• A compaction would destroy information needed by a later step
• Multiple large artifacts must be in context simultaneously
• Cross-referencing between steps requires holding both in memory

Common Chokepoints

The Accumulator: Reading a large file, making changes, reading it again to verify, making more changes. Each read adds the full file to context. A 5K-token file read 6 times = 30K tokens just for one file. → Mitigation: Read once, edit in place, trust the edit tool's output. Don't re-read to verify unless necessary.

The Collector: Gathering findings from multiple agents/sources and synthesizing. Each finding set is 2-5K tokens; 16 agents = 32-80K tokens of findings before synthesis even begins. → Mitigation: Have agents write findings to disk. Read findings file-by-file during synthesis, extract what's needed, compact between files.

The Comparator: Holding two versions of something in context to compare (before/after, v1/v2, run N vs run N+1). Both versions must coexist in context for comparison to work. → Mitigation: Write a structured diff to disk. Don't hold both full versions — hold one version and the delta.

The Auditor Loop: Iterating until convergence. Each iteration accumulates: read target → audit → discuss findings → apply fixes → log results → repeat. After 4 iterations of a 50K-token cycle, you've consumed 200K. → Mitigation: Disk-as-memory architecture (see Phase 4). Compact between iterations.

The Growing Log: Appending to a cumulative log (like veracity-log.json). Each read of the log gets more expensive. A 112KB log = ~28-34K tokens every time it's read (varies by JSON density). → Mitigation: Don't read the full log. Read only the last entry, or use a summary index file.

Phase 3: Fidelity Risk Analysis

For each planned compaction point, analyze what will be lost:

Fidelity Loss Assessment

Compaction Point: [after Run N / after Step X / etc.]

Context size before compaction: ~[X]K tokens
Target size after compaction:   ~[Y]K tokens
Compression ratio:              [X/Y]:1
Information loss:                ~[1 - Y/X]%

IMPERATIVE information in context:
  - [list] → Will these survive? [yes: in CLAUDE.md / no: at risk]

FACTUAL information in context:
  - [list key facts] → Externalized to disk? [file path / NOT YET — RISK]

REASONING chains in context:
  - [list key decisions with rationale]
  - Which are reconstructible? [can re-derive from code/data]
  - Which are NOT reconstructible? [subjective choices, user preferences, debate resolutions]
  → Not-reconstructible reasoning MUST be written to disk before compaction

RELATIONAL information in context:
  - [list cross-references, dependency chains, causal links]
  → These WILL be destroyed by compaction. Summaries flatten relationships.
  → Write explicit relationship maps to disk.

EPHEMERAL information in context:
  - [list raw tool output, verbose logs]
  → Safe to lose. Verify key facts were extracted first.

Fidelity Under Compaction

Each compaction is a summary of a summary. Assume total loss of anything not externalized to disk.

There are no reliable percentages for how much survives — it depends on the summarizer's priorities, the content mix, and what the conversation looked like at compaction time. What IS known:

• First compaction: Key facts and recent context usually survive. Nuance, relationships, and early-conversation reasoning usually do not.
• Second compaction: You are now working from a compressed summary of a compressed summary. The dependency chain (RELATIONAL → REASONING → FACTUAL → IMPERATIVE) begins to break. Individual facts may survive but the connections between them are gone.
• Third+ compaction: Effectively a new session with partial amnesia. Decisions may be based on premises that were silently altered or dropped during prior compactions.

No empirical measurement of compaction fidelity exists. These are heuristic observations, not data. Treat any compaction you did not plan for as a total loss of RELATIONAL and REASONING information.

Rule of thumb: If your workflow requires more than 2 compaction cycles, you need a disk-as-memory architecture. Don't rely on compaction to manage long workflows.

Phase 4: Disk-as-Memory Architecture

The solution to both hard failures (overflow) and soft failures (fidelity loss) is the same: externalize state to structured files on disk, so the context window only ever holds one step's worth of work.

State File Design

Every complex workflow should have a state file. Design it to contain everything needed to resume the workflow from any point, without relying on conversation history.

{
  "_schema": "context-engineer/workflow-state/v1",
  "_description": "Complete workflow state — context window can be cleared and resumed from this file alone",

  "workflow": {
    "name": "veracity-convergence-loop",
    "goal": "Reach 97/100 on 3 consecutive audits",
    "target_file": "/path/to/SKILL.md",
    "started": "2026-03-01T10:00:00Z"
  },

  "progress": {
    "current_step": 4,
    "total_steps": null,
    "consecutive_target_hits": 1,
    "converged": false
  },

  "history": [
    {
      "step": 1,
      "score": 62,
      "delta": null,
      "findings": {"critical": 6, "high": 10, "medium": 10, "low": 6},
      "fixes_applied": 16,
      "report_path": "/path/to/run1/consolidated_report.md",
      "reasoning_log": "/path/to/run1/decisions.md",
      "timestamp": "2026-03-01T10:15:00Z"
    }
  ],

  "active_findings": [
    {
      "id": "F015",
      "severity": "MEDIUM",
      "status": "deferred",
      "reason": "Display-only issue, not functional",
      "introduced_in_step": 1,
      "last_checked_step": 3
    }
  ],

  "relationships": [
    {
      "type": "caused_regression",
      "source": "Fix applied in step 2 (FActScore lineage)",
      "target": "New finding in step 3 (overclaim about 'pioneered')",
      "note": "Fixing one claim exposed an adjacent overclaim"
    }
  ],

  "decisions": [
    {
      "step": 2,
      "decision": "Deferred code fence nesting fix",
      "rationale": "Markdown display issue only; Claude interpreter handles it correctly",
      "reversible": true
    }
  ],

  "compact_instructions": "Preserve: current step number, target score (97), consecutive hits count. Read state from /path/to/state.json for full history."
}

Checkpoint Protocol

Before each compaction point:

1. Extract all Category 2 (FACTUAL) information from context into the state file
2. Log all Category 3 (REASONING) chains into a decisions file
3. Map all Category 5 (RELATIONAL) connections into the relationships array
4. Verify all Category 1 (IMPERATIVE) information exists in CLAUDE.md or skill files
5. Release all Category 4 (EPHEMERAL) information — it's safe to lose
6. Validate — Read the state file back and confirm that every RELATIONAL and REASONING item currently in context also appears, explicitly, in the written record. Do not proceed to compaction until this check passes. This step exists because steps 1-3 are themselves lossy: the decision of what to write is a human judgment that can miss things.

Then compact with explicit instructions:

/compact Preserve: workflow state is in [path/to/state.json]. Current step: N. Goal: [goal].
Read state file at start of next step. Do not rely on conversation history for any facts.

Recovery Protocol

After compaction (or in a new session), recovery is:

1. Read the state file
2. Read the current target file
3. Read the last consolidated report (if continuing mid-cycle)
4. You now have everything needed to continue. Conversation history is irrelevant.

Test: Apply the Victory Condition — /clear, read state file, continue. If it works, you're good.

Phase 5: Generating the Context Engineering Plan

After analysis, produce a concrete plan in this format:

# Context Engineering Plan: [Workflow Name]

## Budget
- Available context: ~[X]K tokens
- Estimated per-step cost: ~[Y]K tokens
- Maximum steps before overflow (no mitigation): [N]
- Maximum steps with disk-as-memory: unlimited

## Predicted Chokepoints
1. [Step N]: [description of why this overflows]
   Mitigation: [specific action]

2. [Step M]: [description of fidelity risk]
   Mitigation: [specific action]

## State File Location
[path to state.json]

## Checkpoint Schedule
- After Step 1: Write [X] to disk, compact
- After Step N: Write [Y] to disk, compact
- [pattern]

## Fidelity-Critical Informat

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [joonchungpersonal-dev](https://github.com/joonchungpersonal-dev)
- **Source:** [joonchungpersonal-dev/claude-skills](https://github.com/joonchungpersonal-dev/claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.