Autoresearch

Autoresearch

Autonomous iterative improvement for any measurable system.

Built on Karpathy's autoresearch pattern, extended with:

• Critic agent with binary pass/fail gates and holdout validation — generator/verifier separation so the agent can't grade its own homework
• Coverage-driven exploration instead of pure hill-climbing — don't spend 40 experiments tuning one knob when another knob has never been touched
• Metacognitive self-modification — a meta-reviewer agent that can improve the researcher's own optimization strategy between runs
• Archive of stepping stones — maintain a diverse population of good configurations, not just the single best, so you can branch out when stuck in a local optimum

See the references/ folder for the academic and practitioner sources behind each of these extensions.

Triggers

• "autoresearch"
• "run autoresearch on..."
• "iteratively improve..."
• "optimize [X] autonomously"
• "run experiments on..."
• "self-improve [system]"
• "find better parameters for..."
• "tune [X]"
• User wants to autonomously improve any measurable system

The Pattern

Three agents work in a pipeline, coordinated by a small set of files on disk:

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  ┌─────────────┐    ┌─────────────┐    ┌──────────────────┐     │
│  │  RESEARCHER │───▶│   CRITIC    │───▶│  META-REVIEWER   │     │
│  │  (proposes) │    │ (validates) │    │ (every N exps)   │     │
│  └─────────────┘    └─────────────┘    └──────────────────┘     │
│        │                   │                    │              │
│        ▼                   ▼                    ▼              │
│  1. Analyze weakness  4. Holdout gate      7. Review full log  │
│  2. Form hypothesis   5. Overfitting check 8. Update strategy  │
│  3. Change 1 param    6. Binary pass/fail  9. Suggest new      │
│     + run eval           → KEEP/DISCARD       dimensions       │
│        │                   │                    │              │
│        └───────────────────┴────────────────────┘              │
│                            │                                   │
│                    ┌───────▼────────┐                          │
│                    │ COVERAGE MAP   │                          │
│                    │ What's been    │                          │
│                    │ explored?      │                          │
│                    │ What's missing?│                          │
│                    └────────────────┘                          │
│                            │                                   │
│                    ┌───────▼────────┐                          │
│                    │    ARCHIVE     │                          │
│                    │ Top N configs  │                          │
│                    │ Branch from    │                          │
│                    │ diverse parents│                          │
│                    └────────────────┘                          │
└─────────────────────────────────────────────────────────────────┘

File Roles

| File | Role | Who Edits | |------|------|-----------| | research.md | Agent instructions — what to optimize, how to think, domain knowledge | Human initially; meta-reviewer can append | | eval script | Fixed measurement — runs the metric calculation | Nobody (frozen) | | parameters file | Tunable knobs — the ONLY thing the researcher changes | Researcher agent | | experiments/ | Accountability trail — every experiment logged | All agents (append-only) | | archive.json | Top N parameter configs with scores | Researcher agent | | coverage.json | Which parameter dimensions have been explored | Auto-tracked |

The eval script is sacred. No agent ever modifies it. This prevents "improving" the score by weakening the test — the single most common failure mode in autonomous optimization systems.

Prerequisites

• A measurable objective — a number you want to go up or down
• An evaluation method — a script, API call, or command that produces that number
• Tunable parameters — things you can change that affect the metric
• Sufficient historical data or test cases to evaluate against (enough to split into train + holdout)

Workflow

Phase 1: Define the Objective

Ask the user (or infer from context):

1. What are we optimizing? (e.g., forecast accuracy, prompt quality, page load time, ad ROAS)
2. What's the metric? (e.g., MAE, F1, p95 latency, Lighthouse score, cost-per-conversion)
3. Direction? Lower is better, or higher is better?
4. How do we measure it? (e.g., backtest against actuals, run eval suite, run benchmark)
5. What can we change? (e.g., model parameters, prompt sections, config values, weights)
6. What data do we evaluate against? (e.g., last 30 days of actuals, test dataset, production logs)
7. Can we split the data? The eval should support a --holdout flag so the critic can validate on data the researcher never saw.

Phase 2: Scaffold the Files

Create 6 files in the project. Adapt file names and formats to the domain.

1. research.md — Agent Instructions (mutable)

The "program.md" equivalent from Karpathy's original. It tells the researcher agent:

• What system we're optimizing
• What the metric means
• Domain knowledge and constraints
• What parameters are available to tune
• Rules of engagement

v2 change: The meta-reviewer agent can append to research.md in a clearly marked ## Learned Insights section. This is the metacognitive self-modification idea — the agent improves not just the parameters, but its own optimization strategy.

Use the template: [templates/research-template.md](templates/research-template.md).

2. Eval Script — Fixed Measurement

A script that:

• Takes the current parameters as input (or reads them from the parameters file)
• Runs the evaluation against the test data
• Outputs a single primary metric (and optionally secondary metrics)
• Is deterministic — same parameters always produce the same score
• Supports a --holdout flag to evaluate on a held-out data split (the critic uses this)
• Outputs a per-segment breakdown so the critic can check for regressions, not just the aggregate metric

This can be a Python/Node/Shell script, CLI command, API call, database query, etc.

Mark it read-only after creation:

chmod 444 eval.py  # or eval.sh, eval.js, etc.

3. Parameters File — The Tunable Knobs

A structured file (YAML, JSON, Markdown — whatever fits the domain) containing every parameter the agent can adjust. For each parameter:

• Current value
• Description of what it controls
• Acceptable range or constraints
• Last modified date and the experiment that changed it

Use the template: [templates/parameters-template.md](templates/parameters-template.md).

4. experiments/ Directory — The Log

Each experiment gets its own file: experiments/NNN-hypothesis-slug.md.

Use the template: [templates/experiment-template.md](templates/experiment-template.md).

5. archive.json — Stepping Stones Archive

Maintain the top N parameter configs (default: 5) with their scores. Instead of always branching from the current best, the researcher can branch from any archived config — this is how you escape local optima.

{
  "entries": [
    {
      "id": "exp-025",
      "metric": 3.50,
      "params_snapshot": { },
      "lineage": "baseline → exp-001 → exp-009 → exp-025",
      "notes": "Best blending weights, no structural features"
    }
  ],
  "max_entries": 5
}

Keep configs that are diverse, not just the top 5 by raw score. When two configs have similar scores, keep the one that's more different from existing entries — this is the quality-diversity principle from MAP-Elites and from the HyperAgents archive.

6. coverage.json — Exploration Map

Auto-tracked file recording which parameter dimensions have been explored and how thoroughly:

{
  "dimensions": {
    "blending.recent_weight": {"experiments": 12, "range_tested": [0.50, 0.90], "last_explored": "exp-036"},
    "momentum.trend_weight": {"experiments": 0, "range_tested": null, "last_explored": null},
    "mean_reversion.strength": {"experiments": 2, "range_tested": [0.2, 0.5], "last_explored": "exp-041"}
  }
}

The researcher agent reads this before each experiment to steer toward under-explored dimensions. Don't spend 20 experiments micro-tuning one knob when another knob has never been touched.

Phase 3: Establish Baseline

1. Run the eval script with current parameters (full data and holdout)
2. Record the baseline metric on both splits
3. Log as experiment #000 (the starting point)
4. Initialize archive.json with the baseline config
5. Initialize coverage.json with all parameter dimensions at 0
6. Analyze the eval output — identify the biggest sources of error

Phase 4: Run the Loop (Three-Agent Pipeline)

The Researcher Agent

The researcher proposes and tests changes:

1. Check coverage map. Read coverage.json. If any dimension has 0 experiments, explore it before micro-tuning explored dimensions.
2. Analyze current weaknesses. Look at the eval output from the previous run. Identify the single biggest source of error. Quantify it — "Shop 1 week 3 is 7.5% under-forecast" is useful; "some weeks seem high" is not.
3. Consider branching from archive. If the current line of exploration has stalled (last 3 experiments failed), load a different config from archive.json and explore from there.
4. Form a hypothesis. Be specific. Target the biggest error or the least-explored dimension. One variable at a time.
5. Make the change. Modify ONLY the parameters file.
6. Evaluate. Run the eval script on the training data. Record the metric.
7. Pass to critic. If the training metric improved, hand off to the critic. If it worsened, revert immediately and log.

The Critic Agent

The critic validates with binary pass/fail gates. Verification is fundamentally easier than generation (the P vs NP intuition — checking an answer is easier than producing one). The critic doesn't propose changes — it only validates or rejects.

• Gate 1 — Holdout validation. Run the eval with --holdout. Did the metric improve on data the researcher never saw? If the training metric improved but holdout worsened → FAIL (overfitting).
• Gate 2 — No-regression check. Did any individual segment (shop, time period, category, traffic source) regress by more than 2× the overall improvement? Example: overall MAPE dropped 0.3%, but one segment got 1.5% worse → FAIL (robbing Peter to pay Paul).
• Gate 3 — Stability check. Is the improvement larger than the noise floor? If the metric only improved by 0.01% and the eval has 0.05% natural variance → FAIL (noise, not signal).
• Gate 4 — Directional sanity. Does the parameter change make domain sense? A critic with domain knowledge (from research.md) can flag: "You increased the holiday threshold to 3.0× — that means you're barely trimming holidays at all. This is likely overfitting to the few holiday weeks in the test set."

Decision: ALL gates must pass → ✅ KEEP. Any gate fails → ❌ DISCARD with the specific gate failure noted.

The critic's gates are defined in research.md and are configurable per domain. Some domains need stricter or different gates. The critic never modifies the eval script.

The Meta-Reviewer Agent (every N experiments)

Every 10 experiments (configurable), a fresh meta-reviewer agent reads the entire experiment log and does four things:

1. Pattern analysis. "The last 8 experiments all tried blending weight variations and 6 failed. The parameter space for blending is exhausted."
2. Coverage gaps. "Momentum, mean reversion, and week-of-month have never been explored. These represent structural changes that could break through the current ceiling."
3. Strategy update. Append to research.md under ## Learned Insights:

```markdown ## Learned Insights (auto-generated by meta-reviewer)

### Insight from review at experiment #030 (2026-04-03)

• Blending weights are saturated (0.78–0.82 range all within noise)
• The remaining error is structural: week 3 consistently under-forecasts by 7%
• RECOMMENDATION: Explore week-of-month adjustments before any more blending experiments
• RECOMMENDATION: Try per-segment parameter overrides — segments have different error profiles

```

1. Archive pruning. Suggest removing archive entries that are clearly dominated or too similar to other entries.

This is metacognitive self-modification from the HyperAgents paper — the system improves not just the parameters but its own improvement strategy. A meta-reviewer is the mechanism that stops the researcher from spending 50 experiments on blending weights when the real breakthrough is a structural change the researcher couldn't see from inside the loop.

Phase 5: Report Results

Present a summary table at the end of the session:

## Autoresearch Session: [System Name]
Date: YYYY-MM-DD
Duration: [time]

### Results
| Metric | Before | After | Change | Holdout |
|--------|--------|-------|--------|---------|
| [Primary] | X.XX | X.XX | -X.X% | X.XX |
| [Secondary] | X.XX | X.XX | -X.X% | X.XX |

### Experiments Run: N (kept: K, critic-rejected: C, researcher-reverted: R)
| # | Hypothesis | Change | Train | Holdout | Critic | Decision |
|---|------------|--------|-------|---------|--------|----------|
| 1 | [description] | [param: old → new] | -X.X% | -X.X% | ✅ PASS | ✅ KEEP |
| 2 | [description] | [param: old → new] | -X.X% | +X.X% | ❌ Gate 1 | ❌ DISCARD |

### Critic Gate Statistics
| Gate | Passed | Failed | Rejection Rate |
|------|--------|--------|----------------|
| Holdout validation | 15 | 3 | 17% |
| No-regression | 16 | 2 | 11% |
| Stability | 14 | 4 | 22% |
| Directional sanity | 18 | 0 | 0% |

### Coverage Map
| Dimension | Experiments | Range Tested | Status |
|-----------|-------------|--------------|--------|
| blending weights | 15 | 0.50–0.90 | Saturated |
| momentum | 0 | — | Unexplored |

### Meta-Reviewer Insights
[Key strategy changes recommended during the session]

### Archive (Top 5 Configs)
[Diverse set of high-performing parameter configs]

### Remaining Weaknesses
[Biggest remaining errors — seeds for the next session]

Phase 6: Optionally Set Up Recurring Runs

If the user wants continuous improvement:

• Schedule a recurring run (cron, launchd, or any scheduler your agent supports)
• Each session picks up from the current parameters and archive
• The meta-reviewer's learned insights carry forward across sessions
• Results are posted or logged for human review

Rules of Engagement

Core Rules (from Karpathy's original)

1. One variable at a time. Never change two parameters in one experiment. If you're tempted to change multiple things, run them as separate experiments.
2. The eval script is sacred. No agent ever modifies the eval script. If the eval is wrong, tell the human. Modifying the eval is like a student grading their own test.
3. Always revert failures. If a change doesn't improve the metric (or fails critic validation), restore the parameters file before the next experiment.
4. Log everything. Even failed experiments are valuable. "Tuesday multiplier 0.75 overcorrected" prevents trying 0.70 next. Critic rejections are especially valuable — they reveal overfitting patterns.
5. Target the biggest error. Don't fine-tune a 2% error when there's a 25% error somewhere else.
6. Be honest about sample size. If a parameter only has 2 data points, note it. Don't over-tune on small samples.

v2 Rules (new)

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pbc-os
• Source: pbc-os/smb-starter-kit
• License: MIT

Install and usage instructions live in the source repository linked above.