# Autoresearch

> Use for autoresearch, autonomous experiments, optimization loops, "optimize X overnight/in a loop", or "experiment loop"; sets up iterative trials for an optimization target.

- **Type:** Skill
- **Install:** `agentstack add skill-paulrberg-agent-skills-autoresearch`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [PaulRBerg](https://agentstack.voostack.com/s/paulrberg)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [PaulRBerg](https://github.com/PaulRBerg)
- **Source:** https://github.com/PaulRBerg/agent-skills/tree/main/skills/autoresearch

## Install

```sh
agentstack add skill-paulrberg-agent-skills-autoresearch
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Autoresearch

Autonomous experiment loop: try ideas, measure results, keep what works, discard what doesn't, never stop.

Works for any optimization target: test speed, bundle size, LLM training, build times, Lighthouse scores, binary size, latency, memory usage.

## Setup

If `autoresearch.md` already exists in the working directory, **skip setup and resume the loop** — read `autoresearch.md`, `autoresearch.jsonl`, and `git log`, then continue experimenting.

Otherwise:

1. **Gather context**: Ask (or infer from `$ARGUMENTS` and conversation) the **Goal**, **Command** to benchmark, **Primary metric** (name + direction), **Files in scope**, and **Constraints**.
2. **Create branch**: `git checkout -b autoresearch/-` (e.g. `autoresearch/test-speed-2026-03-21`).
3. **Read source files**: Understand the workload deeply before writing anything. Read every file in scope.
4. **Write session files**: Create `autoresearch.md` and `autoresearch.sh` (see templates below). If constraints require correctness validation (tests must pass, types must check), also create `autoresearch.checks.sh`. Commit all.
5. **Run baseline**: Execute the first experiment with no changes to establish the baseline metric.
6. **Start looping**: Begin the experiment loop immediately after the baseline is logged.

### `autoresearch.md`

The heart of the session. A fresh agent with no context should be able to read this file alone and run the loop effectively. Invest time making it excellent.

```markdown
# Autoresearch: 

## Objective

## Metrics
- **Primary**:  (, lower/higher is better)
- **Secondary**: , , ...

## How to Run
`./autoresearch.sh` — outputs `METRIC name=value` lines.

## Files in Scope

## Off Limits

## Constraints

## What's Been Tried

```

Update `autoresearch.md` periodically — especially "What's Been Tried" — so resuming agents have full context.

### `autoresearch.sh`

Bash script that runs the benchmark and outputs structured metrics.

```bash
#!/bin/bash
set -euo pipefail

# Pre-checks (fast,  /tmp/autoresearch-output.log 2>&1

# Extract and output metrics as METRIC lines
val_bpb=$(grep "^val_bpb:" /tmp/autoresearch-output.log | awk '{print $2}')
echo "METRIC val_bpb=$val_bpb"
```

Rules:

- Use `set -euo pipefail`.
- Output `METRIC name=value` lines to stdout (one per metric). The primary metric name must match what's documented in `autoresearch.md`.
- Metric names: word chars, dots, or `µ` (e.g. `val_bpb`, `total_µs`, `bundle.size_kb`).
- Keep the script fast — every second is multiplied by hundreds of runs.
- For fast/noisy benchmarks (\&1 | tail -50
pnpm typecheck 2>&1 | grep -i error || true
```

When this file exists:

- Run it after every **passing** benchmark (exit 0).
- If checks fail, log the experiment as `checks_failed` and revert.
- Check execution time does NOT affect the primary metric.
- Keep output minimal — suppress verbose progress, only show errors.

When this file does not exist, skip checks entirely.

## The Experiment Loop

**LOOP FOREVER.** Never ask "should I continue?" — the user expects autonomous work.

Each iteration:

01. **Formulate hypothesis**: Based on prior results, source code understanding, and any ideas in `autoresearch.ideas.md`, choose what to try next.
02. **Edit code**: Modify the in-scope files. Make a single, focused change per experiment.
03. **Commit**: `git add -A && git commit -m ""`
04. **Run benchmark**:
    ```bash
    timeout 600 ./autoresearch.sh > run.log 2>&1
    ```
    If the command times out or crashes, treat it as a failure.
05. **Parse metrics**: Extract `METRIC` lines from the output:
    ```bash
    grep '^METRIC ' run.log
    ```
    If no METRIC lines found, the run crashed — read `tail -50 run.log` for the error.
06. **Run checks** (if `autoresearch.checks.sh` exists and benchmark passed):
    ```bash
    timeout 300 ./autoresearch.checks.sh > checks.log 2>&1
    ```
07. **Evaluate and log**:
    - **Improved** (primary metric better than best so far) → status `keep`. The commit stays.
    - **Worse or equal** → status `discard`. Revert: stage autoresearch files first, then reset.
    - **Crash** (benchmark failed) → status `crash`. Fix if trivial, otherwise revert and move on.
    - **Checks failed** → status `checks_failed`. Revert.
08. **Log to JSONL**: Append one line to `autoresearch.jsonl`:
    ```json
    {"run":1,"commit":"a1b2c3d","metric":0.9979,"metrics":{"val_bpb":0.9979,"peak_vram_mb":45060.2},"status":"keep","description":"baseline","timestamp":1711036800000,"confidence":null}
    ```
09. **On discard/crash/checks_failed — revert code changes**:
    ```bash
    # Preserve autoresearch session files, revert everything else
    git add autoresearch.jsonl autoresearch.md autoresearch.sh autoresearch.ideas.md autoresearch.checks.sh 2>/dev/null || true
    git checkout -- .
    git clean -fd
    ```
10. **Check confidence**: After 3+ runs, run the confidence script from the skill's installation directory. On macOS, avoid `readlink -f`; resolve the skill directory once and invoke the script directly:
    ```bash
    skill_dir="$HOME/.agents/skills/autoresearch"
    bash "$skill_dir/scripts/confidence.sh"
    ```
    Interpret the score:
    - **>= 2.0x**: Improvement is likely real (green).
    - **1.0-2.0x**: Above noise but marginal (yellow).
    - **< 1.0x**: Within noise — consider re-running to confirm (red).
11. **Update session**: Periodically update `autoresearch.md` "What's Been Tried" section and run the summary script to review progress.

Repeat forever until interrupted.

## JSONL Schema

Each line in `autoresearch.jsonl` is a JSON object:

| Field         | Type           | Description                                    |
| ------------- | -------------- | ---------------------------------------------- |
| `run`         | number         | 1-indexed experiment count                     |
| `commit`      | string         | Short git SHA (7 chars)                        |
| `metric`      | number         | Primary metric value                           |
| `metrics`     | object         | All metrics dict (primary + secondary)         |
| `status`      | string         | `keep`, `discard`, `crash`, or `checks_failed` |
| `description` | string         | What this experiment tried                     |
| `timestamp`   | number         | Unix timestamp (ms)                            |
| `confidence`  | number or null | MAD-based confidence score (null if \<3 runs)  |

## Resuming

When `autoresearch.md` exists in the working directory:

1. Read `autoresearch.md` for full context (objective, what's been tried, constraints).
2. Read `autoresearch.jsonl` to reconstruct state (best metric, run count, last segment).
3. Read `git log --oneline -20` for recent commit history.
4. Check `autoresearch.ideas.md` if it exists — prune stale entries, experiment with promising ones.
5. Continue the loop from where it left off. Do not re-run the baseline.

## Ideas Backlog

When you discover complex but promising optimizations you won't pursue right now, append them as bullets to `autoresearch.ideas.md`. Don't let good ideas get lost.

On resume, check this file — prune stale/tried entries, experiment with the rest. When all paths are exhausted, delete the file and write a final summary to `autoresearch.md`.

## Loop Rules

See `references/loop-rules.md` for the full reference. Key rules:

- **Primary metric is king.** Improved → keep. Worse/equal → discard.
- **Simpler is better.** Remove code for equal perf = keep. Ugly complexity for tiny gain = discard.
- **Don't thrash.** Repeatedly reverting the same idea? Try something structurally different.
- **Think longer when stuck.** Re-read source files, reason about what the CPU/compiler/runtime is actually doing. Deep understanding beats random variation.
- **Crashes**: fix if trivial (typo, missing import), otherwise log and move on. Don't over-invest.
- **NEVER STOP.** The user may be away for hours. Keep going until interrupted.

## User Messages During Experiments

If the user sends a message while an experiment is running, finish the current run-evaluate-log cycle first, then incorporate their feedback in the next iteration.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [PaulRBerg](https://github.com/PaulRBerg)
- **Source:** [PaulRBerg/agent-skills](https://github.com/PaulRBerg/agent-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-paulrberg-agent-skills-autoresearch
- Seller: https://agentstack.voostack.com/s/paulrberg
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.