Computational Experiments

Computational Experiments

> Lifecycle skill for algorithmic research projects where the code IS the scientific contribution.

Modes

| Mode | What it does | Phases | |------|-------------|--------| | Scaffold | Create/audit package structure + algorithm skeleton | 1–2 | | Experiment | Design and run pre-specified sweep campaigns | 1, 3–4 | | Explore | Adaptive experiment loop: modify → run → evaluate → keep/discard | 1, 3E–4 | | Autonomous | Parallel self-correcting sweep with sub-agents | 1, 3A–4 | | Figures | Generate publication output from results | 1, 4 | | Full | Complete pipeline | 1–5 |

Default: Full. Detect mode from user request or ask if ambiguous.

--scaffold Flag

Sets a stage progression template for the experiment campaign. Templates provide structured checklists and exit criteria for each stage.

| Scaffold | Stages | Best for | |----------|--------|----------| | standard | Init → Tune → Creative → Ablate | Algorithm development, ML, simulation | | robustness | Main spec → Alternatives → Placebo → Sensitivity | Causal inference, econometrics | | replication | Exact → Our data → Extensions → Robustness | Replicate-and-extend papers |

Templates live in templates/experiments/. When a scaffold is active:

1. Present the current stage's checklist before starting work
2. Gate progression: don't move to the next stage until exit criteria are met
3. Log stage transitions in the experiment breadcrumb

Default (no flag): user-defined stages (current behavior). Scaffold is a guide, not a cage — users can skip stages or reorder with explicit acknowledgment.

--budget Flag

Sets a campaign-level time budget in minutes for the entire experiment run. When set:

1. Record start time at the beginning of Phase 3 (any variant)
2. Check remaining budget before launching each new config, sweep batch, or explore iteration
3. Soft stop when budget is exhausted: finish the current run, save all results collected so far, skip remaining configs
4. Never hard-kill a running experiment mid-execution — always let the current run complete

| Mode | How budget applies | |------|-------------------| | Experiment | Skip remaining sweep configs when time is up | | Explore | Exit the explore loop at the next iteration boundary | | Autonomous | Do not spawn new agent batches; let running batches finish | | Figures | Budget does not apply (figure generation is fast) |

Reporting: When a budget triggers early stop, append to the breadcrumb:

- **Budget:** Stopped after / configs (budget:  min, elapsed:  min)

Default: No budget (run all configs to completion). Typical values: --budget 30 for quick exploration, --budget 120 for overnight sweeps.

Use Experiment when the design space is known upfront (grid/random sweep). Use Explore when the design space is unknown and the agent should adaptively search (inspired by Karpathy's autoresearch).

When to Use

• "Set up experiments for my algorithm" / "Scaffold the project"
• "Run the benchmark" / "Design the sweep"
• "Explore what auction parameters reduce collusion" / "Try different architectures"
• "Generate convergence plots" / "Make publication figures from results"
• Any project where algorithms, simulations, or optimisation loops are the contribution

When NOT to Use

• Empirical data analysis (observational/survey data) → /data-analysis
• Experimental design for human subjects → /experiment-design
• Causal inference strategy → /causal-design
• LaTeX compilation → /latex

Workflow

Phase 1: Detect & Configure

1. Read project context: CLAUDE.md, MEMORY.md, .context/project-recap.md if they exist
2. Detect structure: Scan for src/, experiments/, tests/, pyproject.toml, setup.py
3. Detect language: Check existing files. Default: Python with uv. Ask if ambiguous.
4. Read MEMORY.md: Look for [LEARN:code] tags, notation registry, key decisions — apply established conventions
5. Determine mode: From user request or ask. If existing package exists, skip scaffold.
6. Inventory existing code: Map modules, entry points, config objects, result files

Read references/package-scaffold.md if scaffold mode is active.

Phase 2: Scaffold

Skip if: Package structure already exists and user requested experiment/figures mode.

1. Package structure: Create src//, tests/, experiments/configs/, scripts/
2. Build config: pyproject.toml with hatchling, dev dependencies (pytest, matplotlib, numpy)
3. Algorithm skeleton: Base classes (Algorithm, Experiment, Metric) with # TODO: markers. For multi-agent projects, use BaseAgent, Environment, MultiAgentSimulation instead — see references/multi-agent-patterns.md
4. Test skeleton: Unit test stubs, convergence test template, smoke test
5. Experiment infrastructure: Config dataclass template, runner script template
6. Gitignore: Add results/, *.pkl, wandb/, __pycache__/

Read references/algorithm-templates.md for skeleton code. Read references/package-scaffold.md for directory layout.

Gate: Verify package installs with uv pip install -e ".[dev]" before proceeding.

Phase 3: Experiment Design

Prerequisite: Working package (Phase 2 or pre-existing).

1. Config schema: Python dataclass with validation, defaults, serialization to/from YAML
2. Config hashing: SHA-256 fingerprint for reproducibility tracking (see references/experiment-patterns.md)
3. Sweep definitions: Grid sweep (all combinations), random sweep (budget-limited), manual configs
4. Seed management: np.random.default_rng(seed) everywhere. Seeds passed through config, never global state.
5. Runner script: Config → initialize → loop → collect → save. Reads config, runs n_seeds repetitions, saves per-seed results.
6. Baseline implementations: Same interface as main algorithm, registered in config
7. Result aggregation: Per-seed CSV → aggregated stats (mean ± std). Canonical column naming.
8. Parallelization: concurrent.futures.ProcessPoolExecutor for independent seeds/configs. For large sweeps (10+ configs × 10+ seeds, GPU-bound, or >30-min runs): move to [HPC cluster] HPC — see [docs/guides/hpc.md](../../docs/guides/hpc.md) in Task Management and copy templates/slurm/{array,gpu}.sbatch into hpc/ with sync-up.sh / sync-down.sh. Recent reference implementations: Projects/NLP/{example-project-a,benchmark-gaming-llm-safety}/hpc/.
9. Checkpointing: Save intermediate results to allow resume on crash
10. Dual output: Dated archive + "latest" symlink for quick access (see references/experiment-patterns.md)

For multi-agent simulations, also include:

1. Agent composition: Heterogeneous population specs with distribution presets (uniform, weighted, custom)
2. Feature toggles: Opt-in capabilities per agent type (memory, messaging) — enables ablation studies
3. Messaging service: Round-based broadcast channel for inter-agent communication
4. Multi-level metrics: Agent-level (individual performance) + system-level (aggregate outcomes) + emergent-level (herding, convergence)

See references/multi-agent-patterns.md for all multi-agent patterns.

Read references/experiment-patterns.md for config, sweep, and runner patterns.

Gate: Run a smoke test — single config, single seed, verify output files are created.

Phase 3E: Explore Loop (Adaptive Experiments)

Use instead of Phase 3 when: The design space is unknown, the user wants to adaptively search rather than run a pre-specified sweep, or the user says "explore", "try things", "see what works".

Read references/explore-loop.md for the full protocol. Summary:

1. Branch: Create experiments/ branch (e.g. experiments/mar14-collusion-params)
2. Baseline: Run the code as-is, record baseline metric in results.tsv
3. Loop (autonomous until interrupted or budget exhausted):

• Propose a modification based on results so far (informed by prior keep/discard outcomes)
• Apply the change to the target file(s)
• git commit the change
• Run with time budget: timeout s uv run python > run.log 2>&1
• Extract metric from output
• Log to results.tsv: commit hash, metric, status (keep/discard/crash), description
• If improved → keep (advance branch)
• If not improved → discard (git reset --hard HEAD~1)
• If crashed → attempt fix once, else discard and move on

1. Stop when: user interrupts, --budget time exhausted (campaign-level), per-run timeout exceeded, or N consecutive discards with no progress

Key principles:

• One change per iteration — never stack multiple untested changes
• Log every attempt, including discards and crashes
• The TSV is the ground truth — it captures the full exploration history
• Simplicity criterion: prefer changes that simplify the code for equal or better metrics

Gate: results.tsv exists with at least a baseline entry before entering the loop.

Phase 3A: Autonomous Sweep (Parallel Self-Correcting)

Use instead of Phase 3 when: 10+ configs, known failure-prone experiments, or user says "autonomous", "hands-off", "self-correcting sweep".

Read references/autonomous-sweep.md for the full protocol. Summary:

1. Batch configs into groups of max 4 parallel agents
2. Spawn sub-agents — each runs its batch, detects failures by signature (shape mismatch, NaN, empty array, timeout), diagnoses, fixes, retries (max 3 cycles)
3. Aggregate results — coordinator collects batch results, reports success/fixed/unresolved counts
4. Generate figures for all successful configs

Key constraints:

• Max 4 parallel agents
• Max 3 retry cycles per config
• Agents must not modify shared code simultaneously
• Each fix is committed for traceability

Graceful degradation: If some batches fail while others succeed, collect all successful results and report failures. Only stop entirely if ALL batches fail. See shared/skill-design-patterns.md (Graceful Degradation section).

Gate: At least one config must succeed. If all fail, report the unresolved errors and stop.

Breadcrumb: After any Phase 3 variant completes, append to .planning/state.md (if exists) or .context/current-focus.md:

### [/computational-experiments] Experiments complete [YYYY-MM-DD HH:MM]
- **Done:** [N configs run, N seeds, mode: experiment/explore/autonomous]
- **Outputs:** [result files at , N successful / N total]
- **Next:** Publication output (figures/tables)

Phase 4: Publication Output

Prerequisite: Result files exist in results/ or experiments/results/.

1. Convergence plots: Metric vs iteration/budget (mean ± std across seeds, multiple methods)
2. Comparison charts: Bar chart or table comparing methods across problems
3. Ablation plots: Performance with/without each component (for multi-agent: use feature toggle ablation configs)
4. Sensitivity analysis: Metric vs parameter (line plot with shaded confidence band)
5. Tables: Performance tables exported as .tex via \input{} — never hard-code results
6. Figure script: scripts/make_all_figures.py that regenerates all figures from saved results

Read references/figure-recipes.md for matplotlib recipes.

Output routing:

• Figures → paper/figures/ as PDF (per overleaf-separation rule)
• Tables → paper/tables/ as .tex (per no-hardcoded-results rule)
• Figure scripts → scripts/ (never inside paper/)

Phase 5: Audit & Save

1. Reproducibility check:

• Seed determinism: same config + seed → same output?
• Config completeness: can results be regenerated from saved configs?
• Output freshness: are figures newer than their source data?
• Dependency pinning: are versions locked in pyproject.toml or uv.lock?

1. Code review: Invoke /code-review on all generated scripts (via Skill tool)
2. Record learnings: Write [LEARN:code] tags for project-specific conventions discovered
3. Suggest next steps: compilation with /latex, additional experiments, /replication-package

Breadcrumb: After Phase 5 completes, append to .planning/state.md (if exists) or .context/current-focus.md:

### [/computational-experiments] Phase 5 complete [YYYY-MM-DD HH:MM]
- **Done:** [reproducibility check, code review score, N learn tags recorded]
- **Outputs:** [figures at , tables at ]
- **Next:** [suggested next steps]

Evolution Protocol

This skill improves with each invocation on a project:

1. Before work: Read MEMORY.md for existing [LEARN:code] entries and Key Decisions
2. During work: Note any convention, naming pattern, or structural choice that should persist
3. After work: Record [LEARN:code] tags and update Key Decisions table

Examples of learnings to capture:

• [LEARN:code] This project uses ElicitationConfig dataclass, not YAML files
• [LEARN:code] Metrics are in src/utils/metrics.py, not a separate package
• [LEARN:code] Seeds are managed via utils/seeds.py with MASTER_SEED + offset

Cross-References

| Resource | When read | |----------|-----------| | references/package-scaffold.md | Phase 2 (project structure) | | references/algorithm-templates.md | Phase 2 (skeleton code) | | references/experiment-patterns.md | Phase 3 (configs, sweeps, runners, config hashing, dual output) | | references/multi-agent-patterns.md | Phase 2–3 (agent composition, messaging, multi-level metrics) | | references/multi-agent-infrastructure.md | Phase 2–3 (feature toggles, config hashing, simulation runner, visualization) | | references/explore-loop.md | Phase 3E (adaptive explore loop) | | references/autonomous-sweep.md | Phase 3A (parallel self-correcting sweep) | | references/figure-recipes.md | Phase 4 (matplotlib recipes) | | shared/publication-output.md | Phase 4 (table/figure format standards) | | shared/multi-language-conventions.md | Phase 1 (if non-Python) | | docs/guides/hpc.md (Task Management) | Phase 3 (move to Avon for large/GPU/long sweeps) | | templates/slurm/*.sbatch (Task Management) | Phase 3 (drop-in SLURM templates; all log git-SHA to OUT_DIR) | | no-hardcoded-results rule | Phase 4 (output routing) | | overleaf-separation rule | Phase 4 (file placement) | | /code-review skill | Phase 5 (auto-invoked) | | /data-analysis skill | Redirect if task is empirical, not computational | | /replication-package skill | Phase 5 (suggested next step) | | /cross-language-check skill | Phase 5 (suggested next step for verification) | | references/multi-analyst-design.md | Phase 3–5 (many-analysts robustness diagnostic) | | shared/worker-critic-protocol.md | Phase 3–4 (inline review of generated code/results) | | shared/checkpoint-resumability.md | All phases (save/resume on crash) | | templates/experiments/standard.md | --scaffold standard (init/tune/creative/ablate) | | templates/experiments/robustness.md | --scaffold robustness (econometrics robustness) | | templates/experiments/replication.md | --scaffold replication (replicate-and-extend) | | /figure-feedback skill | Phase 4 (VLM analysis of generated plots) |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: flonat
• Source: flonat/flonat-research
• License: MIT

Install and usage instructions live in the source repository linked above.