Experiment Planner

Experiment Planner

Overview

Use this skill before paper writing when the user needs to turn a research idea into a testable story and experiment plan. It is an adapter over existing research-agent ideas, not a replacement for the user's writing, review, rebuttal, figure, evidence, or GitHub release skills.

Core Boundaries

• Default domain: general deep learning and computer science research. Adapt to

collaborative perception, 3D perception, or autonomous driving only when the task context calls for it.

• For discussion-only planning, keep output in chat unless the user asks for a

saved artifact. When the user asks to implement or run experiments in a repository, persist the pre-run result contract: update the paper's final LaTeX tables when a manuscript is in scope; otherwise update the existing experiment-planning document or create experiment-plan.md at the repository root.

• Do not launch long experiments, deploy GPU jobs, modify code, or retry failed

runs unless the user explicitly asks for execution.

• Do not replace paper-section-playbook, paper-refinement-skills,

paper-review-panel, rebuttal-response-skills, paper-visual-craft, or github-project-release; hand off to them only after the research plan or results are ready.

• Treat external projects as references, not installed dependencies. Read

references/source-map.md before discussing provenance or upgrading this skill from upstream sources.

Default Workflow

1. Grill consensus: use $grill-me style interaction to clarify problem,

motivation, proposed claim, baseline/control, compute budget, success criteria, and unacceptable shortcuts. Ask one high-impact question at a time when the answer changes the experiment plan.

1. Literature inspiration: after a preliminary consensus, use

$research-evidence for related papers, novelty risk, prior experiment patterns, and unsupported claims. Use $search-first when the task may need existing code, datasets, tools, or implementations.

1. Story viability check: decide whether the idea can support a clean paper

story: important problem, credible gap, specific method difference, feasible validation, and claims that will not outrun the evidence.

1. Claim freeze: freeze the smallest verifiable claim before planning runs.

Avoid changing the story repeatedly while experiments are running.

1. Paper/table contract freeze: before scheduling runs, define the final

main-result, ablation, and necessary diagnostic tables. For every metric, record its plain-language definition, unit, direction, aggregation, and any delta reference. Use -- for unavailable values and do not write claims from placeholder cells.

1. Idea validation first: design the smallest pilot/smoke/sanity experiment

that can falsify or support the core hypothesis. If multiple GPUs are idle, parallelize only independent exploration runs with clear ownership.

1. Minimum sufficient matrix: only after the pilot passes, add the main

result and claim-critical ablations. Add robustness, diagnostics, efficiency, qualitative results, or failure analysis only when they support a paper claim or answer a credible reviewer question; do not add them for symmetry.

1. Subagent coordination: keep the main session responsible for planning,

task decomposition, and final result acceptance. Use explorer for read-only repo/config/protocol investigation. Use worker for implementation with explicit file or module ownership. Do not manually override subagent model or reasoning settings unless the user explicitly requests it.

1. Run discipline: test that the command starts and produces plausible small

outputs; remove test data after smoke checks; launch the full run only after sanity passes; inspect the first few samples/logs/artifacts; stop continuous monitoring once the run is confirmed healthy unless the user asks otherwise.

1. Conditional seed policy: record an existing seed and keep compared runs

under the same evaluation and checkpoint-selection policy. For expensive training such as autonomous driving, accept a single training run by default. Require repeated seeds only when variance could change the central claim, the margin is small, the runs are inexpensive, or the venue requires them.

1. Default decisions: ask only questions whose answers materially change

the plan. If a non-critical choice goes unanswered, use the recommended default and record it as an assumption.

Output Contract

Default to a concise in-chat experiment matrix. Before producing a matrix, read references/experiment-matrix.md.

The matrix must include:

• research question
• core hypothesis
• paper claim
• storyline
• literature inspiration
• baseline/control
• table contract
• metric definitions
• idea validation experiment
• expected signal
• failure modes
• diagnostic checks
• follow-up experiments
• subagent/task ownership
• compute/resource assumptions
• seed policy
• success gate
• claim gate
• next action

Use unknown or needs user input for unresolved fields instead of inventing project facts. Keep recommendations executable, but do not perform execution inside this skill unless the user asks for implementation or running commands.

Handoff Rules

• Use $research-evidence before making novelty, citation, or literature

coverage claims.

• Use $search-first before proposing new implementation utilities, pipelines,

tool integrations, or dataset-processing code.

• Use writing skills only after the experiment story is stable enough to draft

a paper section, rebuttal, review, table, or figure.

• For code work, assign worker tasks with disjoint write scopes and remind the

worker not to revert others' changes.

• For investigation, assign explorer tasks that are specific, read-only, and

non-overlapping with the main session's current work.

Failure Modes To Catch

• The idea is interesting but not falsifiable with available data or compute.
• The proposed contribution is only a presentation change, not a testable method

or analysis difference.

• The baseline/control is missing, unfair, or weaker than the claim requires.
• The pilot experiment cannot distinguish mechanism from implementation noise.
• The plan jumps to full benchmark runs before smoke and sanity checks pass.
• The story changes after seeing results without recording a clear reason.
• Subagents receive vague tasks, overlapping write scopes, or authority to run

long jobs without main-session acceptance.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sidiangongyuan
• Source: sidiangongyuan/codex-skills-library
• License: MIT

Install and usage instructions live in the source repository linked above.