# Experiment Design

> >

- **Type:** Skill
- **Install:** `agentstack add skill-fcakyon-phd-skills-experiment-design`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [fcakyon](https://agentstack.voostack.com/s/fcakyon)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [fcakyon](https://github.com/fcakyon)
- **Source:** https://github.com/fcakyon/phd-skills/tree/main/plugin/skills/experiment-design

## Install

```sh
agentstack add skill-fcakyon-phd-skills-experiment-design
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Experiment Design Methodology

You are helping a researcher design rigorous experiments. Follow this methodology systematically.

## Step 1: Understand the Research Question

Before designing any experiment:
- Ask what specific hypothesis or claim the experiment should support
- Identify the dependent variable (metric) and independent variables (factors)
- Clarify the baseline: what is the current best result or default configuration?

## Step 2: Single-Variable Isolation

Every ablation study must change exactly ONE variable at a time. For each factor:

1. **Define the factor** — what is being varied (e.g., loss function, learning rate, architecture component)
2. **List levels** — all values this factor will take (e.g., CE, focal, VAR)
3. **Fix everything else** — document what stays constant (seed, data split, epochs, hardware)
4. **Predict outcome** — before running, state what you expect and why

Template for each ablation row:
```
| Run ID | Factor | Value | Fixed Config | Expected Outcome |
|--------|--------|-------|-------------|-----------------|
```

## Step 3: Experiment Matrix

For multi-factor studies, use a structured matrix:

1. **Full factorial** — if factors are few (≤3) and levels are few (≤3 each)
2. **Sequential elimination** — if factors are many: run single-factor ablations first, then combine winners
3. **Latin square** — if full factorial is too expensive: sample representative combinations

Always calculate total runs before committing:
```
Total runs = product of all factor levels
GPU hours = total runs × hours_per_run
```

## Step 4: Resource Estimation

For each experiment plan, estimate:
- **GPU hours**: runs × time_per_run (check with user's hardware)
- **API costs**: if using external APIs (Gemini, OpenAI), estimate tokens × price
- **Wall clock time**: accounting for sequential dependencies and GPU availability
- **Storage**: checkpoint sizes × number of runs

Flag if total cost exceeds reasonable bounds and suggest prioritization.

## Step 5: Config Stub Generation

Generate configuration stubs that match the user's existing config format. Read existing configs first to match:
- File format (YAML, JSON, TOML)
- Key naming conventions
- Directory structure for outputs
- Logging/tracking integration (wandb, neptune, tensorboard)

## Step 6: Execution Plan

Create a concrete execution plan:
1. Order runs by dependency (baselines first, then ablations)
2. Identify which runs can be parallelized across GPUs
3. Create a shell script or batch runner matching the project's existing patterns
4. Include checkpointing strategy for long runs

## Step 7: Analysis Plan

Before running, define how results will be analyzed:
- Which metrics to compare (primary + secondary)
- Statistical significance test if applicable (paired t-test, bootstrap CI)
- How to handle failed/crashed runs
- Visualization: what plots to generate (comparison tables, bar charts, learning curves)

## Verification Checkpoints

Before finalizing the experiment plan:
- [ ] Each ablation changes exactly one variable
- [ ] Baseline is clearly defined and will be run with same setup
- [ ] Resource estimate is within budget
- [ ] Config stubs match existing project format
- [ ] Analysis plan is defined before execution begins
- [ ] Seeds are fixed for reproducibility

## Output Format

Always produce:
1. **Experiment matrix table** — all runs with their configurations
2. **Resource estimate** — GPU hours, API costs, storage
3. **Execution script** — ready-to-run commands matching project conventions
4. **Analysis plan** — metrics, comparisons, visualizations

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [fcakyon](https://github.com/fcakyon)
- **Source:** [fcakyon/phd-skills](https://github.com/fcakyon/phd-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-fcakyon-phd-skills-experiment-design
- Seller: https://agentstack.voostack.com/s/fcakyon
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.