Experiment Design

Experiment Design

> Interview-driven design workflow producing design documents, power analysis scripts, and pre-analysis plans.

Modes

| Mode | What it produces | Entry point | |------|-----------------|-------------| | Power | Power analysis script + sample size table | "How many participants do I need?" | | Design | Full design document (hypotheses, conditions, measures, randomization) | "Design my experiment" | | PAP | Pre-analysis plan (AEA/OSF/EGAP format) | "Write a PAP" | | Survey | Structured survey specification from natural language or QSF | "Build a survey" / "Parse my Qualtrics" |

Default: Design. If user provides a .qsf file, auto-select Survey mode.

When to Use

• Designing a new experiment or survey
• Calculating required sample sizes
• Writing or auditing a pre-analysis plan
• Parsing a Qualtrics .qsf file to understand its structure
• Building a survey specification from a natural language description

When NOT to Use

• Running the analysis → /data-analysis
• Auditing identification strategy for observational studies → /causal-design
• Generating synthetic test data → /synthetic-data

Shared References

• Method probing questions: shared/method-probing-questions.md — ask before designing (Experiments/RCTs, Survey sections)
• Validation tiers: shared/validation-tiers.md — tier determines required power and pre-registration
• Escalation protocol: shared/escalation-protocol.md — escalate when design has validity threats
• Engagement-stratified sampling: shared/engagement-stratified-sampling.md — stratify social media samples by engagement
• Inter-coder reliability: shared/intercoder-reliability.md — reliability planning for content analysis designs

---

Mode: Power

Read references/power-analysis-recipes.md for language-specific code patterns.

Workflow

1. Interview — ask for:

• Primary outcome variable and expected effect size (or domain norms)
• Design type (between-subjects, within-subjects, factorial, cluster-randomized)
• Number of conditions/groups
• Significance level (default: 0.05) and desired power (default: 0.80)
• Any clustering or stratification

1. Generate script — R (DeclareDesign/pwr) or Python (statsmodels.stats.power)
2. Execute and report — produce a sample size table showing N for power = {0.80, 0.90, 0.95}
3. Write to project — save script to code/power_analysis.R (or .py), results to output/power_analysis_results.md

HPC escalation: If the power analysis uses Monte Carlo simulation (e.g., DeclareDesign with >10k replications, or a multi-design sweep), move execution to your HPC cluster — drop the simulation script into hpc/ with templates/slurm/array.sbatch (array over seeds/designs). The SHA-logging snippet in the template pins results to the DGP version. See [docs/guides/hpc.md](../../docs/guides/hpc.md).

Effect Size Guidance

If the user doesn't know the expected effect size, guide them:

| Source | How to use | |--------|-----------| | Prior literature | "What did similar studies find?" | | Pilot data | Calculate from pilot descriptives | | SESOI | "What's the smallest effect worth detecting?" | | Domain norms | Cohen's benchmarks as absolute last resort (small=0.2, medium=0.5, large=0.8 for d) |

Never default to Cohen's benchmarks without acknowledging they are arbitrary.

---

Mode: Design

Workflow

1. Research question interview — structured questions:

• What is the causal question?
• What is the treatment / intervention?
• What is the primary outcome? Secondary outcomes?
• What is the target population?
• What is the assignment mechanism? (random, stratified, clustered, matched)

1. Design specification — produce a structured document covering:

• Hypotheses (directional, with expected signs)
• Conditions (treatment arms, control)
• Randomization procedure
• Outcome measures and scales
• Sample and recruitment strategy
• Timeline

1. Identification check — state the estimand, identifying assumptions, and potential threats
2. Write design document — save to docs/experiment-design.md or project-appropriate location
3. Lock the design — record in MEMORY.md Estimand Registry (if project has one) or flag for the user to lock before analysis

This design document is what /data-analysis Phase 3 checks for before allowing estimation.

---

Mode: PAP

Read references/pap-template.md for the full template structure.

Workflow

1. Check for existing design — look for design document in docs/, log/plans/, or .context/
2. If no design exists — run Design mode first, then continue
3. Generate PAP — structured pre-analysis plan following AEA/OSF/EGAP conventions:

• Study information (title, authors, IRB, registration)
• Design overview (hypotheses, conditions, randomization, sample)
• Data collection (instruments, timing, blinding)
• Analysis plan (main specification, inference, multiple testing corrections)
• Robustness and sensitivity analyses
• Power analysis (reference or embed results from Power mode)

1. Output — save to docs/pre-analysis-plan.md (or .tex if user prefers LaTeX)
2. Registration prompt — remind user to register at AEA RCT Registry, OSF, or EGAP

---

Mode: Survey

Survey mode is a first-class capability with two entry points: QSF parsing and natural language construction.

Entry Point A: QSF Parsing

When user provides a Qualtrics .qsf file:

1. Parse the JSON — extract survey flow, blocks, questions, embedded data, skip logic
2. Map question types — read references/qsf-parsing-guide.md for type mapping:

• Matrix, Likert, slider, numeric, constant sum, rank order, best-worst scaling
• Text entry (single-line, multi-line, essay)
• Multiple choice (single answer, multiple answer)

1. Detect design elements:

• Factorial conditions from randomizer blocks and embedded data
• Attention checks and comprehension checks
• Known validated scales (read references/known-scales-registry.md)
• Skip/display logic and branching

1. Produce structured summary — questions, conditions, scales, logic flow, warnings
2. Flag issues — missing attention checks, unbalanced conditions, potential order effects

Entry Point B: Natural Language Construction

When user describes an experiment in natural language:

1. Parse the design — extract factorial structure from text

• Example: "3 (Source: AI vs Human vs None) x 2 (Product: Hedonic vs Utilitarian)" → 3x2 between-subjects

1. Interview for details:

• What DVs to measure? (recommend appropriate scale types)
• What manipulation checks?
• What attention/comprehension checks?
• Demographics and control variables?

1. Generate survey specification — structured document with:

• Survey flow (consent → demographics → manipulation → DVs → manipulation check → debrief)
• Question text and response options for each item
• Condition assignments and randomization logic
• Attention check placement (read references/survey-design-checklist.md)

1. Scale recommendations — read references/known-scales-registry.md for validated scales matching the constructs

Survey Quality Checks

Read references/survey-design-checklist.md for the full checklist. Key checks:

• Attention checks: At least 1 per 5 minutes of survey length. Calibrated pass rates (85-95%). See Krosnick (1991), Meade & Craig (2012).
• Response style mitigation: Flag scales vulnerable to acquiescence bias. Recommend reverse-coding for scales with 4+ items in the same direction.
• Scale quality: Semantic type detection (satisfaction, trust, intention, risk). Reverse-coded item flagging. Well-known scale recognition.
• Order effects: Randomize question order within blocks where appropriate. Flag potential priming from early questions.

---

Cross-References

| Resource | When read | |----------|-----------| | references/power-analysis-recipes.md | Power mode | | references/pap-template.md | PAP mode | | references/survey-design-checklist.md | Survey mode (quality checks) | | references/identification-strategies.md | Design mode (identification check) | | references/qsf-parsing-guide.md | Survey mode (QSF parsing) | | references/known-scales-registry.md | Survey mode (scale recognition) | | design-before-results rule | Design + PAP modes produce the locked design | | /data-analysis skill | Consumes the design document | | /causal-design skill | For observational identification (not experiments) | | /synthetic-data skill | For generating pilot data |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: flonat
• Source: flonat/claude-research
• License: MIT

Install and usage instructions live in the source repository linked above.