Psy Ana Designer

Analysis Designer

Version

v1.3 — stable, 2026-06-10. Sub-skill of [amazing-psycoder](../SKILL.md).

Purpose

Transform experimental data and scientific questions into a complete analysis plan. Confirm step-by-step, progressively filling in the analysis config YAML (single source of truth), with every decision's origin recorded. Proactively compare methods, weighing pros and cons with reasoning — the user decides.

Core belief: Statistical analysis is not about applying templates — it is about selecting the optimal method for the scientific question.

When Not to Use

• Generating code: psy-ana-coder
• Auditing code: psy-ana-reviewer
• Designing experiments: psy-exp-designer

Design Philosophy

• Scientific question driven — Understand the question first, then choose the tool
• Data determines method — Design type, variables, and distribution characteristics determine the analysis approach
• Progressive confirmation — Fill specific config sections at each phase; Gates cannot be skipped; default items marked [ASSUMED], reviewed at Gate 5
• Traceable decisions — Every item annotated with its source: 用户确认 / 范式惯例 / 通用默认 / 自动推断
• Comparative recommendation — Method A vs B, 12-dimension comparison, with reasoning; user chooses
• Ask before selecting — Confirm fundamental data characteristics before recommending methods

Red Lines

| # | Rule | |---|------| | 1 | Do not recommend analysis methods until the scientific question is understood | | 2 | Do not finalize method selection until fundamental data characteristics are confirmed | | 3 | Every method selection must include a 12-dimension comparison rationale | | 4 | All data exclusion rules must be confirmed by the user | | 5 | Never recommend a method unsuitable for the data structure |

Analysis Config as Single Source of Truth

version: "1.0"     # Config schema version, used for coder compatibility validation

experiment:       # Reference to experiment config + data file organization
  config_path:
  data_path:       # Data file directory
  file_pattern:    # File naming pattern, e.g. "sub-{subject_id}_stroop.csv"
  file_format:     # csv / tsv / txt / xlsx
  multi_file:      # true (one file per subject) / false (all subjects in one file)
  n_subjects:      # Total number of subjects

design:           # Extracted from experiment config
  ivs:            # Independent variables (name, type, number of levels)
  dvs:            # Dependent variables (name, type, unit)
  design_type:    # within / between / mixed

questions:        # Scientific questions and analysis methods for each DV
  - id: Q1
    question:     # Scientific question (natural language)
    dv:           # Dependent variable
    method_a:     # Method A (user's inclination / common practice)
    method_b:     # Method B (recommended / superior)
    rationale:    # Selection rationale
    user_choice:  # User's final decision
    model_formula:# R model formula

cleaning:         # Data cleaning rules
  rt_lower:       # RT lower bound (ms)
  rt_upper:       # RT upper bound (ms)
  accuracy_min:   # Minimum accuracy
  trial_exclusion:# Trial exclusion rule (SD multiplier)
  missing_policy: # listwise / mice

model:            # Statistical model global settings
  seed:           # Random seed
  contrast:       # Contrast scheme (treatment / sum / helmert) — controls how factor levels are coded, affecting main effect interpretation
  correction:     # Multiple comparison correction (Bonferroni / FDR / Tukey / none) — controls p-value adjustment method

output:           # Output settings
  save_path:
  report_format:  # RMarkdown / Quarto
  figures:        # Figure list
  effect_sizes:   # Cohen's d / η²p

The sole goal of the conversation: fill every item in this config. Each phase fills its corresponding section.

Question Protocol

Each phase follows:

1. Show current state — List confirmed and unconfirmed items (decision checklist format, do not display YAML)
2. At most 3 questions per round — Prioritize questions that unlock the most downstream decisions. Complex phases (Phase 3/4) may be split into multiple rounds; keep each round focused
3. Write answers into config — Update immediately, show changes
4. Output phase decision checklist — Table format, each item annotated with source
5. Advance after user confirmation — "以上确认无误，进入下一阶段？"
6. Mark default items — Inferred and default values marked ⚠️, uniformly reviewed at Gate 5
7. Gates cannot be skipped — If the user gives a perfunctory answer (e.g. "just pick A because it's simple"), follow up with "确定不比较12维度？选择A的后果是..." for at least one more round

---

Phase 1: Understand Experiment and Scientific Questions

Fill config: design · experiment.config_path

Goal: Understand what the experiment did and what questions it aims to answer.

1.1 Obtain Experiment Config

First confirm the location of the experiment config YAML: > "请提供实验的 config YAML 文件路径。这个文件通常是 psy-exp-designer 生成的。如果没有，也可以直接描述你的实验设计。"

When a config file exists: Auto-extract after reading — do not ask:

• IVs (name, type, number of levels)
• DVs (name, type: rt/acc/score, unit)
• Design type (within/between/mixed)
• Paradigm name

When no config file: Collect core information via 3 questions: > 1. "你的实验有几个自变量（IV）？每个叫什么名字、几个水平？" > 2. "你的因变量（DV）是什么？测量什么？（RT/准确率/分数）" > 3. "实验设计是被试内（同一批人做所有条件）还是被试间（不同组做不同条件）？"

After collection, fill the design section, annotating source as 用户确认.

# Auto-filled
design:
  ivs: [{name: condition, type: categorical, levels: [congruent, incongruent]}]
  dvs: [{name: rt, type: continuous, unit: ms}, {name: acc, type: binary}]
  design_type: within

1.2 Follow up on Scientific Questions

> "你的核心研究问题是什么？不是'我要用什么方法'，是'我要回答什么科学问题'。"

If the user says "I want to do ANOVA," follow up: "ANOVA 是方法。你具体想检验什么？"

Guiding example: "Stroop 一致 vs 不一致条件 RT 是否有显著差异？"

If the experiment config has multiple DVs, confirm the corresponding scientific question for each one.

1.3 DV → Question Mapping

| DV | Scientific Question | Expected Direction | |----|---------------------|--------------------| | rt | 一致 vs 不一致 RT 差异？ | 不一致更慢 | | acc | 条件间错误率差异？ | 探索性 |

Gate 1: Every DV corresponds to at least one clear scientific question. Questions are recorded in natural language in config questions[].question.

Phase 1 Decision Checklist (example):

| # | Decision Item | Value | Source | |---|---------------|-------|--------| | 1 | Experiment paradigm | Stroop | 自动推断 | | 2 | Design type | Within-subject | 自动推断 | | 3 | DV: rt | Continuous, ms | 自动推断 | | 4 | DV: acc | Binary | 自动推断 | | 5 | Scientific question Q1 | 一致 vs 不一致 RT 差异 | 用户确认 | | 6 | Scientific question Q2 | 条件间准确率差异 | 用户确认 |

---

Phase 2: Understand Data Structure

Fill config: experiment.data_path · design (confirmation)

Goal: Confirm what the data looks like — do not assume. This is the critical foundation for code generation — data structure determines all downstream data import and cleaning logic.

2.1 Data File Organization (Must Confirm First)

> "你的实验数据是怎么存储的？"

Confirm the following step by step — order cannot be skipped:

2.1.1 File Hierarchy

> "有多少个被试？每个被试一个数据文件，还是所有被试在一个文件里？"

| Common Scenario | Example | Subsequent Handling | |----------------|---------|---------------------| | One file per subject | sub-001.csv, sub-002.csv … | Batch read + merge, or process per file | | All subjects in one file | all_data.csv | Group by subject_id column | | One file per subject per condition | sub-001_congruent.csv, sub-001_incongruent.csv | Need to understand filename encoding rules |

2.1.2 File Naming Rules

If one file per subject, must confirm naming rules to enable batch reading:

> "数据文件是怎么命名的？例如 sub-001_stroop.csv？被试编号在文件名什么位置？"

Example confirmations:

• Naming pattern: sub-{subject_id}_{task}.csv or P{number}_session{1/2}.txt?
• Subject ID length: fixed width (e.g. 001-099) or variable?
• Any group prefix: control/sub-001.csv vs experimental/sub-001.csv?

Record in config:

experiment:
  data_path: "data/"               # Data directory
  file_pattern: "sub-{subject_id}_stroop.csv"  # File naming pattern
  file_format: "csv"               # csv / tsv / txt / xlsx
  multi_file: true                 # Whether multi-file

2.1.3 File Format and Reading Method

> "数据文件是什么格式？"

| Format | Confirmation Items | R Read | Python Read | |--------|--------------------|--------|-------------| | CSV | Delimiter (, ; \t)? Has header? | readr::read_csv() | pd.read_csv() | | TSV/TXT | Delimiter (\t space)? Encoding? | readr::read_tsv() | pd.read_csv(sep='\t') | | Excel | Which sheet? Which row does data start? | readxl::read_excel() | pd.read_excel() | | Special format | e.g. Psychopy csv + log mixed | Need to confirm skip rows / extraction rules | Same as above |

If raw data files are available, ask the user to show the first few rows or str() output of at least two subjects' files to verify structural consistency.

2.2 Single-File Data Structure

Confirm the internal data structure of individual files:

2.2.1 Column Names and Variable Mapping

Confirm column names one by one — do not ask irrelevant questions:

> "确认以下列名：被试 ID 列名？条件列名？几个水平？RT 列名？单位 ms/s？正确性列？编码方式？"

For column names inferable from the experiment config, present them directly and only ask about what is uncertain.

> "除了这些核心列，数据文件里还有其他列吗？（如 trial 序号、block 编号、stimulus 文件名等）有的话也告诉我——后续分析可能需要用到。"

2.2.2 Data Structure Verification

Ask the user to show the first few rows of at least 1-2 subjects' files: > "方便展示一下某个被试数据文件的前几行吗？我确认一下结构。" > or "在 R 里跑一下 str(read_csv('sub-001.csv')) 给我看。"

Verify:

• Whether column names match the user's description
• Whether data types are correct (RT is numeric, not string; condition column is categorical)
• Whether the number of trials per subject per condition matches expectations
• Whether there are extra marker columns (e.g. practice=1 for practice trials)

2.3 Design Matrix

Confirm (infer from config as much as possible; only ask what cannot be inferred):

• Within/between? → Read from config
• Trials per condition? Trials per subject?
• Any missing data or dropouts?

Gate 2: Data file organization confirmed (file hierarchy + naming rules + format); variable mapping table confirmed; design type confirmed. If multi-file naming rules are not confirmed, do not proceed to Phase 3.

Phase 2 Decision Checklist (example):

| # | Decision Item | Value | Source | |---|---------------|-------|--------| | 1 | Data file organization | One file per subject, 30 subjects | 用户确认 | | 2 | File naming pattern | sub-{subjectid}stroop.csv | 用户确认 | | 3 | File format | CSV, comma-delimited, with header | 用户确认 | | 4 | subjectid column name | subjectid | 用户确认 | | 5 | Condition column + levels | condition, 2 levels (congruent/incongruent) | 用户确认 | | 6 | RT column + unit | rt, ms | 用户确认 | | 7 | Accuracy column + coding | acc, 1=correct/0=incorrect | 用户确认 | | 8 | Trials per subject per condition | ~60 | 用户确认 | | 9 | Design type | Within-subject | 自动推断 | | 10 | Additional columns | trial_id, block | 用户确认 |

---

Phase 3: Confirm Data Characteristics + Match Methods

Fill config: questions[].method_* · questions[].rationale · model.correction · model.seed

Goal: Confirm fundamental data characteristics first, then match analysis methods. This is the core phase.

3.1 Data Characteristics Confirmation (via Questions, Not Code-Dependent)

> "在确定方法之前，先确认几个关于数据的问题：" > > 1. "RT 分布大概什么样？严重右偏还是大致对称？各条件准确率有没有接近 100% 或 0%？" > 2. "不同被试之间波动大吗？有没有表现特别异常的（准确率很低/RT 特别快或慢）？" > 3. "有缺失试次吗？大概比例是多少？"

If the user is unsure: Have the user do a quick check and come back — > "在 R 里跑一下 summary(data) 和 hist(data$rt)，一分钟。告诉我分布大概什么样。"

Adjust subsequent method selection based on responses:

| User Feedback | Method Adjustment | |---------------|-------------------| | RT severely right-skewed | → Prefer Gamma GLMM or log(RT) for Option B | | Some condition accuracy ~95% | → ANOVA prohibited, must use logistic model | | Large between-subject variance | → Prefer mixed model for Option B | | ~3% missing, random | → Listwise deletion acceptable |

3.2 Method Matching

Auto-match candidates from experiment config design type + data characteristics from Phase 3.1. See [methods/](methods/) for detailed descriptions of each method.

Method Selection Decision Tree:

DV type?
  ├── Continuous (RT/Score)
  │     ├── Within-subject → 2 groups: paired t-test / lmer
  │     │                    3+ groups: repeated measures ANOVA / lmer
  │     │                    Non-normal: Wilcoxon / Friedman
  │     │                    Severely right-skewed: Gamma GLMM
  │     ├── Between-subject → 2 groups: Welch t-test
  │     │                     3+ groups: one-way ANOVA / Kruskal-Wallis
  │     └── Mixed design → Mixed ANOVA / lmer
  │
  ├── Binary (correct/incorrect)
  │     ├── Any condition accuracy >90% or  "Set a random seed for reproducibility. Any integer works — 20240610, 42, etc. ⚠️ Default: current date in YYYYMMDD format."

The seed is recorded in `model.seed` and injected into the generated code as `set.seed()` / `np.random.seed()`. This ensures the analysis produces identical results on every run.

**Gate 3**: Methods selected for every scientific question (with 12-dimension comparison rationale), multiple comparison scheme confirmed, seed set.

**Phase 3 Decision Checklist**: Each question's A vs B choice and rationale, multiple comparison scheme. All annotated with source.

---

## Phase 4: Analysis Details

**Fill config**: `cleaning` · `output` · `model.contrast`

**Goal**: Confirm cleaning standards, missing data handling, effect sizes, and figures.

> **About `model.contrast` (contrast scheme) vs `model.correction` (multiple comparison correction)**:  
> - **contrast** controls how factor levels are coded (treatment = compared to reference level, sum = compared to grand mean, helmert = compared to mean of previous levels), affecting coefficient interpretation in summary() output  
> - **correction** controls p-value adjustment method for post-hoc pairwise comparisons (Bonferroni/FDR/Tukey)  
> - Default: contrast = treatment ⚠️, correction = Bonferroni ⚠️ (unless specific needs arise)

### 4.1 Data Cleaning Standards

| Cleaning Item | Common Default | Why |
|---------------|----------------|-----|
| RT lower bound | 100-150ms ⚠️ | Below this, response cannot be genuine |
| RT upper bound | 2000-3000ms ⚠️ | Depends on task difficulty |
| Correct trials only | Analyze only correct RTs ⚠️ | RTs from incorrect trials are unreliable |
| Subject exclusion | acc 5% or patterned → multiple imputation (mice). Non-random → sensitivity analysis.

### 4.3 Effect Sizes + Figures

APA 7th defaults, marked ⚠️: t→Cohen's d+CI, ANOVA→η²p, mixed model→conditional differences+CI.

**Figure Selection Decision Tree**:

What do you want to show? ├── Distribution │ └── Within-subject two groups → Raincloud plot (preferred) / Individual connecting lines │ Multi-group comparison → Violin plot / Boxplot+scatter / Ridge plot (3+ groups) │ Univariate → Histogram / Density plot │ Normality check → QQ plot │ ├── Correlation │ └── Two continuous variables → Scatter+regression / Correlation ellipse / Marginal distribution │ Multi-variable (4+) → Correlation heatmap / Correlogram (3-8) │ Large sample >5000 → Hexbin │ ├── Comparison │ └── Between-subject multi-group → Bar chart / Lollipop chart / Cleveland dot plot │ Pre-post → Dumbbell chart / Slope chart / Individual connecting lines │ Multi-variable profile → Radar chart / Parallel coordinates │ Interaction → Interaction plot + significance annotation │ ├── Evolution │ └── Ti

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: soupandpsy
• Source: soupandpsy/amazing-psycoder-skills
• License: MIT

Install and usage instructions live in the source repository linked above.