Research Design

Research Design

Design experiments before running them. Bad design wastes months.

Workflow

Step 1 — Methodology Paradigm

Choose one and justify:

| Paradigm | When to use | |----------|-------------| | Simulation | Can't control real environment; FL, scheduling, network protocols | | Controlled experiment | Can manipulate variables; lab conditions | | Real-world dataset | Benchmark comparison; image/NLP tasks | | User study | Measuring human factors | | Theoretical analysis | Proving bounds, convergence, complexity | | Design Science Research (DSR) | Build + evaluate artifact in real context |

Step 2 — Variables

Independent variables (you manipulate):
  - [var1]: values [a, b, c]
  - [var2]: values [x, y]

Dependent variables (you measure):
  - [metric1]: [how measured]
  - [metric2]: [how measured]

Controlled variables (you fix):
  - [setting1]: value [X]
  - [setting2]: value [Y]

Confounding variables (potential threats):
  - [threat1]: [how mitigated]

Step 3 — Baselines Selection

Rule: choose baselines that represent each major paradigm in your SOTA taxonomy.

| Baseline | Paper | Why included | Code available? | |----------|-------|--------------|-----------------| | [Method A] | Author et al. (year) | Best published result on [metric] | ✅ github.com/... | | [Method B] | Author et al. (year) | Representative of [approach class] | ✅ | | [Naive/Oracle] | N/A | Lower/upper bound | trivial |

Avoid: cherry-picked weak baselines, baselines without code, outdated methods unless they're the canonical comparison.

Step 4 — Metrics

Primary metric (1): drives all main comparisons — must directly measure what your RQ claims to improve.

Secondary metrics (2-4): provide context — efficiency, robustness, scalability.

Metric formulas — write them explicitly:

Accuracy = (TP+TN) / (TP+TN+FP+FN)
Communication cost = Σ(rounds × message_size_MB)
Convergence speed = rounds_to_reach_target_accuracy

Statistical significance:

• Run ≥5 seeds, report mean ± std
• Use Wilcoxon signed-rank test (non-parametric) or paired t-test
• Report effect size (Cohen's d)

Step 5 — Ablation Study Design

Identify your method's key components. Test each one by removing/replacing it:

| Variant | Description | Expected effect | |---------|-------------|-----------------| | Full model | All components | Best | | w/o [component A] | Remove A | Shows A's contribution | | w/o [component B] | Remove B | Shows B's contribution | | [component A] replaced with [baseline] | Shows your design choice matters |

Step 6 — Dataset / Environment

dataset:
  name: [dataset name]
  source: [URL or paper]
  size: [N samples / N clients / N episodes]
  split: train/val/test = X%/Y%/Z%
  preprocessing: [steps]
  why_chosen: [relevance + availability + used by baselines]

hardware:
  GPU: [model, VRAM]
  CPU: [cores]
  RAM: [GB]
  estimated_runtime: [per experiment]

Step 7 — Reproducibility Checklist

• [ ] Fixed random seeds for all experiments
• [ ] All hyperparameters documented in one config file
• [ ] Dataset preprocessing steps scripted (not manual)
• [ ] All baselines run with same compute budget
• [ ] Environment: requirements.txt or environment.yml or Dockerfile
• [ ] Results logged to file (not just stdout)
• [ ] Code pushed to version control before experiments

Output Format

## Experiment Design: [Your Method Name]

**Paradigm:** [choice + 1-line justification]

**RQ being tested:** [paste from research-question]

### Variables
- Independent: ...
- Dependent: ...
- Controlled: ...

### Baselines
| Baseline | Source | Role |
...

### Metrics
- Primary: [metric] — measures [what]
- Secondary: [metric1, metric2]

### Ablation Plan
| Variant | Removes | Tests |
...

### Datasets/Environments
...

### Statistical Analysis
- N runs: [≥5]
- Significance test: [which test]
- Reporting: mean ± std

Links to Other Skills

• Requires → research-question (RQ defines what you measure)
• Requires → sota-survey (baselines come from here)
• Feeds into → experiment-tracking (run the protocol)
• Feeds into → paper-writing (Methodology section)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kinhluan
• Source: kinhluan/skills
• License: MIT

Install and usage instructions live in the source repository linked above.