Experiment Orchestrator

Experiment Design Orchestrator

Core Principles

Experiments are the fastest way to learn

Every experiment is a controlled exploration. The goal is not to prove hypotheses correct, but to gain reliable learning as quickly as possible. The value of experiments lies in learning speed, not experiment count.

Orchestration Philosophy

1. Design -> Execution: both stages are essential: Execution without design is blind; the execution stage includes result analysis and report generation
2. Human review is a necessary gate for experiments: Both experiment plans and experiment reports must be reviewed by humans; the execution process can be automated
3. Guardrail metrics have veto power: No matter how positive the primary metric, guardrail metric breaches trigger immediate pause

Orchestration Protocol

The orchestration protocol follows the unified standard in [orchestrator-protocol.md](../../../codex-templates/orchestrator-protocol.md).

Pipeline

pipeline: experiment-orchestrator
version: 8.0
post_pipeline:
  - action: stage-summary
    output: output/phase-reports/pm-metrics-ops/experiment-orchestrator.md

stages:
  - id: phase-1
    name: "Experiment Design"
    depends_on: []
    skills: [experiment-design]
    gate:
      condition: "Experiment design reviewed and confirmed by human"
      fail_action: "Block experiment launch, modify and re-review"

  - id: phase-2
    name: "Experiment Execution"
    depends_on: [phase-1]
    skills: [experiment-execution]
    gate:
      condition: "Sample size sufficient and statistical testing complete, experiment report reviewed and confirmed by human"
      fail_action: "Extend experiment period or expand traffic"

Stage Execution Plan

Invoke experiment-design

Invoke: ${experiment-design}
Input:
  hypothesis: provided by user (hypothesis statement)
  available_traffic: provided by user (available traffic)
  metrics_system: metrics-system -> metrics.json (optional)
  historical_data: analysis-funnel/analysis-retention (optional)
Output: output/pm-metrics-ops/experiment-design/
Validation: Hypothesis structured (If-Then-Because-For); primary metric directly corresponds to hypothesis; guardrail metrics cover retention, revenue, and technical dimensions; sample size calculation parameters well-founded
Mode: AI->Human

Invoke experiment-execution

Invoke: ${experiment-execution}
Input:
  experiment_design: output/pm-metrics-ops/experiment-design/experiment_design.json
  experiment_data: provided by user
  termination_conditions: output/pm-metrics-ops/experiment-design/experiment_design.json
  product_background: provided by user (optional)
Output: output/pm-metrics-ops/experiment-execution/
Validation: Experiment group traffic allocation correct; guardrail metrics not triggered; experiment data collection complete; statistical significance calculated correctly; statistical conclusions consistent with data; action recommendations consistent with conclusions; guardrail metrics fully covered; heterogeneity effects analyzed (at least 3 segment dimensions)
Mode: AI->Human

Stage Summary (post_pipeline)

Follows the stage summary protocol in [orchestrator-protocol.md](../../../codex-templates/orchestrator-protocol.md).

| Parameter | Value | |------|-----| | Sub-Skill output path | output/pm-metrics-ops/ | | Summary output path | output/phase-reports/pm-metrics-ops/experiment-orchestrator.md |

Downstream connections: primary: decision-orchestrator (experiment complete, convert experiment conclusions into decision actions) alternatives:

• target: release-orchestrator

reason: Experiment results significant, recommend full rollout condition: Experiment results statistically significant (p Produces PRD Continue with AI-inferred values? (Inferred values confidence =0.5 | When confidence project_dir validity -> Input confidence assessment

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: LuckyOneTwoThree
• Source: LuckyOneTwoThree/vibe-skill
• License: MIT
• Homepage: https://luckyonetwothree.github.io/all-skill-html/

Install and usage instructions live in the source repository linked above.