Lifesight Experiment Design

Lifesight Experiment Design

Incrementality is the ground truth of the whole platform — the difference between "this channel got credit" and "this channel caused growth." This spoke does two jobs: design a test that will answer a causal question, and read the results of one that ran. It's also where you recommend what's worth testing — usually the biggest, most expensive assumption the user is about to act on.

Prerequisites (router handles): workspace calibrated, profile loaded. Operate under lifesight-core; present under lifesight-rendering. Load both.

Designing a test

1. Pin the causal question. What decision hangs on it? "Is Linear TV worth its

spend?" "Will scaling TikTok actually add revenue or just shift it?" A test with no decision attached is wasted budget.

1. Pick the design (via ask_mia's experiment workflow):

• Geo holdout — withhold a channel in matched control markets to measure the

lift it's currently driving. The default for "is this incremental?"

• Geo scale-up — increase spend in treatment markets to probe further up the

response curve. For "should I spend more here?"

• Time / segment designs where geo isn't feasible.

1. Sanity-check feasibility before launching: is there enough spend/volume and

enough matched markets to detect a realistic effect? Name the minimum detectable lift and the test duration up front — an underpowered test wastes weeks and answers nothing.

Reading results

Pull results via ask_mia, then interpret in this order — significance gates everything:

1. Significance first. Below ~90%, the result is inconclusive — do not act on the

lift, no matter how big it looks. Say so plainly.

1. Then lift + direction. Note the holdout nuance explicitly: in a holdout, a

negative lift means the marketing was effective (withholding it dropped the metric). Don't misread that as "the channel failed."

1. Then efficiency. Incremental ROAS / CPA from the experiment is the causal ground

truth — it outranks platform-reported numbers and even the MMM estimate.

1. Power context. A null result on an underpowered test is "we couldn't tell,"

not "no effect." Compare the observed lift to the minimum detectable lift before concluding anything.

Judgment checks (mandatory)

• No significance, no conclusion. The most common error is acting on an

insignificant lift. Hold the line even under "but the number's big" pressure.

• Holdout sign convention — negative lift = effective. State it so it can't be misread.
• Experiment > model > platform. When they disagree, the clean experiment wins;

use it to recalibrate, not to rationalize.

• Tests take time and assume no contamination. Don't imply instant answers; flag

spillover risk (media bleeding into control markets, un-geo-targetable national buys).

Output shape

• Design: the test plan — markets/split, duration, the spend change, what it will

detect (minimum detectable lift), and the decision it will settle.

• Read: significance → lift (with sign explained) → incremental ROAS/CPA → verdict

→ recommended action (scale, cut, recalibrate the model, or re-test with more power).

Next steps to offer

"Recalibrate the budget with this result" (→ budget-optimization) · "Deep-dive the channel we tested" (→ channel-deep-dive) · "Explain how geo-lift works" (→ measurement-coach) · "Design the follow-up test".

Red flags — STOP

• Acting on a lift below ~90% significance → inconclusive, say so
• Reading a holdout's negative lift as failure → it means effective
• Calling an underpowered null "no effect" → it's "couldn't detect"
• Implying an experiment gives an instant answer → name the duration
• Letting platform/MMM numbers override a clean experiment

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: lifesight
• Source: lifesight/lifesight
• License: MIT
• Homepage: https://lifesight.io/mcp

Install and usage instructions live in the source repository linked above.