Hypothesis Testing

Hypothesis Testing

Rigorous first-pass evaluation of a user hypothesis against a dataset.

Inputs

• Path to a dataset file or folder.
• The hypothesis, stated in natural language (e.g. "customers in segment A spend more than those in segment B", "there's no difference in conversion by landing page").
• Optional: desired significance level α (default 0.05).

Step 1 — Formalise the hypothesis

Before testing, restate the user's claim as:

• H₀ (null): typically "no effect" / "no difference" / "no relationship".
• H₁ (alternative): the user's claim, one- or two-sided as appropriate.
• Variables involved: identify which columns map to the hypothesis, with their dtypes.
• Test type selected based on the data shape (see decision table below).

Show this formalisation to the user and proceed unless they object.

Step 2 — Select a test

| User hypothesis shape | Variables | Default test | |---|---|---| | Group A mean ≠ Group B mean | 1 numeric + 1 binary categorical | Welch's t-test (unequal var) or Mann-Whitney if non-normal | | Difference across 3+ groups | 1 numeric + 1 categorical | One-way ANOVA or Kruskal-Wallis | | Association between two categoricals | 2 categoricals | Chi-square (or Fisher's exact if small cells) | | Correlation ≠ 0 | 2 numerics | Pearson (linear) or Spearman (monotonic) | | Proportion differs from value | 1 binary | One-sample proportion z-test | | Paired before/after | 1 numeric, paired | Paired t-test or Wilcoxon signed-rank | | Time trend | 1 numeric + time | Mann-Kendall or regression slope test |

Check assumptions (normality via Shapiro-Wilk for small n or visual for large n; equal variance via Levene's) and fall back to the non-parametric counterpart when they fail.

Step 3 — Execute

Recommended: uv run --with pandas --with scipy --with statsmodels python -c '...'.

Report:

• Sample size(s)
• Effect size (Cohen's d, r, Cramér's V, odds ratio — whichever fits)
• Test statistic and p-value
• 95% confidence interval for the effect

Step 4 — Verdict

Classify as one of:

• Supports — p 0.05" with "no effect" — distinguish "evidence of no effect" (tight CI around zero) from "no evidence of effect" (wide CI).

Output

Write -hypothesis-.md:

1. The hypothesis as stated by the user, verbatim.
2. Formalised H₀ / H₁.
3. Test chosen and why (including assumption checks).
4. Results table.
5. Verdict in bold: supports / refutes / inconclusive, with a plain-English explanation.
6. Caveats (confounders, selection effects, multiple-comparisons risk if this is one of many tests).

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: danielrosehill
• Source: danielrosehill/Claude-Data-Analyst-plugin
• License: MIT

Install and usage instructions live in the source repository linked above.