Rct Power Calculations

Power analysis determines the sample size required to detect treatment effects of a given magnitude with specified probability. Adequate power prevents Type II errors (failing to detect real effects). Power calculations must account for clustering (design effects), non-compliance (reduces effective sample), attrition (reduces final sample), and multiple testing (inflates α). Underpowered studies waste resources and risk being misinterpreted as evidence of no effect.

Power = 80% is conventional. Higher power (90%) is better when study is expensive or one-shot.

Conduct Power Analysis Before Data Collection Calculate required sample size BEFORE starting enrollment or randomization Post-hoc power calculations are meaningless. Power must guide design, not rationalize results (Hoenig & Heisey 2001) Underpowered studies waste resources, fail to detect real effects, mislead policy with false nulls

Account for Clustering Design Effects When using cluster randomization, multiply base sample size by design effect: DE = 1 + (m-1) × ICC Clustering reduces effective sample size due to within-cluster correlation. Ignoring this causes severe underpowering (Donner & Klar 2000) Studies underpowered by 2-4x, inability to detect real effects despite large nominal sample

Inflate for Attrition and Non-Compliance Adjust sample size for expected attrition and imperfect compliance before finalizing design Attrition reduces final sample. Non-compliance attenuates ITT effects. Both reduce power substantially Final analysis underpowered even if initial enrollment meets naive power target

Use Realistic Effect Sizes Base MDE on pilot data, similar interventions, or smallest policy-relevant effect, not on wishful thinking Overoptimistic effect size assumptions guarantee underpowered study. True effects in social science typically 0.1-0.3 SD (Vivalt 2020) Study designed to detect implausibly large effects, fails to detect realistic effects, wasted resources

Adjust for Multiple Comparisons When testing multiple outcomes, increase sample size or reduce α to maintain family-wise error rate Multiple testing inflates Type I error. With 5 tests at α=0.05, false positive rate is 26% not 5% Study powered for single test but not for actual analysis plan, false discoveries

Assumed MDE or effect size is realistic based on prior evidence or theory Review similar interventions, conduct pilot, consult domain experts, use smallest policy-relevant effect Study either grossly overpowered (wasteful) or underpowered (fails to detect real effects)

Assumed intra-cluster correlation reflects true within-cluster similarity Use baseline data from same clusters, similar studies, or conservative upper bound (ICC=0.10-0.20) Design effect wrong → sample size wrong → power incorrect

Expected attrition and take-up rates match what actually occurs Base on pilot data, similar studies in same context, plan retention strategies Actual power lower than designed power, inability to detect effects

When conducting power analysis:

1. Specify primary outcome and effect size (MDE or expected effect)
2. Choose significance level (α=0.05 typical) and desired power (0.80-0.90)
3. Calculate base sample size for simple design
4. Apply design effect if cluster randomization (multiply by 1+(m-1)×ICC)
5. Inflate for expected non-compliance (divide by compliance²)
6. Inflate for expected attrition (divide by (1-attrition_rate))
7. Adjust for multiple testing if applicable
8. Check feasibility; if infeasible, reduce scope or accept larger MDE

@app.cell
def comprehensive_power_calculation():
    # Full power calculation with all real-world adjustments
    # Demonstrates compounding inflation factors for RCT sample size

    import numpy as np
    from scipy.stats import norm

    # Parameters
    alpha = 0.05
    power = 0.80
    mde = 0.25  # Effect size in SD units (realistic for social programs)
    p = 0.50  # Treatment allocation

    # Base calculation
    z_alpha = norm.ppf(1 - alpha/2)
    z_power = norm.ppf(power)
    n_base = ((z_alpha + z_power)**2) / (p * (1-p) * mde**2)

    print(f"COMPREHENSIVE POWER ANALYSIS")
    print(f"=" * 60)
    print(f"Parameters: α={alpha}, power={power}, MDE={mde} SD")
    print(f"\n1. Base sample size: {int(np.ceil(n_base))}")

    # Adjustment 1: Clustering
    icc = 0.05
    cluster_size = 30
    design_effect = 1 + (cluster_size - 1) * icc
    n_after_clustering = n_base * design_effect

    print(f"\n2. Clustering adjustment:")
    print(f"   ICC={icc}, avg cluster size={cluster_size}")
    print(f"   Design effect: {design_effect:.2f}")
    print(f"   After clustering: {int(np.ceil(n_after_clustering))}")

    # Adjustment 2: Non-compliance
    compliance_rate = 0.70
    n_after_compliance = n_after_clustering / (compliance_rate**2)

    print(f"\n3. Non-compliance adjustment:")
    print(f"   Expected compliance: {compliance_rate:.0%}")
    print(f"   Inflation factor: {1/compliance_rate**2:.2f}")
    print(f"   After compliance: {int(np.ceil(n_after_compliance))}")

    # Adjustment 3: Attrition
    attrition_rate = 0.20
    n_final = n_after_compliance / (1 - attrition_rate)

    print(f"\n4. Attrition adjustment:")
    print(f"   Expected attrition: {attrition_rate:.0%}")
    print(f"   Inflation factor: {1/(1-attrition_rate):.2f}")
    print(f"   FINAL REQUIRED N: {int(np.ceil(n_final))}")

    print(f"\n" + "=" * 60)
    print(f"Total inflation: {n_final/n_base:.2f}x base sample size")
    print(f"Baseline N needed: {int(np.ceil(n_final))}")
    print(f"Endline N expected: {int(np.ceil(n_final * (1-attrition_rate)))}")

    return int(np.ceil(n_final)),

Calculate power for cluster-randomized trial with design effect

@app.cell
def cluster_rct_power():
    # Cluster RCT power accounting for design effect
    # Shows how clustering can reduce power by 2-4x

    import numpy as np
    from scipy.stats import norm

    alpha = 0.05
    power = 0.80
    mde = 0.30

    # Individual randomization (baseline)
    z_a = norm.ppf(1 - alpha/2)
    z_p = norm.ppf(power)
    n_individual = ((z_a + z_p)**2) / (0.25 * mde**2)

    print(f"CLUSTER vs. INDIVIDUAL RANDOMIZATION")
    print(f"=" * 60)
    print(f"Individual randomization: {int(np.ceil(n_individual))} participants")

    # Cluster randomization scenarios
    scenarios = [
        {"icc": 0.05, "cluster_size": 20},
        {"icc": 0.05, "cluster_size": 50},
        {"icc": 0.10, "cluster_size": 30},
        {"icc": 0.20, "cluster_size": 30},
    ]

    for scenario in scenarios:
        icc = scenario["icc"]
        m = scenario["cluster_size"]
        de = 1 + (m - 1) * icc
        n_cluster = n_individual * de
        n_clusters_needed = int(np.ceil(n_cluster / m))

        print(f"\nICC={icc}, cluster size={m}:")
        print(f"  Design effect: {de:.2f}")
        print(f"  Total N needed: {int(np.ceil(n_cluster))}")
        print(f"  Clusters needed: {n_clusters_needed}")
        print(f"  Inflation: {de:.2f}x individual design")

    print("\nKey lesson: Higher ICC or larger clusters → much larger N needed")

    return ()

Design effect = 1 + (m-1) × ICC. With ICC=0.10 and m=30, DE=3.9, meaning you need 4x the individual-randomized sample size. Cluster randomization trades power for validity when spillovers are a concern.

Conducting power analysis after data collection (post-hoc power) Meaningless exercise, doesn't inform anything, misinterpreted as evidence quality Power analysis is for design phase only. Never calculate "observed power" after study

Ignoring clustering design effect in sample size calculation Study underpowered by 2-4x, fails to detect real effects despite large nominal sample ALWAYS multiply by design effect DE = 1+(m-1)×ICC for cluster randomization

Using overoptimistic effect size assumptions Study designed to detect implausibly large effects, fails for realistic effects Use conservative estimates from pilots, literature, or smallest policy-relevant effect

Not inflating for attrition and non-compliance Final analysis underpowered even if enrollment meets naive target Always inflate for expected attrition (÷ (1-attrition_rate)) and compliance (÷ compliance²)

Treating power as binary threshold (80% good, 79% bad) Arbitrary decisions, missing that power is continuous and context-dependent Power is a continuum. Consider costs, effect importance, and Type I vs. II error tradeoffs

For 80% power, α=0.05, 50/50 allocation:

• MDE = 0.10 SD: ~3,140 total
• MDE = 0.20 SD: ~786 total
• MDE = 0.30 SD: ~350 total
• MDE = 0.40 SD: ~198 total
• MDE = 0.50 SD: ~128 total

Then multiply by:

• Design effect if clustering: 1+(m-1)×ICC
• 1/compliance² for non-compliance
• 1/(1-attrition_rate) for attrition

If power calculation shows infeasible N:

• Increase MDE (accept detecting only larger effects)
• Reduce scope (fewer outcomes, simpler design)
• Improve efficiency (stratification, ANCOVA with baseline)
• Pool with other studies (consortium approach)
• Use alternative design (regression discontinuity, DID)
• Accept higher Type II error risk (document explicitly)

Do NOT: proceed with underpowered study and hope for best.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Erlbaum. Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research. Handbook of Development Economics, 4, 3895-3962. Donner, A., & Klar, N. (2000). Design and Analysis of Cluster Randomization Trials in Health Research. Arnold Publishers. Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician, 55(1), 19-24. Vivalt, E. (2020). How much can we generalize from impact evaluations? Journal of the European Economic Association, 18(6), 3045-3089.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sshtomar
• Source: sshtomar/claude-code-skills-social-science
• License: MIT

Install and usage instructions live in the source repository linked above.