Rct Data Analysis

RCT analysis leverages randomization to identify causal effects with minimal assumptions. Intention-to-treat (ITT) provides policy-relevant estimates of assignment effects. Local average treatment effects (LATE/IV) recover efficacy for compliers. Balance checks verify randomization, attrition analysis guards against selection bias, and heterogeneity analysis reveals for whom treatment works.

Analysis must preserve the integrity of randomization through proper inference and transparent reporting.

Report ITT as Primary Result Intention-to-treat analysis MUST be reported as the main result, even with non-compliance ITT maintains randomization, provides policy-relevant parameter (effect of offering treatment), and prevents selection bias from endogenous compliance (Angrist & Pischke 2009) Selection bias, loss of causal interpretation, inability to make policy recommendations

Cluster-Robust Standard Errors Use cluster-robust (or HC3-robust) standard errors, never classical SEs Clustered/robust SEs account for within-cluster correlation and heteroskedasticity. Classical SEs severely underestimate uncertainty (Bertrand et al. 2004) Type I error rates of 45% instead of 5%, false positives, invalid inference

Balance Table Before Outcomes Check and report covariate balance on baseline characteristics before analyzing outcomes Balance checks verify randomization worked. Large imbalances suggest implementation problems or sampling variation requiring controls Hidden confounding, invalid attribution of effects, inability to diagnose randomization failures

Address Attrition Explicitly Report attrition rates by treatment arm, test for differential attrition, bound effects if substantial Differential attrition creates selection bias that undermines randomization. Must be addressed or results are invalid (Lee 2009) Selection bias, invalid causal claims, inability to distinguish true effects from attrition-driven patterns

Adjust for Multiple Testing When Examining Multiple Outcomes Use Bonferroni, Holm, or FDR adjustment when testing multiple hypotheses Multiple testing inflates Type I error rate. Without adjustment, 5% significance means 26% false positive rate with 5 tests False discoveries, overstated evidence, publication of spurious results

Loss to follow-up is unrelated to treatment assignment and potential outcomes Compare attrition rates by treatment arm, test for correlation with baseline characteristics Use Lee bounds, IPW, or explicitly bound treatment effects under worst-case scenarios

Random assignment affects outcome only through treatment receipt (not directly) Conceptual argument, check for alternative mechanisms (Hawthorne effects, etc.) ITT is still valid and policy-relevant; LATE estimates are biased

Assignment doesn't flip treatment direction (no defiers) Check if always-takers exist in control group and never-takers in treatment LATE estimates are weighted average that may be misleading

When analyzing RCT data:

1. Check data quality and completeness
2. Verify balance on baseline covariates
3. Calculate and report attrition rates (overall and differential)
4. Estimate ITT (primary analysis)
5. Estimate LATE if non-compliance exists (secondary)
6. Test for heterogeneous effects (pre-specified subgroups)
7. Conduct robustness checks (alternative specifications)
8. Report all results transparently (including null findings)

@app.cell
def itt_analysis_rct(df):
    # Intention-to-treat analysis: Effect of assignment to treatment
    # This is the policy-relevant parameter even with imperfect compliance

    import statsmodels.formula.api as smf
    import pandas as pd

    # Verify data structure
    required = {'outcome', 'treatment', 'unit_id'}
    assert required.issubset(df.columns), f"Missing columns: {required - set(df.columns)}"

    # CRITICAL: Use cluster-robust or HC3-robust standard errors
    if 'cluster_id' in df.columns:
        # Cluster-robust (for cluster-randomized trials)
        model = smf.ols("outcome ~ treatment", data=df).fit(
            cov_type='cluster',
            cov_kwds={'groups': df['cluster_id']}
        )
        se_type = "Cluster-robust"
    else:
        # HC3-robust (for individual randomization)
        model = smf.ols("outcome ~ treatment", data=df).fit(cov_type='HC3')
        se_type = "HC3-robust"

    # Extract results
    itt_effect = model.params['treatment']
    itt_se = model.bse['treatment']
    itt_pval = model.pvalues['treatment']
    ci_low, ci_high = model.conf_int().loc['treatment']

    # Context for interpretation
    control_mean = df[df['treatment'] == 0]['outcome'].mean()
    control_sd = df[df['treatment'] == 0]['outcome'].std()
    effect_size = itt_effect / control_sd

    print("INTENTION-TO-TREAT (ITT) ANALYSIS")
    print("=" * 60)
    print(f"Outcome: {df.columns[df.columns.get_loc('outcome')]}")
    print(f"N: {len(df)} ({df['treatment'].sum()} treatment, {(~df['treatment'].astype(bool)).sum()} control)")
    print(f"Standard errors: {se_type}")
    print(f"\nITT Effect: {itt_effect:.4f}")
    print(f"Std. Error: {itt_se:.4f}")
    print(f"95% CI: [{ci_low:.4f}, {ci_high:.4f}]")
    print(f"p-value: {itt_pval:.4f}")
    print(f"\nControl mean: {control_mean:.4f}")
    print(f"Effect size: {effect_size:.3f} SD")
    print(f"Relative effect: {100*itt_effect/control_mean:.1f}%")

    return model,

Verify randomization balance before analyzing outcomes

@app.cell
def check_balance(df):
    # Balance checks verify randomization succeeded
    # Large imbalances may require controls or suggest implementation issues

    import pandas as pd
    import numpy as np
    from scipy import stats

    # Baseline covariates (measured before randomization)
    baseline_vars = ['age', 'education', 'income', 'baseline_outcome']

    results = []
    for var in baseline_vars:
        # Means by treatment group
        control_mean = df[df['treatment'] == 0][var].mean()
        treat_mean = df[df['treatment'] == 1][var].mean()

        # Normalized difference (Imbens & Rubin 2015)
        control_var = df[df['treatment'] == 0][var].var()
        treat_var = df[df['treatment'] == 1][var].var()
        norm_diff = (treat_mean - control_mean) / np.sqrt((control_var + treat_var) / 2)

        # T-test
        t_stat, p_val = stats.ttest_ind(
            df[df['treatment'] == 0][var].dropna(),
            df[df['treatment'] == 1][var].dropna()
        )

        results.append({
            'Variable': var,
            'Control': f"{control_mean:.3f}",
            'Treatment': f"{treat_mean:.3f}",
            'Norm Diff': f"{norm_diff:.3f}",
            'p-value': f"{p_val:.3f}"
        })

    balance_table = pd.DataFrame(results)

    print("BALANCE TABLE")
    print("=" * 60)
    print(balance_table.to_string(index=False))

    # Joint F-test
    from statsmodels.formula.api import ols
    formula = "treatment ~ " + " + ".join(baseline_vars)
    joint_model = ols(formula, data=df).fit()

    print(f"\nJoint F-test: F={joint_model.fvalue:.3f}, p={joint_model.f_pvalue:.4f}")

    # Flag concerns
    large_diffs = sum(abs(float(r['Norm Diff'])) > 0.25 for r in results)
    if large_diffs > 0:
        print(f"\nNote: {large_diffs} variables have |norm diff| > 0.25")
        print("Consider including these as controls in regression")

    return balance_table,

Balance checks use baseline (pre-randomization) covariates only. Large imbalances (|norm diff| > 0.25) don't invalidate randomization but suggest including those variables as controls to improve precision. Never use post-treatment variables for balance checks.

Not using cluster-robust or heteroskedasticity-robust standard errors Type I error rates can be 45% instead of 5%, leading to massive false positive rates ALWAYS use covtype='cluster' or covtype='HC3'. Never use default SEs.

Not reporting ITT when non-compliance exists Selection bias from analyzing compliers only, loss of causal interpretation ALWAYS report ITT as primary result. LATE/TOT are secondary sensitivity analyses.

Ignoring attrition or not testing for differential attrition Selection bias can completely invalidate results if attrition is differential Report attrition by arm, test differential attrition, use Lee bounds if substantial

P-hacking through subgroups without adjustment Finding "significant" effects that are just Type I errors from multiple testing Pre-specify subgroups in PAP, use Bonferroni/Holm adjustment, report all tests

Using baseline covariates that were measured post-randomization Conditioning on post-treatment variables creates bias (bad controls) Only use covariates measured before randomization in balance checks and controls

Minimum reporting for RCT results:

• Sample size (by treatment arm)
• Balance table on baseline characteristics
• Attrition rates (overall and by arm)
• ITT estimates with cluster-robust/HC3 SEs
• 95% confidence intervals
• Control group mean (for context)
• Effect size in SD units
• P-values (exact, not

Results are questionable if:

• No balance table provided
• Classical (non-robust) standard errors used
• ITT not reported for study with non-compliance
• Attrition >20% without bounding analysis
• Multiple outcomes tested without adjustment
• Post-treatment covariates included as controls
• Results only reported for "compliers" without ITT

Angrist, J. D., & Pischke, J. S. (2009). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press. Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249-275. Lee, D. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies, 76(3), 1071-1102. Imbens, G. W., & Rubin, D. B. (2015). Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge University Press.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sshtomar
• Source: sshtomar/claude-code-skills-social-science
• License: MIT

Install and usage instructions live in the source repository linked above.