Difference In Differences

Difference-in-Differences (DID) is a causal inference method that estimates treatment effects by comparing changes over time between treated and control groups. It leverages both cross-sectional and temporal variation to identify causal effects when randomization is infeasible.

DID is the workhorse of policy evaluation in economics and social sciences.

Parallel Trends Testing MUST test the parallel trends assumption using pre-treatment data Parallel trends is THE identifying assumption. Without it, DID estimates are biased. Violations have invalidated numerous published studies (Roth, 2022) Biased treatment effect estimates that could be entirely spurious

Cluster-Robust Standard Errors MUST cluster standard errors at the treatment level Serial correlation within units severely underestimates standard errors. Bertrand, Duflo & Mullainathan (2004) show 45% false positive rate with unclustered SE Type I error rates can exceed 45% instead of 5%

Event Study Specification MUST run event study with leads and lags when multiple pre-periods exist Event studies reveal dynamics, test pre-trends, and detect anticipation effects simultaneously (Sun & Abraham, 2021) Missing pre-trends violations, anticipation effects, or dynamic treatment effects

Clear Treatment Timing Document exactly when treatment starts and who is treated Ambiguous treatment timing or definition makes results unreplicable and potentially wrong Wrong attribution of effects to treatment

Treatment and control groups would follow parallel trajectories absent treatment Visual inspection of pre-trends, event study coefficients on leads, formal pre-trends test Consider synthetic control, changes-in-changes, or interactive fixed effects

Units don't change behavior before treatment actually occurs Check event study leads immediately before treatment Redefine treatment timing to when announcement occurred

No spillovers between treated and control units Test for spatial correlation, check economic linkages Exclude contaminated controls or use spatial methods

Treatment and control groups are comparable Compare covariate distributions and pre-treatment outcomes Consider matching or weighting before DID

When implementing DID:

1. Verify panel structure and treatment timing
2. Check pre-treatment balance between groups
3. Test parallel trends assumption (CRITICAL)
4. Implement main DID specification
5. Run event study for dynamics
6. Conduct robustness checks
7. Interpret magnitude and significance carefully

@app.cell
def did_with_diagnostics(df):
    # Implementing DID with all required diagnostics to ensure valid causal
    # inference. Following Angrist & Pischke (2009) and recent best practices
    # from Roth et al. (2023) on credible DID designs.

    import pandas as pd
    import numpy as np
    import statsmodels.formula.api as smf
    import matplotlib.pyplot as plt
    import os

    # MANDATORY: Validate panel structure
    required = {"outcome", "treat", "post", "unit_id", "time", "cluster_id"}
    assert required.issubset(df.columns), f"Missing columns: {required - set(df.columns)}"

    # Verify treatment is binary
    assert df["treat"].isin([0, 1]).all(), "Treatment must be binary (0/1)"

    # Check panel balance
    panel_check = df.groupby("unit_id")["time"].nunique()
    is_balanced = (panel_check == panel_check.iloc[0]).all()
    print(f"Panel structure: {'Balanced' if is_balanced else 'Unbalanced'}")

    # MANDATORY: Test parallel trends visually
    pre_data = df[df["post"] == 0].copy()
    trends = pre_data.groupby(["time", "treat"])["outcome"].mean().unstack()

    fig, axes = plt.subplots(1, 2, figsize=(15, 6))

    # Pre-trends plot
    axes[0].plot(trends.index, trends[0], marker='o', label='Control', linewidth=2)
    axes[0].plot(trends.index, trends[1], marker='s', label='Treatment', linewidth=2)
    axes[0].set_xlabel("Time")
    axes[0].set_ylabel("Outcome (mean)")
    axes[0].set_title("Pre-Treatment Trends (Parallel Trends Check)")
    axes[0].legend()
    axes[0].grid(True, alpha=0.3)

    # MANDATORY: Statistical test for parallel trends
    if len(pre_data["time"].unique()) >= 3:
        # Test for differential trends in pre-period
        pre_data["time_trend"] = pre_data.groupby("unit_id")["time"].transform(lambda x: x - x.min())
        pre_data["treat_trend"] = pre_data["treat"] * pre_data["time_trend"]

        trend_test = smf.ols(
            "outcome ~ treat + time_trend + treat_trend + C(unit_id) + C(time)",
            data=pre_data
        ).fit(cov_type="cluster", cov_kwds={"groups": pre_data["cluster_id"]})

        trend_coef = trend_test.params.get("treat_trend", 0)
        trend_pval = trend_test.pvalues.get("treat_trend", 1)

        print("\n" + "=" * 60)
        print("PARALLEL TRENDS TEST")
        print("=" * 60)
        print(f"Differential trend coefficient: {trend_coef:.4f}")
        print(f"P-value: {trend_pval:.4f}")
        if trend_pval 

Classic 2×2 DID with one treatment group and two periods

```python
@app.cell
def did_2x2_simple(df):
    # Implementing the canonical 2×2 DID where we have one treatment group,
    # one control group, and two time periods. This is the simplest and most
    # transparent DID design (Card & Krueger, 1994).

    import statsmodels.formula.api as smf
    import pandas as pd

    # Verify 2×2 structure
    assert df["treat"].nunique() == 2, "Need exactly 2 groups"
    assert df["time"].nunique() == 2, "Need exactly 2 time periods"
    assert df["post"].nunique() == 2, "Need pre and post indicators"

    # Calculate means for 2×2 table
    means = df.groupby(["treat", "post"])["outcome"].mean().unstack()

    print("2×2 DID Table")
    print("=" * 50)
    print(means)

    # Manual DID calculation for transparency
    diff_treat = means.loc[1, 1] - means.loc[1, 0]  # Treatment group change
    diff_control = means.loc[0, 1] - means.loc[0, 0]  # Control group change
    did_manual = diff_treat - diff_control

    print(f"\nManual DID Calculation:")
    print(f"  Treatment group change: {diff_treat:.4f}")
    print(f"  Control group change: {diff_control:.4f}")
    print(f"  DID estimate: {did_manual:.4f}")

    # Regression approach (should match manual calculation)
    df["did"] = df["treat"] * df["post"]
    model = smf.ols("outcome ~ treat + post + did", data=df).fit(
        cov_type="cluster",
        cov_kwds={"groups": df["cluster_id"]}
    )

    print(f"\nRegression DID estimate: {model.params['did']:.4f}")
    print(f"Clustered SE: {model.bse['did']:.4f}")
    print(f"P-value: {model.pvalues['did']:.4f}")

    # Interpretation
    baseline_mean = means.loc[1, 0]  # Pre-treatment mean for treated group
    pct_effect = (model.params['did'] / baseline_mean) * 100

    print(f"\nEffect size: {pct_effect:.1f}% relative to baseline")

    return model,

In 2×2 DID:

• The coefficient represents the average treatment effect on the treated (ATT)
• Compare manual and regression estimates to verify implementation
• Report both absolute and relative (%) effects for context

Staggered treatment adoption with heterogeneous effects

@app.cell
def did_staggered_robust(df):
    # Standard TWFE DID is biased with staggered adoption and heterogeneous
    # effects (Goodman-Bacon, 2021). We implement both standard TWFE and
    # robust alternatives (Callaway & Sant'Anna, 2021).

    import pandas as pd
    import numpy as np
    import statsmodels.formula.api as smf

    # Check for staggered adoption
    treatment_times = df[df["treat"] == 1].groupby("unit_id")["time"].min()
    n_cohorts = treatment_times.nunique()

    print("Staggered Adoption Structure")
    print("=" * 50)
    print(f"Number of treatment cohorts: {n_cohorts}")
    print("\nTreatment timing distribution:")
    print(treatment_times.value_counts().sort_index())

    if n_cohorts > 1:
        print("\nWARNING: Staggered adoption detected!")
        print("Standard TWFE may be biased. Showing both approaches:")

    # 1. Standard TWFE (potentially biased)
    print("\n1. STANDARD TWFE APPROACH")
    print("-" * 40)

    df["treated_post"] = (df["treat"] == 1) & (df["post"] == 1)
    twfe = smf.ols(
        "outcome ~ treated_post + C(unit_id) + C(time)",
        data=df
    ).fit(cov_type="cluster", cov_kwds={"groups": df["cluster_id"]})

    print(f"TWFE estimate: {twfe.params['treated_post[T.True]']:.4f}")
    print(f"SE: {twfe.bse['treated_post[T.True]']:.4f}")

    # 2. Cohort-specific effects (diagnostic)
    print("\n2. COHORT-SPECIFIC EFFECTS")
    print("-" * 40)

    cohort_effects = {}
    for cohort_time in treatment_times.unique():
        cohort_units = treatment_times[treatment_times == cohort_time].index

        # Get cohort data plus never-treated controls
        cohort_data = df[
            (df["unit_id"].isin(cohort_units)) |
            (df["treat"] == 0)
        ].copy()

        cohort_data["post_cohort"] = cohort_data["time"] >= cohort_time
        cohort_data["did_cohort"] = (
            cohort_data["unit_id"].isin(cohort_units) &
            cohort_data["post_cohort"]
        )

        cohort_model = smf.ols(
            "outcome ~ did_cohort + C(unit_id) + C(time)",
            data=cohort_data
        ).fit(cov_type="cluster", cov_kwds={"groups": cohort_data["cluster_id"]})

        effect = cohort_model.params.get("did_cohort[T.True]", np.nan)
        cohort_effects[cohort_time] = effect
        print(f"Cohort {cohort_time}: {effect:.4f}")

    # Check for heterogeneity
    effects_array = np.array(list(cohort_effects.values()))
    heterogeneity = effects_array.std()

    print(f"\nHeterogeneity check:")
    print(f"  SD of cohort effects: {heterogeneity:.4f}")
    if heterogeneity > abs(effects_array.mean()) * 0.5:
        print("  WARNING: Substantial heterogeneity detected!")
        print("  Consider using robust DID methods (CS, SA, or Sun-Abraham)")

    # 3. Simple aggregation (equally weighted)
    print("\n3. SIMPLE AVERAGE OF COHORT EFFECTS")
    print("-" * 40)
    simple_avg = effects_array.mean()
    print(f"Average effect: {simple_avg:.4f}")
    print(f"Difference from TWFE: {simple_avg - twfe.params['treated_post[T.True]']:.4f}")

    return twfe, cohort_effects,

With staggered adoption:

1. TWFE can be biased due to "bad comparisons"
2. Check cohort-specific effects for heterogeneity
3. Consider modern robust estimators (Callaway-Sant'Anna, Sun-Abraham)
4. Report both standard and robust estimates for transparency

Handling DID when parallel trends fails

@app.cell
def did_with_failed_pretrends(df):
    # When parallel trends fails, standard DID is biased. We demonstrate the
    # failure, then show alternative approaches: controlling for differential
    # trends, matching before DID, or synthetic control methods.

    import pandas as pd
    import numpy as np
    import statsmodels.formula.api as smf
    import matplotlib.pyplot as plt
    import os
    from sklearn.linear_model import LogisticRegression
    from sklearn.neighbors import NearestNeighbors

    # First, demonstrate the pre-trends failure
    pre_data = df[df["post"] == 0].copy()
    pre_data["time_numeric"] = pre_data.groupby("unit_id")["time"].transform(
        lambda x: x - x.min()
    )

    # Test for differential trends
    pre_data["treat_x_time"] = pre_data["treat"] * pre_data["time_numeric"]
    trend_test = smf.ols(
        "outcome ~ treat + time_numeric + treat_x_time + C(unit_id)",
        data=pre_data
    ).fit(cov_type="cluster", cov_kwds={"groups": pre_data["cluster_id"]})

    diff_trend = trend_test.params["treat_x_time"]
    p_value = trend_test.pvalues["treat_x_time"]

    print("PRE-TRENDS DIAGNOSTIC")
    print("=" * 60)
    print(f"Differential trend: {diff_trend:.4f} (p={p_value:.4f})")

    if p_value 

When parallel trends fails:
1. NEVER ignore it - standard DID will be biased
2. Try multiple approaches and check sensitivity
3. Be transparent about the failure in reporting
4. Consider that treatment might truly be endogenous
5. Sometimes the honest answer is "we cannot identify the causal effect"

  Not clustering standard errors
  Type I error rates can be 45% instead of 5% (Bertrand et al., 2004)
  ALWAYS cluster at treatment assignment level (usually unit level)

  Ignoring pre-trends test
  Entire estimate may be spurious if trends differ
  Always test and visualize pre-trends, report even if fails

  Using TWFE with heterogeneous effects and staggered adoption
  Estimate is weighted average including "bad comparisons" (negative weights)
  Use Callaway-Sant'Anna, Sun-Abraham, or other robust estimators

  Including bad controls (post-treatment variables)
  Bias from controlling for mechanisms
  Only control for pre-treatment characteristics

  Not checking for anticipation effects
  Underestimate true effect if behavior changes before official treatment
  Check event study leads just before treatment

- DID coefficient = Average Treatment Effect on the Treated (ATT)
- Units are same as outcome variable (be specific!)
- Relative effect = (coefficient / pre-treatment mean) × 100%
- Check both statistical and economic significance

- Pre-trend coefficients significantly different from zero
- Event study shows effects before treatment (anticipation)
- Very different estimates with different specifications
- Implausibly large effects (>50% changes are rare)

- All checks pass → Report main estimate with confidence
- Pre-trends fail → Try alternative methods or acknowledge limitation
- Heterogeneous effects → Report by subgroup
- Dynamic effects → Focus on event study rather than single coefficient

Angrist, J.D. & Pischke, J.S. (2009). "Mostly Harmless Econometrics." Princeton University Press. Canonical DID exposition.
Bertrand, M., Duflo, E. & Mullainathan, S. (2004). "How Much Should We Trust DID Estimates?" QJE. Serial correlation and clustering.
Roth, J. et al. (2023). "What's Trending in DID?" AER Insights. Modern best practices and pre-testing.
Goodman-Bacon, A. (2021). "DID with Variation in Treatment Timing." Journal of Econometrics. TWFE bias.
Callaway, B. & Sant'Anna, P.H.C. (2021). "DID with Multiple Time Periods." Journal of Econometrics. Robust estimation.
Sun, L. & Abraham, S. (2021). "Estimating Dynamic Treatment Effects." Journal of Econometrics. Event study bias correction.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [sshtomar](https://github.com/sshtomar)
- **Source:** [sshtomar/claude-code-skills-social-science](https://github.com/sshtomar/claude-code-skills-social-science)
- **License:** MIT

Install and usage instructions live in the source repository linked above.