Core Methodology

Core methodology establishes the fundamental principles of rigorous statistical analysis. These principles protect against errors that have invalidated countless published studies and ensure reproducibility. This skill is ALWAYS loaded because its requirements apply to every analysis.

These are not suggestions - they are mandatory safeguards developed from decades of statistical failures and corrections.

Defensive Input Validation MUST validate all inputs before any analysis, including file encoding detection for text files Silent failures on different datasets have led to retracted papers. Gelman & Loken (2014) document the "garden of forking paths" problem where unstated data decisions invalidate p-values. Encoding mismatches cause silent character corruption (e.g., accented characters become garbage), leading to incorrect data interpretation Analysis fails silently on new data, producing incorrect results that appear valid. Wrong encoding causes character corruption, missing data from parsing failures, and incorrect string comparisons

RATIONALE Comments Every statistical choice MUST have a RATIONALE comment explaining why Reproducibility requires understanding not just what was done but why. Nosek et al. (2015) found only 39% of psychology studies replicated, often due to undocumented choices Future researchers cannot understand or replicate the analysis

No Silent Data Transformations Every data modification must be explicit and explained Simmons et al. (2011) show how "researcher degrees of freedom" in data handling can produce false positives. Silent transformations hide these choices P-hacking and false discoveries from undisclosed analytical flexibility

Data Authenticity and Simulation Transparency MUST use real data from documented sources when analyzing actual datasets. Synthetic/simulated data is ONLY acceptable for: (1) power calculations, (2) pedagogical examples, (3) algorithm demonstrations. ALL simulated data MUST be explicitly labeled with clear warnings Fabricated data masquerading as real invalidates causal claims and erodes scientific credibility. Nosek et al. (2015) emphasize working with authentic data for reproducible science. Context matters: simulation for legitimate purposes (e.g., Monte Carlo studies, power analysis) is acceptable when transparent If fabrication detected in real analysis: invalid findings, undetectable p-hacking, publication of false results presented as discovered truth, erosion of trust in research

Reproducibility Information Document seeds, versions, and file hashes Computational reproducibility is fundamental to science. Ioannidis et al. (2009) found most computational results cannot be reproduced without full environment details Results cannot be verified or replicated

Data Provenance Documentation MUST document the complete lineage of all data: original sources, collection methods, preprocessing steps performed before analysis, data access dates, and any transformations applied outside the analysis code Data provenance is essential for reproducibility and validity assessment. Without knowing where data came from and what happened to it before analysis, results cannot be verified or understood. Stodden et al. (2016) emphasize that data provenance enables others to assess data quality, detect errors, and understand limitations. Missing provenance information makes it impossible to evaluate whether data collection methods introduce bias or whether preprocessing steps affect conclusions Inability to verify data authenticity, assess data quality, detect preprocessing errors, understand limitations, or replicate results. Hidden preprocessing steps can introduce bias or errors that invalidate findings

Explicit Assumption Statements State all statistical assumptions before analysis Violations of unstated assumptions are a primary source of invalid inference. Every statistical method has assumptions that must be checked Invalid statistical inference from violated assumptions

Teach through questions, not commands Ask "What would it mean if..." rather than stating conclusions Develops statistical intuition rather than rote application

Always discuss practical significance alongside statistical significance Report magnitudes with units and context P-values alone are meaningless; effect size determines practical importance

Always show uncertainty in estimates Confidence intervals, standard errors, prediction intervals Point estimates without uncertainty are misleading

For EVERY analysis:

1. Document data provenance (sources, collection, preprocessing)
2. Validate data structure and types
3. State assumptions explicitly
4. Check for data quality issues
5. Document all transformations
6. Choose appropriate methods
7. Implement with defensive checks
8. Include reproducibility info
9. Interpret with appropriate caution

@app.cell
def rigorous_analysis_pattern(dependencies):
    #This template demonstrates the mandatory structure for
    # all statistical analyses. Every element serves a specific purpose
    # in ensuring reproducibility and validity.

    import pandas as pd
    import numpy as np
    import sys
    import warnings
    from datetime import datetime

    # MANDATORY: Input validation
    assert 'df' in locals(), "DataFrame 'df' must be provided"
    required_columns = {'outcome', 'treatment', 'unit_id'}
    missing = required_columns - set(df.columns)
    assert not missing, f"Missing required columns: {missing}"

    # MANDATORY: Data type validation
    numeric_columns = ['outcome']
    for col in numeric_columns:
        assert pd.api.types.is_numeric_dtype(df[col]), f"{col} must be numeric"

    # MANDATORY: Document data provenance
    print("=" * 60)
    print("DATA PROVENANCE")
    print("=" * 60)
    print("Original data source: [Document where data came from]")
    print("Collection method: [How was data collected? Survey, administrative records, etc.]")
    print("Collection dates: [When was data collected?]")
    print("Data access date: [When did you access/download the data?]")
    print("Preprocessing steps (before this analysis):")
    print("  - [List any cleaning, merging, or transformations done outside this notebook]")
    print("  - [Document any data preparation steps]")
    print("Data version/identifier: [File hash, version number, or unique identifier]")
    
    # MANDATORY: Document data properties
    print("\n" + "=" * 60)
    print("DATA VALIDATION")
    print("=" * 60)
    print(f"N observations: {len(df):,}")
    print(f"N units: {df['unit_id'].nunique():,}")
    print(f"Missing outcome: {df['outcome'].isna().sum()} ({df['outcome'].isna().mean()*100:.1f}%)")

    # MANDATORY: Show any data transformations explicitly
    if df['outcome'].isna().any():
        print(f"\nWARNING: Dropping {df['outcome'].isna().sum()} observations with missing outcome")
        df_analysis = df.dropna(subset=['outcome']).copy()
    else:
        df_analysis = df.copy()

    # MANDATORY: State assumptions
    assumptions = """
    STATISTICAL ASSUMPTIONS:
    1. Independence: Observations are independent (may be violated if clustered)
    2. Correct specification: The model captures the true relationship
    3. No measurement error: Variables are measured accurately
    """
    print(assumptions)

    # [Actual analysis code here]

    # MANDATORY: Reproducibility information (always the last cell)
    import platform
    import json as _json
    _repro = {
        "python": sys.version,
        "platform": platform.platform(),
        "pandas": pd.__version__,
        "statsmodels": __import__('statsmodels').__version__,
        "numpy": np.__version__,
        "matplotlib": __import__('matplotlib').__version__,
    }
    print("Reproducibility Information")
    print("=" * 60)
    print(_json.dumps(_repro, indent=2))

    return df_analysis,

Detecting file encoding before reading CSV files

@app.cell
def detect_and_load_csv(filepath):
    #RATIONALE: Encoding mismatches cause silent character corruption.
    # We detect encoding first, document it for reproducibility, and
    # handle common encoding issues before reading data.

    import pandas as pd
    import os
    import chardet  # pip install chardet

    # MANDATORY: Check file exists
    assert os.path.exists(filepath), f"File not found: {filepath}"

    # MANDATORY: Detect encoding before reading
    print("=" * 60)
    print("ENCODING DETECTION")
    print("=" * 60)
    
    # Read a sample to detect encoding (faster than reading entire file)
    with open(filepath, 'rb') as f:
        raw_data = f.read(10000)  # Read first 10KB for detection
    
    detection_result = chardet.detect(raw_data)
    detected_encoding = detection_result['encoding']
    confidence = detection_result['confidence']
    
    print(f"Detected encoding: {detected_encoding}")
    print(f"Confidence: {confidence:.2%}")
    
    # Common encodings to try if detection is uncertain
    encodings_to_try = [
        detected_encoding,  # Try detected first
        'utf-8',           # Most common modern encoding
        'latin-1',         # Common for European data
        'iso-8859-1',      # Alternative to latin-1
        'cp1252',          # Windows encoding
        'utf-16',          # Unicode with BOM
    ]
    
    # Remove duplicates while preserving order
    encodings_to_try = list(dict.fromkeys(encodings_to_try))
    
    # Try each encoding until one works
    df = None
    used_encoding = None
    
    for encoding in encodings_to_try:
        if encoding is None:
            continue
        try:
            print(f"\nAttempting to read with encoding: {encoding}")
            df = pd.read_csv(filepath, encoding=encoding, nrows=5)  # Test with 5 rows first
            used_encoding = encoding
            print(f"SUCCESS: File readable with {encoding}")
            break
        except (UnicodeDecodeError, UnicodeError) as e:
            print(f"  Failed: {str(e)[:60]}...")
            continue
        except Exception as e:
            # Other errors (not encoding-related) - re-raise
            raise
    
    if df is None:
        raise ValueError(f"Could not read file with any encoding tried: {encodings_to_try}")
    
    # Now read full file with correct encoding
    print(f"\nReading full file with encoding: {used_encoding}")
    df = pd.read_csv(filepath, encoding=used_encoding)
    
    # MANDATORY: Document encoding for reproducibility
    print("\n" + "=" * 60)
    print("ENCODING INFORMATION (for reproducibility)")
    print("=" * 60)
    print(f"File encoding: {used_encoding}")
    print(f"Detection confidence: {confidence:.2%}")
    print(f"File size: {os.path.getsize(filepath):,} bytes")
    
    return df, used_encoding,

Encoding detection prevents silent character corruption. Common issues:

• UTF-8 files read as latin-1: accented characters become garbage
• Windows-1252 files read as UTF-8: parsing fails on special characters
• Mixed encodings: some rows readable, others fail silently

Always document the encoding used for reproducibility.

For CSV files:

1. Detect encoding before reading (use chardet or charset-normalizer)
2. Try common encodings if detection uncertain
3. Document encoding used in reproducibility section
4. If encoding detection unavailable, try: utf-8, latin-1, cp1252 in order
5. Check for encoding errors in string columns after loading

Defensive validation preventing silent failure

@app.cell
def validate_before_analysis(df):
    #This example shows how defensive validation catches
    # errors that would otherwise produce invalid results silently.
    # Based on real retractions where analyses ran on wrong data types.

    import pandas as pd
    import numpy as np

    # Scenario: User thinks 'treatment' is binary but it's actually continuous
    print("Checking treatment variable...")

    # MANDATORY: Validate assumptions about the data
    unique_vals = df['treatment'].unique()
    print(f"Unique treatment values: {unique_vals}")

    if not set(unique_vals).issubset({0, 1}):
        print("\nERROR: Treatment is not binary!")
        print(f"Found values: {unique_vals}")
        print("\nThis would have caused:")
        print("- Wrong model specification")
        print("- Invalid causal interpretation")
        print("- Meaningless 'treatment effect'")

        # Show what would happen with silent failure
        if len(unique_vals) > 2:
            # Demonstrate the error
            mean_by_treatment = df.groupby('treatment')['outcome'].mean()
            print(f"\nMeans by 'treatment': \n{mean_by_treatment}")
            print("\nWARNING: These are NOT treatment effects!")

        raise ValueError("Treatment must be binary (0/1) for causal analysis")

    print("SUCCESS: Treatment is binary")

    # Additional validation
    assert df['outcome'].dtype in ['float64', 'int64'], "Outcome must be numeric"
    assert not df[['treatment', 'outcome']].isna().all(axis=1).any(), "Complete observations required"

    return None,

This validation would catch a critical error where continuous dose is mistaken for binary treatment, preventing false causal claims.

Making all data transformations explicit

@app.cell
def transparent_transformations(df):
    #Every transformation is shown and justified to prevent
    # hidden researcher degrees of freedom. Simmons et al. (2011) show
    # how undisclosed flexibility in data analysis inflates false positive rates.

    import pandas as pd
    import numpy as np

    print("DATA TRANSFORMATION DOCUMENTATION")
    print("=" * 60)

    # Original data summary
    print(f"Original N: {len(df)}")
    print(f"Original outcome range: [{df['outcome'].min():.2f}, {df['outcome'].max():.2f}]")

    # MANDATORY: Document each transformation step
    df_transformed = df.copy()

    # Step 1: Handle outliers (document cutoff and rationale)
    Q1 = df['outcome'].quantile(0.25)
    Q3 = df['outcome'].quantile(0.75)
    IQR = Q3 - Q1
    outlier_threshold = Q3 + 3*IQR  # Using 3*IQR for extreme outliers only

    n_outliers = (df['outcome'] > outlier_threshold).sum()
    print(f"\nStep 1: Windsorizing {n_outliers} extreme outliers (>{outlier_threshold:.2f})")
    print(f"  Rationale: Values >3*IQR likely measurement errors")
    print(f"  Alternative: Could use log transformation instead")

    df_transformed.loc[df_transformed['outcome'] > outlier_threshold, 'outcome'] = outlier_threshold

    # Step 2: Create categorical variable (document cutoffs)
    median_val = df_transformed['outcome'].median()
    print(f"\nStep 2: Creating binary indicator at median ({median_val:.2f})")
    print(f"  Rationale: Testing for heterogeneous effects above/below median")
    print(f"  Note: This reduces power but aids interpretation")

    df_transformed['outcome_high'] = (df_transformed['outcome'] > median_val).astype(int)

    # Step 3: Log transformation (document reason)
    print(f"\nStep 3: Log transformation of outcome")
    print(f"  Skewness before: {df['outcome'].skew():.2f}")

    # Handle zeros if present
    min_positive = df_transformed[df_transformed['outcome'] > 0]['outcome'].min()
    if (df_transformed['outcome'] == 0).any():
        offset = min_positive / 2
        print(f"  Adding offset {offset:.4f} to handle zeros")
        df_transformed['log_outcome'] = np.log(df_transformed['outcome'] + offset)
    else:
        df_transformed['log_outcome'] = np.log(df_transformed['outcome'])

    print(f"  Skewness after: {df_transformed['log_outcome'].skew():.2f}")

    # MANDATORY: Summary of all changes
    print("\n" + "=" * 60)
    print("TRANSFORMATION SUMMARY")
    print("=" * 60)
    print(f"Records modified: {(df['outcome'] != df_transformed['outcome']).sum()}")
    print(f"New variables created: outcome_high, log_outcome")
    print(f"Original data preserved in: df")
    print(f"Tran

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [sshtomar](https://github.com/sshtomar)
- **Source:** [sshtomar/claude-code-skills-social-science](https://github.com/sshtomar/claude-code-skills-social-science)
- **License:** MIT

Install and usage instructions live in the source repository linked above.