Data Journalism

Data journalism methodology

Systematic approaches for finding, analyzing and presenting data in journalism.

Story structure for data journalism

Data journalism framework

The framework for data journalism was established by Philip Meyer, a journalist for Knight-Ridder, Harvard Nieman Fellow and professor at UNC-Chapel Hill. In his book The New Precision Journalism, Meyer encourages journalists to treat journalism "as if it were a science" by adopting the scientific method:

• Make observations / formulate a question
• Research the question / collect, store, and retrieve data
• Formulate a hypothesis
• Test the hypothesis, using both qualitative (interviews, documents) and quantitative (data analysis) methods
• Analyze the results and reduce them to the most important findings
• Present them to the audience

The process is iterative, not sequential.

The data story arc

1. The hook (nut graf)

• What's the key finding?
• Why should readers care?
• What's the human impact?

2. The evidence

• Show the data
• Explain the methodology
• Acknowledge limitations

3. The context

• How does this compare to the past?
• How does this compare to elsewhere?
• What's the trend?

4. The human element

• Individual examples that illustrate the data
• Expert interpretation
• Affected voices

5. The implications

• What does this mean going forward?
• What questions remain?
• What actions could result?

6. The methodology box

• Where did the data come from?
• How was it analyzed?
• What are the limitations?
• How can readers explore further?

Methodology documentation template

## How we did this analysis

### Data sources
[List all data sources with links and access dates]

### Time period
[Specify exactly what time period is covered]

### Definitions
[Define key terms and how you operationalized them]

### Analysis steps
1. [First step of analysis]
2. [Second step]
3. [Continue...]

### Limitations
- [Limitation 1]
- [Limitation 2]

### What we excluded and why
- [Excluded category]: [Reason]

### Verification
[How findings were verified/checked]

### Code and data availability
[Link to GitHub repo if sharing code/data]

### Contact
[How readers can reach you with questions]

Data acquisition

Public data sources

Federal data sources

General:

• Data.gov — Federal open data portal. Many datasets were removed between Feb 2025 and 2026; consult the Harvard LIL Data.gov archive and the Data Rescue Project for preserved copies before assuming anything is still accessible.
• Census Bureau (census.gov) — Demographics, economic data. Many research pages were removed during the 2025 transition; the End of Term Web Archive holds snapshots.
• BLS (bls.gov) — Employment, inflation, wages. Following the 2025 funding lapse, the October 2025 Employment Situation release was canceled and the CPS October 2025 reference period is permanently uncollected. Check revised release dates before relying on series continuity.
• BEA (bea.gov) — GDP, economic accounts.
• FRED / Federal Reserve (fred.stlouisfed.org) — Financial and macroeconomic data; expanded API access through 2026.
• SEC EDGAR — Corporate filings.

Specific domains:

• EPA (epa.gov/data) — Environmental data. At least 80 climate webpages were removed in Dec 2025, the endangerment finding was repealed Feb 12, 2026, and the Climate Change Indicators site was largely gutted. The Environmental Data & Governance Initiative maintains mirrors.
• FDA / openFDA (open.fda.gov) — Drug approvals, recalls, adverse events.
• CDC WONDER — Health statistics. Many datasets were removed from data.cdc.gov after Jan 2025, partially restored under Doctors for America v. Trump (TRO Feb 11, 2025) but with altered terminology in some returns. The volunteer-run RestoredCDC.org mirrors removed content.
• NHTSA FARS / vPIC APIs — Vehicle safety data.
• DOT — Transportation statistics.
• FEC — Campaign finance; 2025-2026 cycle data live.
• USASpending.gov — Federal contracts and grants; API v2 operational.

Court records:

• CourtListener / RECAP (courtlistener.com) — Free PACER alternative covering federal court filings; RECAP Search Alerts launched June 2025 ("Google Alerts for federal courts").
• PACER — Federal court filings; $0.10 per page, $30 per quarter waiver threshold.

State and local:

• State open data portals (search: "[state] open data")
• Tyler Data & Insights (formerly Socrata, rebranded May 2025) hosts many city and state portals
• OpenStreetMap, municipal GIS portals
• State comptroller and auditor reports

International:

• Eurostat, OECD, World Bank Open Data, UN Data — major comparative datasets, mostly stable through 2026.

Specialized:

• NICAR Data Library (IRE) — curated datasets, IRE members only.
• IPUMS (University of Minnesota) — free with account; canonical for harmonized microdata.
• ICPSR (University of Michigan) — social-science data archive.
• ProPublica Data Store — frozen; datasets only run through 2023.

Federal-data preservation (use when source data has been removed):

• Data Rescue Project — citizen + library mirrors of removed federal data; more than 1,230 datasets across 85 offices as of Aug 2025.
• End of Term Web Archive — 500TB / 100M-page snapshot of federal sites at the 2024-2025 transition.
• Internet Archive Wayback Machine — useful for individual page-level recovery.

Data request strategies

Public records requests for datasets

For request mechanics (templates, fee-waiver language, NJ OPRA, appeals, FOIA Improvement Act statutory citations), see the foia-requests skill. Data-specific guidance:

• Request databases, not just documents
• Ask for the data dictionary or schema
• Request in native format (CSV, SQL dump) — not PDFs or scanned printouts
• Specify field-level needs and any computed columns you want included
• For active datasets, ask the cadence (daily, monthly, quarterly) and request standing access if your reporting will continue

Building your own dataset

• Scraping public information (respect robots.txt, ToS, and rate limits)
• Crowdsourcing from readers
• Systematic document review
• Surveys with documented methodology

Commercial data sources for newsrooms

• LexisNexis, Refinitiv, Bloomberg
• Industry-specific databases (often via library proxy through your institution)

Data cleaning and preparation

Common data problems

from typing import Any

import pandas as pd
import numpy as np
from rapidfuzz import fuzz
from itertools import combinations

# Inflation adjustment
import cpi
import wbdata

def standardize_name(name: Any) -> str | None:
    """Standardize name format to 'First Last'."""
    if pd.isna(name):
        return None
    name = str(name).strip().upper()
    # Handle "LAST, FIRST" format
    if ',' in name:
        parts = name.split(',')
        name = f"{parts[1].strip()} {parts[0].strip()}"
    return name

def parse_date(date_str: Any) -> pd.Timestamp | None:
    """Parse dates in various formats."""
    if pd.isna(date_str):
        return None

    formats = [
        '%m/%d/%Y', '%Y-%m-%d', '%B %d, %Y',
        '%d-%b-%y', '%m-%d-%Y', '%Y/%m/%d'
    ]

    for fmt in formats:
        try:
            return pd.to_datetime(date_str, format=fmt)
        except:
            continue

    # Fall back to pandas parser
    try:
        return pd.to_datetime(date_str)
    except:
        return None

def handle_missing(
    df: pd.DataFrame,
    thresh: int | None = None,
    per_thresh: float | None = None,
    required_col: str | None = None,
) -> pd.DataFrame:
    """Drop rows missing values in `required_col` if missingness exceeds either threshold."""
    if required_col is None or df.empty:
        return df
    if required_col not in df.columns:
        return df

    missing = df[required_col].isna().sum()

    if thresh is not None and missing >= thresh:
        return df.dropna(subset=[required_col]).reset_index(drop=True).copy()

    if per_thresh is not None and (missing / len(df) * 100) >= per_thresh:
        return df.dropna(subset=[required_col]).reset_index(drop=True).copy()

    return df

def handle_duplicates(df: pd.DataFrame, thresh: int | None = None) -> pd.DataFrame:
    """Drop duplicate rows when count exceeds `thresh`."""
    if thresh is not None and df.duplicated().sum() >= thresh:
        return df.drop_duplicates().reset_index(drop=True).copy()
    return df

def flag_similar_names(df: pd.DataFrame, name_col: str, threshold: int = 85) -> pd.DataFrame:
    """Flag rows that have potential duplicate names using vectorized comparison."""
    
    names = df[name_col].dropna().unique()
    
    # Use combinations() to avoid nested loop and duplicate comparisons
    dup_names: set[Any] = {
        name
        for name1, name2 in combinations(names, 2)
        if fuzz.ratio(str(name1).lower(), str(name2).lower()) >= threshold
        for name in (name1, name2)
    }
    
    df['has_similar_name'] = df[name_col].isin(dup_names)
    return df

def flag_outliers(series: pd.Series, method: str = 'iqr', threshold: float = 1.5) -> pd.Series:
    """Flag statistical outliers."""
    if method == 'iqr':
        Q1 = series.quantile(0.25)
        Q3 = series.quantile(0.75)
        IQR = Q3 - Q1
        lower = Q1 - threshold * IQR
        upper = Q3 + threshold * IQR
        return (series  upper)
    elif method == 'zscore':
        z_scores = np.abs((series - series.mean()) / series.std())
        return z_scores > threshold

# use descriptive variable names and chain methods
data_clean = (pd

            # Load messy data — raw_data is a placeholder
            # Be sure to use the right reader for the filetype
            .read_csv('..data/raw/raw_data.csv')

            # DATA TYPE CORRECTIONS
            # Ensure proper types for analysis
            .assign(# Convert to numeric (handling errors)
                    amount = lambda x: pd.to_numeric(x['amount'], errors='coerce'),
                    
                    # Convert to categorical (saves memory, enables ordering)
                    status = lambda x: pd.Categorical(x['status']))
            
            .assign(
                    # INCONSISTENT FORMATTING
                    # Problem: Names in different formats
                    # e.g., "SMITH, JOHN" vs "John Smith" vs "smith john"
                    name_clean = lambda x: standardize_name(x['name']),

                    # DATE INCONSISTENCIES
                    # Problem: Dates in multiple formats
                    # e.g., "01/15/2024", "2024-01-15", "January 15, 2024", "15-Jan-24"
                    parse_date = lambda x: parse_date(x['date']),
                    
                    # OUTLIERS
                    # Identify potential data entry errors
                    amount_outlier = lambda x: flag_outliers(x['amount']),
                    
                    )
            
            # Fuzzy duplicates (similar but not identical)
            # Use record linkage or manual review
            .pipe(flag_similar_names, name_col='name_clean', threshold=85)

            # MISSING VALUES
            # Strategy depends on context — set required_col when you need to drop incomplete rows
            .pipe(handle_missing, required_col='amount', per_thresh=20.0)

            # DUPLICATES — Find and handle duplicates
            .pipe(handle_duplicates, thresh=1)
            
            .reset_index(drop=True)
            .copy())

Data validation checklist

## Pre-analysis data validation

### Structural checks
- [ ] Row count matches expected
- [ ] Column count and names correct
- [ ] Data types appropriate
- [ ] No unexpected null columns

### Content checks
- [ ] Date ranges make sense
- [ ] Numeric values within expected bounds
- [ ] Categorical values match expected options
- [ ] Geographic data resolves correctly
- [ ] IDs are unique where expected

### Consistency checks
- [ ] Totals add up to expected values
- [ ] Cross-tabulations balance
- [ ] Related fields are consistent
- [ ] Time series is continuous

### Source verification
- [ ] Can trace back to original source
- [ ] Methodology documented
- [ ] Known limitations noted
- [ ] Update frequency understood

AI-assisted analysis: cautions

AI tools can speed up exploration, code generation, and pattern surfacing — but they have specific failure modes that journalists must guard against. Mata v. Avianca (2023, fabricated court citations sanctioned in federal court) and the Air Canada chatbot ruling (2024, hallucinated refund policy ruled binding on the airline) are the canonical cases of LLM fabrication treated as published fact.

What LLMs reliably get wrong

• Calculations at scale — A model may produce a confident-looking sum, percentage, or rate that's off by 1-15%. Re-run any LLM-produced number in pandas, SQL, or R yourself before publishing.
• Source citations — Models hallucinate plausible URLs, paper titles, dataset names, and FOIA exemptions that don't exist. Verify every cited source by visiting it.
• Dataset columns — When asked to describe a dataset's structure, an LLM may invent columns that aren't there. Cross-check against the actual schema (df.dtypes, df.columns.tolist()).
• Statistical reasoning — LLMs confuse correlation with causation, conflate sample statistics with population parameters, and misapply tests. Treat any analytical claim as a hypothesis to verify, not a finding.

Methodology disclosure

When AI was used in any stage (data cleaning, analysis, visualization, drafting), disclose it in the methodology box. Editors and readers need to know which steps had a human in the loop and which were automated.

• State the tool and version (e.g., "We used Claude 4.7 to draft the cleaning pipeline; the code was reviewed and run by [reporter]").
• State what was verified (e.g., "All numerical results were re-computed in pandas; all source citations were independently confirmed").
• State what was not verified (if relevant).

Reproducibility

When using AI to generate analysis code, save the prompt, the model name, and the version alongside the code. AI-generated code is part of your methodology and should be reproducible by another reporter on the same data.

Statistical analysis for journalism

Basic statistics with context

# Essential statistics for any dataset
def describe_for_journalism(df: pd.DataFrame, col: str) -> pd.DataFrame:
    """Generate journalist-friendly statistics."""
    stats = df[col].describe(percentiles=[0.25, 0.5, 0.75, 0.9, 0.99])
    
    # Add skewness to the describe() output
    stats['skewness'] = df[col].skew()
    
    return stats.to_frame(name=col)

# Example interpretation
stats = describe_for_journalism(salaries, 'salary')

print(f"""
ANALYSIS
---------------
We analyzed {stats['count']:,} salary records.

The median salary is ${stats['median']:,.0f}, meaning half of workers
earn more and half earn less.

The average salary is ${stats['mean']:,.0f}, which is
{'higher' if stats['mean'] > stats['median'] else 'lower'} than the median,
indicating the distribution is {'right-skewed (pulled up by high earners)'
if stats['skewness'] > 0 else 'left-skewed'}.

The top 10% of earners make at least ${stats['90th_percentile']:,.0f}.
The top 1% make at least ${stats['99th_percentile']:,.0f}.
""")

Comparisons and context

# Calculate change metrics for a column
def calculate_change(df: pd.DataFrame, col: str, periods: int = 1) ->

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [jamditis](https://github.com/jamditis)
- **Source:** [jamditis/claude-skills-journalism](https://github.com/jamditis/claude-skills-journalism)
- **License:** MIT
- **Homepage:** https://skills.amditis.tech/

Install and usage instructions live in the source repository linked above.