Smart Eda

Smart EDA

Adaptive exploratory data analysis for tabular data. The skill profiles the dataset first, then picks methods based on the data's actual shape, types, scale, and the user's intent.

Core philosophy

EDA is detective work, not a checklist. John Tukey introduced it as "actively incisive, rather than passively descriptive, with real emphasis on the discovery of the unexpected." The skill operationalizes three attitudes:

• Profile before analyzing. Never apply a method without first checking whether the data fits its assumptions.
• Adapt to the data. A dtype=int64 column with 4 unique values is a categorical, not a numeric. A correlation of 0.9 between two columns might be a leak, not a finding. Branch on what's actually there.
• Surface insights, not just stats. Every EDA closes with top findings, anomalies, hypotheses, and concrete next steps — not a wall of describe() output.

When to use which tier

EDA work scales. The skill operates in three tiers; pick based on the request and the data:

• Tier 1 — 30-second scan. Shape, dtypes, missing percent, dupes, sample. Use when the user just says "what's in this" or as the warmup for any deeper work.
• Tier 2 — Standard EDA (default). Tier 1 + per-column univariate (adapted to type) + correlation + target relationship if a target column exists + time-series plot if datetime found + quality issues + top findings. Use this unless the request explicitly asks for less or more.
• Tier 3 — Deep dive. Tier 2 + formal tests with effect sizes + multivariate outliers + PCA/clustering + change-point detection + hypothesis catalog. Use when the user asks for "thorough", "deep", "everything", "full audit", or when Tier 2 surfaces something that demands closer inspection.

State the chosen tier briefly at the start, e.g. "Running standard EDA — I'll go deeper on anything that looks worth it."

Workflow

Phase 0 — Understand the ask and locate the data

Before any code, answer:

1. What is the user asking? Quick scan, full EDA, specific question (e.g. "find outliers"), or open-ended ("look at this")?
2. Where is the data? Uploaded file in /mnt/user-data/uploads/, a path in their message, or a DataFrame mentioned in prior context?
3. One file or many? If multiple files were uploaded or referenced, this is a multi-file EDA — see Phase 1.5. Common patterns: relational tables (orders + customers + products), monthly/regional snapshots of the same schema, or independent datasets the user wants compared.
4. Is there a target/outcome variable? A column named like target, label, y, outcome, churn, price, success, or an explicit mention. If so, EDA will pivot toward feature-vs-target analysis.
5. Is there a domain hint? "Sales", "patients", "trades", "logs" change which patterns to look for and which quality issues to flag.

If the user uploaded a file with no instruction, default to Tier 2 and proceed. Don't ask 5 clarifying questions — that's the fastest way to make the skill annoying. Asking one question is fine if there's genuine ambiguity (e.g. "Quick look or full EDA?"); otherwise just start and let them redirect.

Phase 1 — Load and run reconnaissance

Use the bundled scripts/profile_data.py to do this efficiently. It handles CSV, TSV, Excel, Parquet, and JSON-lines, infers types, and emits a structured profile — without burning tokens on Claude writing this boilerplate every time:

python /path/to/skill/scripts/profile_data.py  --output /home/claude/profile.json

The profile includes: shape, memory, per-column dtype, semantic type (numeric/categorical/datetime/text/id/boolean), nunique, missing count, sample values, and the basic stats. Read the JSON, then narrate the recon in plain language — don't dump the JSON to the user.

If the file is unusual (>500MB, weird format, multiple sheets), fall back to manual loading with size-aware techniques (see references/large_data.md).

Phase 1.5 — Multi-file analysis (when applicable)

When the user has more than one tabular file, use scripts/multi_file_eda.py. It profiles each file, detects join keys via name/type compatibility and value overlap, infers cardinality (1:1 / 1:N / N:N), measures orphan rates on both sides of every relationship, and produces a single dashboard showing the schema, the file inventory, the detected relationships, and a combined EDA on the auto-joined data.

python /path/to/skill/scripts/multi_file_eda.py file1.csv file2.csv file3.csv \
    --output-dir /mnt/user-data/outputs --target sales

The script:

• Profiles each file individually (per-file quality flags, types, etc.)
• Looks for shared column names (or singular/plural pairs like customer ↔ customers) with compatible types and ≥30% value overlap
• Picks the strongest non-conflicting relationships and performs left-joins, starting from the fact table (the one with the most outgoing N:1 references)
• Runs the standard eda_runner flow on the joined dataset and embeds the results
• Renders a schema diagram with the fact table centered and dimension tables fanning out, with arrows labeled by cardinality

When to use which:

• Single file → standard flow (profile_data.py → eda_runner.py)
• Multiple files, related (orders/customers/products) → multi-file EDA, auto-join
• Multiple files, same schema (monthly snapshots) → concatenate first with pd.concat, then standard flow on the concatenated frame; mention the concat to the user
• Multiple files, unrelated → run standard flow on each separately, present each dashboard

If the multi-file script reports zero relationships and the files are clearly unrelated (e.g., different domains), fall back to per-file analysis. If it reports zero but you suspect there should be relationships (column names differ but data matches), tell the user — they may need to rename a column or specify the keys manually.

Phase 2 — Data quality scan

Catch the issues that derail later analysis. From the profile, surface:

• Missing values — count + percent per column. Flag any > 5% as worth discussing; > 50% as probably-drop candidates.
• Duplicates — full-row, and key-based if an obvious ID column exists.
• Constants / near-constants — nunique() == 1 or top frequency > 99%.
• Mixed types within a column — object columns where values aren't all the same Python type.
• Whitespace / casing inconsistency — for object columns, compare nunique() before and after .str.strip().str.lower().
• Out-of-range / impossible values — age 100.
• Encoding artifacts — characters like Ã, ’, \xa0 in text columns.

Don't fix these yet. The job at this phase is to find them and decide whether they affect later analysis. For deep audits, see references/quality_issues.md.

Phase 3 — Univariate analysis (branch by effective type)

For each column, decide its effective type (which may differ from its dtype) and apply the right methods. The decision rules:

• dtype is numeric AND nunique ≤ 10 → treat as ordinal-categorical: bar chart, value counts, not histogram
• dtype is numeric AND nunique > 10 → numeric: histogram + KDE, boxplot, 5-number summary, skewness, IQR-outlier flag
• dtype is object AND all values match a date/time pattern → datetime: parse and treat as datetime
• dtype is object AND mean string length > 50 → free text: length distribution, top n-grams, language/encoding check
• dtype is object AND nunique / total > 0.95 → ID-like: skip distribution analysis, just confirm uniqueness
• dtype is object otherwise → categorical: value counts, bar chart, rare-category flag ( 1`, mention that a log/Box-Cox transform may help and suggest it

1. Compute IQR-based outlier count; if > 1% of rows, mention it but don't drop anything

For categorical columns, always:

1. Report top-5 values with frequencies
2. Flag high cardinality (nunique > 50) as needing binning before modeling
3. Flag rare categories ( 0.9 as possible redundancy / leakage.

• Categorical × categorical: Cramer's V matrix for the top categorical pairs.
• Don't try to compute correlation between every pair if there are > 30 columns; focus on either (a) pairs with the highest variance or (b) the top correlations.

Multivariate:

• Pair plot if ≤ 8 numeric columns AND ≤ 5k rows (otherwise skip — the plot becomes unreadable)
• VIF for multicollinearity if numeric columns ≥ 5 and a target was given
• PCA with scree plot if numeric columns ≥ 10

For statistical-test interpretation guidance (when each test is appropriate, what its assumptions are, how to report it), see references/statistical_tests.md.

Phase 5 — Specialized analysis

Run only what applies to this data:

• Time series — if a datetime column was detected and the data appears time-ordered: line plot, resample to a sensible frequency, rolling mean + std, decomposition (trend / seasonal / residual), stationarity test (ADF), ACF/PACF. See references/time_series.md.
• Class imbalance — if a categorical target with ratio .py` — reproducible Python script.** Separate deliverable. The user can run, modify, version-control, or commit this script to recreate every chart and finding from the dashboard.

• Per-column plots (univariate distributions, target-relationship plots, correlation heatmap PNGs in the same folder) — supporting material referenced by the dashboard and the markdown.

• For Tier 3 deep dives: also produce an HTML report using the helper script with formal tests, multivariate outliers, PCA, change-point detection.

The two primary deliverables are designed for different audiences:

• The dashboard is for the person who wants to understand the data — managers, decision-makers, anyone scanning for the takeaway
• The markdown is for the person who needs to verify or document the analysis — analysts reviewing the work, reviewers signing off, future-you reading this in three months

Use present_files to surface both. Lead with dashboard.html for users who'll open it in a browser; on text-only surfaces, lead with eda_summary.md. The two complement each other — the dashboard lacks the per-column statistics tables; the markdown lacks the visual pattern-recognition.

Adaptive selection — quick reference

| Situation | Action | |---|---| | > 1M rows | Sample for visualizations, full data for stats; switch scatter to hex-bin | | > 50 columns | Don't run pair plots; focus on high-variance and high-correlation-with-target | | Numeric col with nunique ≤ 10 | Treat as ordinal categorical | | Object col mostly date-like | Parse to datetime | | Object col with avg length > 50 | Treat as free text, not categorical | | Pearson r computed but normality fails | Also report Spearman, prefer it | | ANOVA with unequal variance | Switch to Welch's | | Datetime col + ordered rows | Run time-series workflow | | Target with imbalance > 1:5 | Flag explicitly | | Two columns with r > 0.95 | Investigate as possible leak / redundancy | | Skewness | > 1 | Suggest log / Box-Cox transform |

Working principles

• Use the bundled scripts. scripts/profile_data.py, scripts/eda_runner.py, and scripts/dashboard_html.py exist precisely so Claude doesn't waste tokens rewriting the same boilerplate every time. Read them first; extend them if needed; don't duplicate them.
• Optional dependencies. The interactive HTML dashboard requires plotly (>= 5.0). If it's not installed and the user wants the interactive output, run pip install plotly (use --break-system-packages in container environments). The runner falls back gracefully — if Plotly isn't available, only the static PNG dashboard is built and a note is printed.
• Save plots before showing them. Always write to /mnt/user-data/outputs/eda_plots/ so they survive the conversation.
• Cite assumptions. When you use Pearson, note that it assumes linearity. When you use a t-test, note that it assumes normality and equal variance. When you sample, say so. The user shouldn't need to second-guess the analysis.
• Be honest about uncertainty. If a finding could be a coincidence (small sample, weak effect), say so. If a correlation could be a confounder, say so.
• Don't drop or impute silently. Flag the issues; let the user decide. The skill's job is to inform, not to clean for them (unless they explicitly ask).
• One question is fine, five is not. If the data is ambiguous about intent or target, one short clarifying question is OK. More than that wastes the user's time.

When to reach for the reference files

The reference files contain the exhaustive material. Read them when needed — don't try to remember everything in the SKILL.md.

| File | Read when | |---|---| | references/method_catalog.md | Choosing which technique to apply, especially for unusual column types or multivariate analyses | | references/statistical_tests.md | About to run a hypothesis test — pick the right one, get the assumptions, interpret the result | | references/time_series.md | Datetime structure detected, doing TS-specific work | | references/quality_issues.md | Doing a thorough quality audit; suspicious data | | references/large_data.md | Working with > 1M rows or files larger than memory | | references/visualization_guide.md | Picking the right chart for a given data shape |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: k4thir
• Source: k4thir/smart-eda
• License: MIT

Install and usage instructions live in the source repository linked above.