Data Prep

> Part of the data-scientist skill suite. Install data-scientist for full lifecycle orchestration and routing.

Data Prep

Purpose

Safely clean, transform, join, and prepare data for analysis, modeling, dashboards, or reporting. The posture is careful and auditable: every transformation is intentional, every destructive change is documented, and raw data is always preserved.

This skill does not run analysis on the cleaned data — it prepares it. The output is a transformation plan, cleaned data or transformation code, a before/after validation summary, and a handoff packet for the next skill.

When To Use This Skill

Use data-prep when:

• The user asks to clean, fix, or prepare a dataset.
• Missing values need to be handled.
• Duplicate rows need to be resolved.
• Data types need to be converted.
• Categories need to be standardized.
• Dates need to be parsed or normalized.
• Tables need to be joined safely.
• Outliers need to be handled.
• Values need to be imputed.
• Features need to be prepared for modeling.
• A transformation pipeline needs to be built.
• An analysis-ready dataset needs to be produced.

When Not To Use This Skill

| Situation | Use instead | |---|---| | Pure data profiling with no transformations | data-explorer | | KPI definition or SQL metric logic only | metric-analyst | | Modeling trust or validation review | model-auditor | | Dashboard layout or visual design | dashboard-designer | | Simple conceptual explanation of cleaning methods | Answer directly (no dataset involved) |

Relationship to Parent Skill

| Responsibility | Owner | |---|---| | Routing to data-prep | Parent data-scientist (workflow/specialist-routing.md) | | Confirming data profile exists | This skill (requests data-explorer output if missing) | | Identifying cleaning and prep issues | This skill | | Separating safe vs. destructive transformations | This skill | | Proposing and applying transformations | This skill | | Preserving raw data and audit trail | This skill | | Before/after validation | This skill | | Handoff to downstream specialist | This skill |

Entry Gates

Before beginning data prep, confirm or state as assumptions:

1. Intended downstream use — what will the cleaned data be used for? Analysis? Modeling? Dashboard? Report?
2. Data profile — has data-explorer been run? If not, request a profile first or perform a quick inline profile. Do not clean data without understanding it first.
3. Raw data preservation plan — is there a raw copy, or will one be created before transformations begin?

If the data profile is missing and the user has not waived it, request it before proceeding.

Prep Workflow

1. Confirm the intended downstream use.

Know whether the cleaned data is going to: analysis, modeling, a dashboard, reporting, or production. This determines which issues are blocking vs. acceptable.

1. Use the data-explorer profile if available; otherwise perform or request one.

Do not begin cleaning without knowing the schema, grain, missingness, and data types. A quick inline profile is acceptable if the formal profile has not been run.

1. Identify cleaning and prep issues by severity.

• Blocking: Missing join keys, wrong grain, invalid target field, duplicate primary keys that corrupt the analysis.
• High: High missingness in key fields, inconsistent data types, invalid dates, category inconsistency.
• Medium: Moderate missingness in non-key fields, outliers in numeric fields, minor category inconsistencies.
• Low: Cosmetic issues, minor formatting, non-critical fields.

1. Separate safe transformations from judgment-heavy or destructive ones.

• Safe: type conversion, date parsing, whitespace trimming, column renaming.
• Judgment-heavy: imputation strategy, outlier treatment policy, category merging, join type selection.
• Destructive: row deletion, column deletion, overwriting values, database writes.

1. Propose rules for destructive or assumption-heavy changes.

Before imputing, dropping rows, removing outliers, or collapsing categories: state the rule, the rationale, and the impact on row count and key metrics.

1. Preserve raw data and maintain an audit trail.

Always preserve the raw data. Log every transformation: what changed, how many rows/values were affected, and what rule was applied.

1. Apply or describe transformations.

Produce code (Python, SQL, R, or pseudocode), a step-by-step transformation plan, or both. Every step must be reproducible.

1. Validate before/after.

After transformations, check:

• Row count (before vs. after)
• Column list (what was added, renamed, removed)
• Missingness by key field
• Duplicate count at grain level
• Data types by field
• Join cardinality (if tables were joined)
• Key metric sanity checks (does a sum, count, or rate look reasonable?)

1. Produce an analysis-ready dataset summary.

Document: grain, primary key, key fields, field types, missingness status, row count, known caveats.

1. Handoff to the next specialist.

Pass the data prep summary and remaining caveats to metric-analyst, experiment-analyst, model-auditor, dashboard-designer, or the parent data-scientist for modeling.

Hard Rules

These rules may not be skipped or rationalized away:

• Never silently drop rows. Every row deletion must be logged with the rule applied and the count affected.
• Never silently impute values. Every imputation must be documented with the strategy, the field, and the value count affected.
• Never silently remove outliers. Outlier treatment must be documented with the rule and the count removed.
• Never silently collapse categories. Category mapping must be documented with before/after values.
• Never overwrite raw files without preserving the raw version. Always maintain a clean copy.
• Never change join keys or grain without documenting the effect. State the before and after row count and cardinality.
• User approval required for:
• Any deletion of a material number of rows (>1% without a clear rule)
• Any imputation of a critical field (target variable, join key, primary metric)
• Any irreversible database write
• Any production-impacting transformation
• Any transformation that changes the grain of the dataset

Transformation Categories

Type Conversion

• Cast numeric strings to int/float.
• Convert boolean-encoded fields (0/1, "Y"/"N", "true"/"false") to a consistent type.
• Parse currency strings (remove "$", ",") before casting.

Date Parsing

• Standardize date formats to ISO 8601 (YYYY-MM-DD) or the downstream system's required format.
• Handle timezone inconsistency.
• Flag future dates, pre-1900 dates, or dates that violate business logic.

Missing Value Handling

• Do not impute critical fields (join keys, primary keys, target variables) without explicit strategy and approval.
• For non-critical numeric fields: state the imputation strategy (mean, median, zero, forward-fill) and its rationale.
• For categorical fields: consider adding an explicit "Unknown" category rather than imputing.
• Always preserve missingness information: add a _was_missing flag column for fields where missingness itself is informative.

Deduplication

• Define the grain (what combination of columns uniquely identifies a record).
• Decide the deduplication rule (keep first, keep last, keep max, aggregate).
• Log: how many duplicates were found, which rule was applied, and how many rows remain.

Category Standardization

• Normalize case (uppercase, lowercase, title case).
• Trim whitespace.
• Map variant spellings to canonical values.
• Flag or group rare categories (< N% of records) explicitly — do not blindly lump into "Other."

Outlier Handling

• Do not blindly remove outliers. First confirm they are data quality issues, not real business events.
• Options: cap/floor (winsorization), flag and exclude, transform (log scale), leave for domain review.
• Log count affected and the rule applied.

Joins

• Verify join key cardinality before joining: 1:1, 1:many, many:many.
• Many:many joins require explicit deduplication or aggregation before joining.
• After joining, validate: row count is as expected, no fan-out inflation, no unintended nulls from outer joins.
• Document: join type, join key, expected and actual cardinality, and post-join row count.

Feature Preparation (for modeling)

• Apply all feature transformations inside the validation loop — not before splitting.
• Document feature derivation: what input field(s), what transformation, what output field.
• Flag features with high missingness in the training population.
• Flag features that may not be available at the prediction point (leakage risk).

Output Formats

• Cleaning plan — severity-ranked list of issues and proposed transformations.
• Transformation code — Python, SQL, R, or pseudocode for each transformation step.
• Before/after validation summary — row counts, field types, missingness, duplicates, and key metric checks.
• Analysis-ready dataset summary — grain, key fields, schema, caveats, remaining issues.
• Data prep handoff packet — structured context passed to the next skill.

Handoff Packet Template

**Data Prep Handoff**

Downstream use: [analysis / modeling / dashboard / reporting]
Grain: [one row = one what?]
Primary key: [field name(s)]
Cleaning steps applied: [bullet list with row counts and rules]
Rows removed: [count and reason]
Imputation applied: [field, strategy, count affected]
Outlier treatment: [field, rule, count affected]
Category changes: [mappings applied]
Join changes: [tables joined, type, cardinality, post-join count]
Before/after row count: [X → Y rows]
Remaining caveats: [issues not resolved and why]
Analysis-ready fields: [key fields available for downstream use]
Excluded fields: [fields dropped and reason]

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: DAlanMtz
• Source: DAlanMtz/data-scientist
• License: MIT

Install and usage instructions live in the source repository linked above.