Becoming One With The Data

Becoming One With The Data

Overview

Stage 1 of the training recipe. Write zero model code until this gate passes.

The first step is not to touch any neural net code. It is to inspect your data exhaustively — scanning through thousands of examples, understanding their distribution, and looking for patterns. This is the single highest-value hour in most ML projects, and it is the one agents skip fastest.

Core principle: The neural net is a compressed, compiled summary of your dataset. When it makes a wrong prediction, the error was usually already visible in the data. Find it before you spend six hours training.

The Gate

You may proceed to stage 2 when you can answer, without running a model:

1. What will the model get wrong, and why? Name specific example types.
2. What is the label noise rate? Roughly, from a hand-audited sample.
3. Does local context suffice, or is global context needed? This decides

your architecture.

1. What varies that shouldn't matter? That tells you your augmentation.
2. Is there any way to get the right answer without the real signal? That is

the shortcut your model will learn instead.

If you cannot answer these, you have not looked at the data yet.

How To Actually Do It

1. Look at raw examples. Hundreds of them.

Not df.head(). Not summary statistics. Scan through actual examples, one at a time, for long enough to get bored. Boredom is the point — that is when you start noticing the pattern that breaks everything.

Look at the data exactly as the model will see it. After preprocessing, after tokenization, after the chat template, after truncation. Not the version in the source file.

2. Classify a sample by hand

Take 50 examples and label them yourself, before looking at the provided labels. Two things fall out:

• Where you disagree with the label = your label noise estimate
• What information you used to decide = what the architecture must capture

If you needed to look at the whole image, a local receptive field won't do. If you only needed one phrase, a giant context window is wasted capacity and extra overfitting surface.

3. Hunt for the specific killers

| What to look for | How | Why it matters | |---|---|---| | Duplicates | Hash exact, embed + nearest-neighbor for near | Duplicates across train/test inflate your metric silently | | Train/test leakage | Check for shared IDs, timestamps, source documents, users | The #1 cause of "wow, 99% accuracy" | | Corrupted labels | Hand-audit 50; check per-class error patterns | Sets a ceiling you will otherwise chase forever | | Class imbalance | Value counts, plotted | Decides your loss weighting and your baseline | | Truncation | Length histogram vs. your max_len | Silently deletes the answer from the end of long examples | | Spurious correlations | Can you predict y from metadata alone? | The shortcut the model will learn instead of the task | | Distribution shift | Compare train vs. val vs. production distributions | Explains why the number drops in deployment |

4. Visualize distributions and look at outliers

Plot every attribute you have. Then sort by each one and look at both tails.

Outliers almost always uncover bugs in data quality or preprocessing. An example with 400x the normal length, a label that appears twice, an all-zero input — each one is a bug report you got for free.

5. Write the search/filter/sort tooling first

Build a small script to filter and sort your data by any attribute — label, length, predicted confidence, source, class. You will use it constantly, in every later stage, especially when inspecting the model's worst predictions.

Making this reusable now pays for itself within the hour.

Modern Mapping

Fine-tuning / SFT: Render 200 training examples through the exact chat template the trainer uses and read them as strings. This one habit catches: a missing BOS token, a system prompt that isn't applied, the assistant turn being included in the loss when it shouldn't be (or excluded when it should), examples truncated mid-answer, and a stray {instruction} placeholder that never got formatted.

Eval harnesses: Read the eval set, then check it against your training data for contamination. For any public benchmark, assume contamination until you have checked. Grade 20 items by hand and compare to what your harness reports — if they disagree, your harness is wrong, not the model.

RAG / retrieval: Look at the retrieved chunks, not the query. Half of RAG failures are visible immediately: chunks split mid-sentence, boilerplate headers dominating the embedding, the answer never appearing in any retrieved chunk at all.

Common Mistakes

| Mistake | Fix | |---|---| | Running .describe() and calling it done | Summary stats hide exactly the examples that break you. Read raw examples. | | Looking at the source file, not the model's view | Inspect post-preprocessing, post-tokenization, post-template. | | Trusting a "standard" or public dataset | Standard datasets have well-known label noise and contamination. Check yours. | | Skipping this because the data is huge | Sample 500. Reading 500 examples takes 30 minutes and is never wasted. | | Only looking at correct/typical examples | The tails are where the bugs live. Sort and read both ends. | | Doing this once at the start | Return here after every stage, especially to read the model's worst predictions. |

Red Flags

• "The dataset is standard, I'll skip inspection"
• "I'll look at the data if results are bad"
• "The data pipeline is from an existing repo, it's fine"
• Writing Dataset/DataLoader code before having read any examples
• Reporting a metric without knowing the label noise rate
• Being pleased by a high number instead of suspicious of it

Next

Gate passed? Proceed to ml-superpowers:building-the-training-skeleton.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Umaraslam66
• Source: Umaraslam66/ml-superpowers
• License: MIT

Install and usage instructions live in the source repository linked above.