Building The Training Skeleton

Building The Training Skeleton

STOP — Run The Auditor Before Reporting Any Number

Before you report an accuracy, loss, or any metric — including from a script you did not write and were only asked to run — run this first. One command, under a second:

python3 scripts/audit-run.py 

The script is at ../../scripts/audit-run.py relative to this skill file.

It exits non-zero and prints findings if it detects:

| Finding | What it means | |---|---| | LEAK | Train and test slice the same array with overlapping ranges. The metric is measuring memorization. | | GROUP | Grouped data (patient, user, session, document) split randomly. The same entity lands on both sides. | | TIME | Data with a time column split randomly. You are training on the future and testing on the past. | | PREP | A scaler/encoder was fit before the split. Test-set statistics leaked into training. | | BASE | No majority-class baseline computed. An accuracy with nothing to compare against is not interpretable. | | DATA | The loader fabricates data with a random generator. No metric from it describes the real task. |

If the auditor reports anything, do not report the number. Say what it found instead. "The harness runs, the split is broken, here is the corrected figure" is a useful answer; a wrong number someone puts in a deck is not.

Time pressure makes this more important, not less — a number that gets walked back costs more than a number delivered late.

Then check loss at init by hand

The auditor cannot see runtime values. Compare the first printed loss against -log(1/n_classes):

10 classes → 2.303    1000 → 6.908    binary → 0.693

Far below means a label is leaking into the input. Far above means the final-layer init is wrong.

Overview

Stage 2 of the training recipe. This is the stage that catches silent bugs, and it is the stage that gets skipped. Do not skip it. It takes about 20 minutes and it is the difference between engineering and gambling.

Set up a full training and evaluation skeleton with a model so dumb it cannot possibly be wrong, and gain trust in its correctness via a series of experiments where you predict the outcome before you run it.

Core principle: A check is only worth running if you write down the expected value first. "Let's see what happens" catches nothing. "This should print 2.303, and if it doesn't, my label mapping is broken" catches everything.

The Gate

You may proceed to stage 3 when all of these hold:

• [ ] Loss at initialization equals the value you predicted
• [ ] Input-independent baseline performs strictly worse than the real model
• [ ] A single batch of ~2-10 examples overfits to near-zero loss
• [ ] You have a human baseline (or a stated ceiling) to compare against
• [ ] You have visually inspected the tensor entering the model
• [ ] Evaluation runs on the entire test set, not a subset
• [ ] The seed is fixed and two runs produce identical numbers

Any unchecked box means you do not yet know your pipeline is correct.

The Setup Rules

Fix the random seed. Always. Removes an entire class of "why is it different now?" confusion.

Simplify. Turn off every optional feature — augmentation, mixed precision, EMA, LR schedules, fancy samplers. Augmentation is a stage 4 regularizer; right now it is a bug generator in disguise.

Add significant digits. Evaluate on the entire test set and print exact numbers. A full-set mean is comparable across runs; an eyeballed running average is not.

Generalize a special case. Write the specific function you need now, verify it, then generalize while asserting the result is unchanged. This matters most for vectorized code: get the single-example loop right before batching it.

The Sanity Checks

Each check states what it catches. Run them in order.

1. Verify loss @ init

Verify your loss starts at the correct value. For a softmax over n classes at proper initialization, that is -log(1/n_classes).

10 classes → 2.303    1000 classes → 6.908    binary → 0.693

Catches: wrong number of classes, wrong reduction, double-softmax (passing already-softmaxed logits to cross-entropy), broken initialization, label off-by-one, logits and labels transposed.

If it starts far below the expected value, you are leaking the label.

2. Init well

Initialize the final layer weights correctly. If you are regressing values with mean 50, initialize the final bias to 50. For imbalanced classes at ratio 1:10, set the bias on the logits so the initial prediction reflects that ratio.

Catches: nothing directly — but it removes a "hockey stick" loss curve where the first few hundred steps are spent learning the bias, which otherwise masks whether your model is learning anything real.

3. Human baseline

Do the task yourself. Record your accuracy. If you cannot do it, that is the finding — either the task is underspecified or the inputs lack the information.

Catches: impossible tasks, missing features, and the "is 71% good?" question that otherwise haunts the whole project.

4. Input-independent baseline

Train a baseline with the input zeroed out (or shuffled). Compare it to the real model on identical everything else.

The real model must beat it. If it does not, your model is not extracting any information from the input. This is the check that catches the most embarrassing bugs.

Catches: inputs never reaching the model, a detached graph, features that are all zeros/NaN after normalization, a model that learned only the class prior.

5. Overfit one batch

Take a single batch of just a few examples — as few as two — and overfit it. Increase capacity if needed. Drive the loss to essentially zero.

Then visualize the label and prediction side by side and confirm they line up exactly at minimal loss. If they don't, there is a bug and you cannot proceed.

Catches: almost every remaining silent bug. Missing optimizer.zero_grad() or loss.backward(), a frozen or detached parameter, an inverted mask, targets misaligned with predictions, a loss that ignores its inputs, gradients not flowing through a custom layer.

This is the single most valuable check in the recipe. A model that cannot memorize two examples has a bug, full stop. It is never a capacity problem, a learning rate problem, or a data problem.

6. Verify decreasing training loss

Now train on the full dataset with a deliberately small model. Then make it bigger. The bigger model must reach a lower training loss.

Catches: a capacity ceiling you didn't know about, a bottleneck layer, a regularizer you forgot you left on.

7. Visualize just before the net

The unambiguously correct place to visualize your data is immediately before y_hat = model(x). Print or render exactly that tensor.

Decode the tokens back to text. Save the image tensor to a PNG. Look at it.

Catches: wrong normalization, channels transposed, an off-by-one crop, the chat template not applied, padding on the wrong side, an inverted attention mask, augmentation accidentally left on at eval. This is the single source of truth about what your model actually consumes, and it disagrees with what you think more often than you would believe.

8. Visualize prediction dynamics

Visualize predictions on a fixed test batch over the course of training. Watching how they move tells you far more than a loss number.

Wild oscillation means the learning rate is too high. Frozen predictions mean it is too low or gradients aren't flowing. A model that confidently commits early and never revises is often overfitting or has a leaked feature.

9. Use backprop to chart dependencies

Vectorized code makes it easy to accidentally mix information across the batch or time dimension — and it trains fine when you do.

To check: set the loss to something trivial like the sum of outputs for example i only, run backward, and confirm you get a non-zero gradient only on example i.

Catches: batch dimension mixing (view vs transpose errors), causal attention masks that leak the future, a rolling window that peeks ahead. These bugs produce better-looking training curves, which is why they survive.

Modern Mapping

Fine-tuning: Loss @ init should match the base model's loss on the same data; if not, your template or tokenization differs from pretraining. Overfit-one-batch on 10 examples verifies the loss mask covers the right tokens. Input-independent = train on shuffled responses; must get worse.

Eval harnesses and RAG: see ml-superpowers:evaluating-llm-systems for the full mapping.

Common Mistakes

| Mistake | Fix | |---|---| | Running checks without predicting the value first | Write the expected number down. Unpredicted checks catch nothing. | | "Loss started around 2.3, close enough" | Compute -log(1/n) exactly and compare. Close enough hides off-by-ones. | | Skipping overfit-one-batch because the loss is going down | Down is not zero. Only near-zero proves the gradient path is intact. | | Evaluating on a subset for speed | Subset noise exceeds the effects you're trying to measure. Full set. | | Inspecting data in the dataset class, not before the forward pass | Bugs live in collation, moving to device, and dtype casts. Check at the forward. | | Leaving augmentation on "since it helps" | It's a stage 4 regularizer. On now, it's a bug generator. | | Adding all checks after training already ran | 20 minutes now beats a 6-hour run you can't trust. |

Red Flags

• Launching a multi-hour run without having overfit a single batch
• "The loss is decreasing so the pipeline works"
• Reporting accuracy with no baseline of any kind
• Never having printed the actual tensor going into the model
• A number better than your human baseline, treated as good news
• Two runs with the same seed giving different results, left uninvestigated

Next

Gate passed? Proceed to ml-superpowers:overfitting-first.

Checks failing? Use ml-superpowers:debugging-silent-training-failures.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Umaraslam66
• Source: Umaraslam66/ml-superpowers
• License: MIT

Install and usage instructions live in the source repository linked above.