# Building The Training Skeleton > Use when about to report any accuracy loss or metric to anyone, when running or writing a training or evaluation script, when asked for a number from a script you did not write, before launching a real training run, or when a model trains without error but results are unexplained - **Type:** Skill - **Install:** `agentstack add skill-umaraslam66-ml-superpowers-building-the-training-skeleton` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [Umaraslam66](https://agentstack.voostack.com/s/umaraslam66) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [Umaraslam66](https://github.com/Umaraslam66) - **Source:** https://github.com/Umaraslam66/ml-superpowers/tree/main/skills/building-the-training-skeleton ## Install ```sh agentstack add skill-umaraslam66-ml-superpowers-building-the-training-skeleton ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Building The Training Skeleton ## STOP — Run The Auditor Before Reporting Any Number **Before you report an accuracy, loss, or any metric — including from a script you did not write and were only asked to run — run this first. One command, under a second:** ```bash python3 scripts/audit-run.py ``` The script is at `../../scripts/audit-run.py` relative to this skill file. It exits non-zero and prints findings if it detects: | Finding | What it means | |---|---| | `LEAK` | Train and test slice the same array with overlapping ranges. The metric is measuring memorization. | | `GROUP` | Grouped data (patient, user, session, document) split randomly. The same entity lands on both sides. | | `TIME` | Data with a time column split randomly. You are training on the future and testing on the past. | | `PREP` | A scaler/encoder was fit before the split. Test-set statistics leaked into training. | | `BASE` | No majority-class baseline computed. An accuracy with nothing to compare against is not interpretable. | | `DATA` | The loader fabricates data with a random generator. No metric from it describes the real task. | **If the auditor reports anything, do not report the number.** Say what it found instead. "The harness runs, the split is broken, here is the corrected figure" is a useful answer; a wrong number someone puts in a deck is not. Time pressure makes this more important, not less — a number that gets walked back costs more than a number delivered late. ### Then check loss at init by hand The auditor cannot see runtime values. Compare the first printed loss against `-log(1/n_classes)`: ``` 10 classes → 2.303 1000 → 6.908 binary → 0.693 ``` Far **below** means a label is leaking into the input. Far **above** means the final-layer init is wrong. ## Overview Stage 2 of the training recipe. This is the stage that catches silent bugs, and it is the stage that gets skipped. **Do not skip it. It takes about 20 minutes and it is the difference between engineering and gambling.** Set up a full training and evaluation skeleton with a model so dumb it cannot possibly be wrong, and gain trust in its correctness via a series of experiments where **you predict the outcome before you run it**. **Core principle:** A check is only worth running if you write down the expected value first. "Let's see what happens" catches nothing. "This should print 2.303, and if it doesn't, my label mapping is broken" catches everything. ## The Gate You may proceed to stage 3 when **all** of these hold: - [ ] Loss at initialization equals the value you predicted - [ ] Input-independent baseline performs strictly worse than the real model - [ ] A single batch of ~2-10 examples overfits to near-zero loss - [ ] You have a human baseline (or a stated ceiling) to compare against - [ ] You have visually inspected the tensor entering the model - [ ] Evaluation runs on the **entire** test set, not a subset - [ ] The seed is fixed and two runs produce identical numbers Any unchecked box means you do not yet know your pipeline is correct. ## The Setup Rules **Fix the random seed.** Always. Removes an entire class of "why is it different now?" confusion. **Simplify.** Turn off every optional feature — augmentation, mixed precision, EMA, LR schedules, fancy samplers. Augmentation is a stage 4 regularizer; right now it is a bug generator in disguise. **Add significant digits.** Evaluate on the **entire** test set and print exact numbers. A full-set mean is comparable across runs; an eyeballed running average is not. **Generalize a special case.** Write the specific function you need now, verify it, then generalize while asserting the result is unchanged. This matters most for vectorized code: get the single-example loop right *before* batching it. ## The Sanity Checks Each check states what it catches. Run them in order. ### 1. Verify loss @ init Verify your loss starts at the correct value. For a softmax over `n` classes at proper initialization, that is `-log(1/n_classes)`. ``` 10 classes → 2.303 1000 classes → 6.908 binary → 0.693 ``` **Catches:** wrong number of classes, wrong reduction, double-softmax (passing already-softmaxed logits to cross-entropy), broken initialization, label off-by-one, logits and labels transposed. **If it starts far below the expected value, you are leaking the label.** ### 2. Init well Initialize the final layer weights correctly. If you are regressing values with mean 50, initialize the final bias to 50. For imbalanced classes at ratio 1:10, set the bias on the logits so the initial prediction reflects that ratio. **Catches:** nothing directly — but it removes a "hockey stick" loss curve where the first few hundred steps are spent learning the bias, which otherwise masks whether your model is learning anything real. ### 3. Human baseline Do the task yourself. Record your accuracy. If you cannot do it, that is the finding — either the task is underspecified or the inputs lack the information. **Catches:** impossible tasks, missing features, and the "is 71% good?" question that otherwise haunts the whole project. ### 4. Input-independent baseline Train a baseline with the input zeroed out (or shuffled). Compare it to the real model on identical everything else. **The real model must beat it.** If it does not, **your model is not extracting any information from the input.** This is the check that catches the most embarrassing bugs. **Catches:** inputs never reaching the model, a detached graph, features that are all zeros/NaN after normalization, a model that learned only the class prior. ### 5. Overfit one batch Take a single batch of just a few examples — as few as two — and overfit it. Increase capacity if needed. **Drive the loss to essentially zero.** Then visualize the label and prediction side by side and confirm they line up exactly at minimal loss. If they don't, there is a bug and you cannot proceed. **Catches:** almost every remaining silent bug. Missing `optimizer.zero_grad()` or `loss.backward()`, a frozen or detached parameter, an inverted mask, targets misaligned with predictions, a loss that ignores its inputs, gradients not flowing through a custom layer. **This is the single most valuable check in the recipe.** A model that cannot memorize two examples has a bug, full stop. It is never a capacity problem, a learning rate problem, or a data problem. ### 6. Verify decreasing training loss Now train on the full dataset with a deliberately small model. Then make it bigger. **The bigger model must reach a lower training loss.** **Catches:** a capacity ceiling you didn't know about, a bottleneck layer, a regularizer you forgot you left on. ### 7. Visualize just before the net The unambiguously correct place to visualize your data is immediately before `y_hat = model(x)`. Print or render **exactly that tensor**. Decode the tokens back to text. Save the image tensor to a PNG. Look at it. **Catches:** wrong normalization, channels transposed, an off-by-one crop, the chat template not applied, padding on the wrong side, an inverted attention mask, augmentation accidentally left on at eval. This is the single source of truth about what your model actually consumes, and it disagrees with what you think more often than you would believe. ### 8. Visualize prediction dynamics Visualize predictions on a **fixed** test batch over the course of training. Watching how they move tells you far more than a loss number. Wild oscillation means the learning rate is too high. Frozen predictions mean it is too low or gradients aren't flowing. A model that confidently commits early and never revises is often overfitting or has a leaked feature. ### 9. Use backprop to chart dependencies Vectorized code makes it easy to accidentally mix information across the batch or time dimension — and it trains fine when you do. To check: set the loss to something trivial like the sum of outputs for example `i` only, run backward, and confirm you get a **non-zero gradient only on example `i`**. **Catches:** batch dimension mixing (`view` vs `transpose` errors), causal attention masks that leak the future, a rolling window that peeks ahead. These bugs produce *better-looking* training curves, which is why they survive. ## Modern Mapping **Fine-tuning:** Loss @ init should match the base model's loss on the same data; if not, your template or tokenization differs from pretraining. Overfit-one-batch on 10 examples verifies the loss mask covers the right tokens. Input-independent = train on shuffled responses; must get worse. **Eval harnesses and RAG:** see `ml-superpowers:evaluating-llm-systems` for the full mapping. ## Common Mistakes | Mistake | Fix | |---|---| | Running checks without predicting the value first | Write the expected number down. Unpredicted checks catch nothing. | | "Loss started around 2.3, close enough" | Compute `-log(1/n)` exactly and compare. Close enough hides off-by-ones. | | Skipping overfit-one-batch because the loss is going down | Down is not zero. Only near-zero proves the gradient path is intact. | | Evaluating on a subset for speed | Subset noise exceeds the effects you're trying to measure. Full set. | | Inspecting data in the dataset class, not before the forward pass | Bugs live in collation, moving to device, and dtype casts. Check at the forward. | | Leaving augmentation on "since it helps" | It's a stage 4 regularizer. On now, it's a bug generator. | | Adding all checks after training already ran | 20 minutes now beats a 6-hour run you can't trust. | ## Red Flags - Launching a multi-hour run without having overfit a single batch - "The loss is decreasing so the pipeline works" - Reporting accuracy with no baseline of any kind - Never having printed the actual tensor going into the model - A number better than your human baseline, treated as good news - Two runs with the same seed giving different results, left uninvestigated ## Next Gate passed? Proceed to `ml-superpowers:overfitting-first`. Checks failing? Use `ml-superpowers:debugging-silent-training-failures`. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [Umaraslam66](https://github.com/Umaraslam66) - **Source:** [Umaraslam66/ml-superpowers](https://github.com/Umaraslam66/ml-superpowers) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-umaraslam66-ml-superpowers-building-the-training-skeleton - Seller: https://agentstack.voostack.com/s/umaraslam66 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.