Using Ml Superpowers

Using ML Superpowers

If you were dispatched as a subagent to execute a specific narrow task, ignore this skill.

The Rule

If the task involves a model, a loss, or a metric — invoke the matching skill before writing code. Announce "Using [skill] to [purpose]" and follow it.

ML code fails silently. An agent that writes a training loop without these skills produces something that runs, converges, reports a number, and is wrong in a way neither of you will notice for days.

Routing

Training a model — the recipe, in stage order:

| The situation | Invoke | |---|---| | Any training or fine-tuning work — start here | ml-superpowers:training-neural-networks | | Starting out; unfamiliar dataset; suspicious results | ml-superpowers:becoming-one-with-the-data | | Writing a training loop or eval harness | ml-superpowers:building-the-training-skeleton | | Choosing architecture/size; tempted by dropout | ml-superpowers:overfitting-first | | Fits train, fails val; choosing a regularizer | ml-superpowers:regularizing-a-model | | Running a sweep; final performance push | ml-superpowers:tuning-hyperparameters | | Runs fine, results are wrong or implausible | ml-superpowers:debugging-silent-training-failures |

Everything around the training loop:

| The situation | Invoke | |---|---| | Comparing variants; is this difference real? | ml-superpowers:designing-ml-experiments | | What should I work on next? Splits, error analysis | ml-superpowers:choosing-what-to-fix | | Going to production; offline good, live bad | ml-superpowers:shipping-ml-systems | | Evals for an LLM feature, agent, RAG, or judge | ml-superpowers:evaluating-llm-systems | | Inducing a rule, lesson, or diagnosis from few examples | ml-superpowers:weakest-hypothesis |

When in doubt on model work, start with ml-superpowers:training-neural-networks — it holds the stage order and routes onward.

The Two Things To Remember

1. The recipe is a sequence of gates, and the order is load-bearing:

look at data → dumb baselines → overfit → regularize → tune → squeeze

Regularizing before overfitting is the mistake that costs the most. It makes underfitting and overfitting indistinguishable, and every later decision becomes a guess.

2. Diagnose before acting. Whether it's a training run, an experiment, or an LLM pipeline: measure which failure you have before applying a fix, because the fixes for adjacent failures are usually opposites.

Red Flags

These thoughts mean stop — you are rationalizing:

| Thought | Reality | |---|---| | "It's a standard task" | Standard tasks have standard silent bugs. | | "The loss is going down" | Loss goes down for models with shuffled labels too. | | "I'll sanity check after it trains" | Checks cost 20 minutes. A wasted run costs hours. | | "It's just a fine-tune" | Fine-tunes have more silent failure modes, not fewer. | | "Let me add dropout to improve it" | Not before it overfits. You're masking a bug. | | "94% accuracy, done" | Versus what baseline? Suspicious numbers are usually leakage. | | "I'll tune the LR and see" | "See" is guessing. Predict the outcome first. |

Interop

These skills compose with obra/superpowers if installed — superpowers:systematic-debugging for non-ML bugs, superpowers:verification-before-completion before claiming any result. Neither is required.

Attribution

• Training recipe — Andrej Karpathy,

"A Recipe for Training Neural Networks" (2019)

• Experiment design — Google's

Deep Learning Tuning Playbook

• Prioritization — Andrew Ng, Machine Learning Yearning
• Production systems — Martin Zinkevich,

Rules of Machine Learning

• LLM evals — Hamel Husain, Shreya Shankar

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Umaraslam66
• Source: Umaraslam66/ml-superpowers
• License: MIT

Install and usage instructions live in the source repository linked above.