Overfitting First
Use when choosing a model architecture or size, when tempted to add dropout or augmentation or weight decay, when training and validation loss are both bad, or when unsure whether a model is underfitting or overfitting
Building The Training Skeleton
Use when about to report any accuracy loss or metric to anyone, when running or writing a training or evaluation script, when asked for a number from a script you did not write, before launching a real training run, or when a model trains without error but results are unexplained
Debugging Silent Training Failures
Use when a model trains without errors but produces bad or implausible results, when loss is NaN or flat or oscillating, when accuracy is suspiciously high, when validation and training curves look wrong, or when eval numbers do not reproduce
Weakest Hypothesis
Use when inducing a rule, root cause, or lesson from few examples — reflection steps in self-improvement loops, encoding learnings into memory or prompts, diagnosing from a handful of failure reports, or whenever several explanations fit the evidence and one must be adopted
Designing Ml Experiments
Use when comparing model variants or configs, when a change appears to help but you are unsure the difference is real, when setting up a sweep, or when deciding whether to adopt a change
Tuning Hyperparameters
Use when searching over learning rates or other hyperparameters, when setting up a sweep, when deciding between grid search and random search, or when extracting the last few points of performance before shipping a model
Shipping Ml Systems
Use when putting a model into production, when deciding whether a problem needs ML at all, when offline metrics look good but production results do not match, or when building serving and training pipelines
Training Neural Networks
Use when training, fine-tuning, or evaluating any neural network or ML model, before writing a training loop, when a model trains but underperforms, or when a metric looks suspiciously good or suspiciously bad
Choosing What To Fix
Use when deciding what to work on next on an ML project, when a model is underperforming and the cause is unclear, when setting up train dev and test splits, or when tempted to collect more data or build a bigger model
Evaluating Llm Systems
Use when building evals for an LLM feature or agent, when setting up an LLM-as-judge, when prompt changes cannot be verified as improvements, or when a RAG or agent pipeline works on examples but fails in production
Becoming One With The Data
Use when starting any ML task, before writing a model or training loop, when inheriting an unfamiliar dataset, when eval results are surprising in either direction, or when suspecting label noise, duplicates, leakage, or contamination
Regularizing A Model
Use when a model fits the training set but generalizes poorly, when validation loss rises while training loss falls, when deciding between dropout weight decay augmentation or more data, or when choosing when to stop training
Using Ml Superpowers
Use when starting any machine learning, model training, fine-tuning, or evaluation task - establishes which ML skill to invoke before writing model code