Designing Ml Experiments

Designing ML Experiments

Overview

Most experiments answer nothing: they move several things at once, then measure a difference smaller than run-to-run noise.

Core principle: One question per experiment. Classify every hyperparameter before launching, and know your noise floor before calling a difference real.

Source: Google's Deep Learning Tuning Playbook (Godbole, Dahl, Gilmer, Shallue, Nado).

Classify Every Hyperparameter

Three roles. Which one a parameter takes depends on your question, not the parameter.

| Role | Meaning | Handling | |---|---|---| | Scientific | What you're measuring | Vary deliberately | | Nuisance | Must be tuned for a fair comparison | Re-tune within each scientific arm | | Fixed | Held constant to bound scope | Record as a caveat on the conclusion |

The trap: comparing architecture A vs B at one shared learning rate. LR is a nuisance parameter — re-tune it per arm, or you've measured "which architecture happens to like this LR."

> "The more a given nuisance hyperparameter interacts with the scientific > hyperparameters, the more damaging it is to fix its value."

Optimizer hyperparameters (LR, momentum, Adam betas) almost never get fixed.

Exploration Before Exploitation

> "Most of the time, our primary goal is to gain insight into the problem."

Early experiments buy understanding — which knobs matter, where performance saturates, what interacts. Validation error comes later. An experiment that lowers val loss but explains nothing is worth less than one that maps the landscape.

• Exploring → quasi-random search. Uniform coverage, clean per-axis plots,

no bias toward particular regions.

• Exploiting, once the space is narrowed → Bayesian optimization.

Grid search is wrong in both phases.

Know Your Noise Floor

Before adopting anything, characterize variance:

| Source | What varies | |---|---| | Trial | Same config, different seed | | Study | Different hyperparameter draws within a search | | Data | Different train/val/test splits |

Re-run your best config N times with different seeds. If the improvement sits inside that spread, you have not measured an improvement.

Adopt only when the gain outweighs the complexity it adds.

Batch Size Is Not A Quality Knob

> "The batch size should not be treated as a tunable hyperparameter for > validation set performance."

Choose it for throughput — usually the largest your hardware supports. Then re-tune learning rate, momentum, and every regularization hyperparameter, because their optima are batch-size dependent. Changing batch size restarts tuning.

> Tension with the original recipe: ml-superpowers:regularizing-a-model > lists "decrease the batch size" as a regularizer, following Karpathy (2019) — > smaller batches give noisier batch-norm statistics. The Playbook's position is > that this conflates a throughput decision with a quality one. Prefer the > Playbook: fix batch size for throughput and regularize with knobs whose > effect is legible.

Checkpoint Selection

Prefer retrospective selection over prospective early stopping: run a fixed step budget, then pick the best checkpoint from it.

Never tune max_train_steps inside a study — fix it across all trials. Then read where the best step lands:

• First 10% → budget is too large
• Last 25% → train longer and re-tune the decay schedule

Common Mistakes

| Mistake | Fix | |---|---| | One shared LR across compared architectures | LR is a nuisance param — re-tune per arm | | Adopting a change from a single run | Re-run N seeds. Is it outside trial variance? | | Grid search | Quasi-random while exploring; Bayesian after | | Tuning max_train_steps in the sweep | Fix it; select checkpoints retrospectively | | Batch size tuned for accuracy | Tune it for throughput, then re-tune LR and regularization | | Never recording which params were fixed | Fixed params are caveats. Write them down. |

Red Flags

• "This config is better" from n=1
• A comparison where only the scientific parameter was varied
• An experiment you cannot state as a single question
• Best value sitting at a search boundary, left unextended

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Umaraslam66
• Source: Umaraslam66/ml-superpowers
• License: MIT

Install and usage instructions live in the source repository linked above.