Tuning Hyperparameters

Tuning Hyperparameters

Overview

Stages 5 and 6 of the training recipe. This is the last stage, not the first, and doing it early is how projects burn a GPU budget on a bug.

Core principle: Tuning only pays off on a pipeline you already trust. If you are tuning to make a broken model work, you are searching a space that does not contain the answer.

Prerequisite: ml-superpowers:regularizing-a-model passed. You have a model that overfits, regularization that helps, and an eval you believe.

Stage 5: Tune

Use random search, not grid search

Random search beats grid search for hyperparameter optimization, and the reason is worth internalizing:

Neural nets are far more sensitive to some parameters than others. With a grid over learning rate × weight decay, if learning rate matters and weight decay does not, a 5×5 grid tests only 5 distinct learning rates in 25 runs. Random search over the same budget tests 25 distinct learning rates.

You do not know in advance which parameters are the sensitive ones. Random search does not need you to.

Sample learning rate and weight decay log-uniformly, not uniformly — the meaningful difference is between 1e-4 and 1e-3, not between 0.5 and 0.6.

Bayesian optimization

Toolboxes exist and can help. In practice, careful manual work on the earlier stages beats sophisticated search on a shaky pipeline, and personal experience suggests these tools are hard to beat with when applied to a broad model and hyperparameter space. Use them, but do not expect them to rescue a project.

The tuning discipline

• Change one hyperparameter family per sweep
• Record the full config with every result — an untraceable best run is worthless
• Fix the seed within a sweep so differences come from the parameter, not noise
• Check the best config isn't at the edge of your search range (if it is, your

range was wrong — extend it and re-run)

Stage 6: Squeeze Out The Juice

Once the best model and hyperparameters are found, two final techniques:

Ensembles

Model ensembles are a pretty much guaranteed way to gain ~2% accuracy on almost anything. It is the most reliable free win in the recipe.

If you cannot afford ensembles at inference time, distill the ensemble into a single network.

Leave it training

Often a model is stopped too early. Networks frequently keep improving for a surprisingly long time — leave a run going while you are asleep and check it in the morning.

This is genuinely one of the highest value-per-effort actions available, and it costs no engineering time at all.

Why This Stage Is Last

| If you tune before... | What goes wrong | |---|---| | Looking at the data | You tune around a data bug that a five-minute inspection would have found | | The sanity checks | You search for a learning rate that fixes a detached gradient. None exists. | | Overfitting | You tune a model that can't fit the training set. Every config looks equally mediocre. | | Regularizing | Your best config is best for a model that doesn't generalize. It won't transfer. |

In every one of those cases the sweep completes, reports a winner, and tells you nothing. That is the silent failure mode again, just more expensive.

Modern Mapping

Fine-tuning: The sensitive parameters are learning rate and epoch count. LoRA rank and alpha matter far less than people assume. Sweep LR log-uniformly across two orders of magnitude before touching anything else.

Eval harnesses / prompts: Random search applies to prompt variants too — sample from a space of instruction phrasings rather than hand-tuning one prompt forever. "Ensembling" is self-consistency: sample k times and take the majority.

RAG: The sensitive parameters are chunk size and top-k. Sweep those before adding a reranker.

Common Mistakes

| Mistake | Fix | |---|---| | Grid search | Random search. Same budget, far more distinct values on the axes that matter. | | Sampling LR uniformly | Log-uniform. 1e-4 vs 1e-3 is the real axis. | | Tuning early to fix a bad number | Tuning fixes tuning problems. Go back to stage 2. | | Losing the config for the best run | Log the full config with every result, always. | | Best value sits at the search boundary | Your range was wrong. Extend it and re-run. | | Skipping ensembles as "not worth it" | ~2% for near-zero risk. Distill if inference cost matters. | | Stopping a run because the curve looks flat | Let it run overnight. Flat curves often resume. |

Red Flags

• A sweep launched before the stage 2 sanity checks passed
• Grid search over more than two parameters
• Comparing runs with different seeds and no repeats
• A final number reported from the same split used to pick the config
• Any tuning done to "see if it fixes" an unexplained result

Next

You have finished the recipe. Before claiming a result, use superpowers:verification-before-completion and confirm your final number comes from a split you did not tune against.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Umaraslam66
• Source: Umaraslam66/ml-superpowers
• License: MIT

Install and usage instructions live in the source repository linked above.