# Tuning Hyperparameters > Use when searching over learning rates or other hyperparameters, when setting up a sweep, when deciding between grid search and random search, or when extracting the last few points of performance before shipping a model - **Type:** Skill - **Install:** `agentstack add skill-umaraslam66-ml-superpowers-tuning-hyperparameters` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [Umaraslam66](https://agentstack.voostack.com/s/umaraslam66) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [Umaraslam66](https://github.com/Umaraslam66) - **Source:** https://github.com/Umaraslam66/ml-superpowers/tree/main/skills/tuning-hyperparameters ## Install ```sh agentstack add skill-umaraslam66-ml-superpowers-tuning-hyperparameters ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Tuning Hyperparameters ## Overview Stages 5 and 6 of the training recipe. This is the **last** stage, not the first, and doing it early is how projects burn a GPU budget on a bug. **Core principle:** Tuning only pays off on a pipeline you already trust. If you are tuning to make a broken model work, you are searching a space that does not contain the answer. **Prerequisite:** `ml-superpowers:regularizing-a-model` passed. You have a model that overfits, regularization that helps, and an eval you believe. ## Stage 5: Tune ### Use random search, not grid search Random search beats grid search for hyperparameter optimization, and the reason is worth internalizing: **Neural nets are far more sensitive to some parameters than others.** With a grid over learning rate × weight decay, if learning rate matters and weight decay does not, a 5×5 grid tests only **5 distinct learning rates** in 25 runs. Random search over the same budget tests **25 distinct learning rates**. You do not know in advance which parameters are the sensitive ones. Random search does not need you to. Sample learning rate and weight decay **log-uniformly**, not uniformly — the meaningful difference is between 1e-4 and 1e-3, not between 0.5 and 0.6. ### Bayesian optimization Toolboxes exist and can help. In practice, careful manual work on the earlier stages beats sophisticated search on a shaky pipeline, and personal experience suggests these tools are hard to beat with when applied to a broad model and hyperparameter space. Use them, but do not expect them to rescue a project. ### The tuning discipline - Change **one** hyperparameter family per sweep - Record the full config with every result — an untraceable best run is worthless - Fix the seed within a sweep so differences come from the parameter, not noise - Check the best config isn't at the edge of your search range (if it is, your range was wrong — extend it and re-run) ## Stage 6: Squeeze Out The Juice Once the best model and hyperparameters are found, two final techniques: ### Ensembles Model ensembles are a pretty much guaranteed way to gain **~2% accuracy** on almost anything. It is the most reliable free win in the recipe. If you cannot afford ensembles at inference time, **distill the ensemble into a single network**. ### Leave it training Often a model is stopped too early. Networks frequently keep improving for a surprisingly long time — leave a run going while you are asleep and check it in the morning. This is genuinely one of the highest value-per-effort actions available, and it costs no engineering time at all. ## Why This Stage Is Last | If you tune before... | What goes wrong | |---|---| | Looking at the data | You tune around a data bug that a five-minute inspection would have found | | The sanity checks | You search for a learning rate that fixes a detached gradient. None exists. | | Overfitting | You tune a model that can't fit the training set. Every config looks equally mediocre. | | Regularizing | Your best config is best for a model that doesn't generalize. It won't transfer. | In every one of those cases the sweep *completes*, reports a winner, and tells you nothing. That is the silent failure mode again, just more expensive. ## Modern Mapping **Fine-tuning:** The sensitive parameters are learning rate and epoch count. LoRA rank and alpha matter far less than people assume. Sweep LR log-uniformly across two orders of magnitude before touching anything else. **Eval harnesses / prompts:** Random search applies to prompt variants too — sample from a space of instruction phrasings rather than hand-tuning one prompt forever. "Ensembling" is self-consistency: sample k times and take the majority. **RAG:** The sensitive parameters are chunk size and top-k. Sweep those before adding a reranker. ## Common Mistakes | Mistake | Fix | |---|---| | Grid search | Random search. Same budget, far more distinct values on the axes that matter. | | Sampling LR uniformly | Log-uniform. 1e-4 vs 1e-3 is the real axis. | | Tuning early to fix a bad number | Tuning fixes tuning problems. Go back to stage 2. | | Losing the config for the best run | Log the full config with every result, always. | | Best value sits at the search boundary | Your range was wrong. Extend it and re-run. | | Skipping ensembles as "not worth it" | ~2% for near-zero risk. Distill if inference cost matters. | | Stopping a run because the curve looks flat | Let it run overnight. Flat curves often resume. | ## Red Flags - A sweep launched before the stage 2 sanity checks passed - Grid search over more than two parameters - Comparing runs with different seeds and no repeats - A final number reported from the same split used to pick the config - Any tuning done to "see if it fixes" an unexplained result ## Next You have finished the recipe. Before claiming a result, use `superpowers:verification-before-completion` and confirm your final number comes from a split you did not tune against. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [Umaraslam66](https://github.com/Umaraslam66) - **Source:** [Umaraslam66/ml-superpowers](https://github.com/Umaraslam66/ml-superpowers) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-umaraslam66-ml-superpowers-tuning-hyperparameters - Seller: https://agentstack.voostack.com/s/umaraslam66 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.