Training Loop Debugger
Use when reviewing ML training code, suspicious metrics, unstable loss, broken evaluation, or a model that appears too good to be true.
Install
$ agentstack add skill-maximussthegreat-ml-researcher-os-training-loop-debugger Open-source listing — not yet scanned by AgentStack. Follow the source repository for install instructions.
Security review
⚠ Flagged1 finding(s); flagged for manual review. · v0.1.0 How review works →
- • Prompt-injection patterns
- • Secret / credential exfiltration
- • Dangerous shell & filesystem operations
- • Untrusted network calls
- • Known-malicious package signatures
- high Dangerous shell/eval execution.
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ● Dynamic code execution Used
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
About
Training Loop Debugger
Use this skill to review training and evaluation code before trusting results.
Goal
Find the smallest code or protocol issue that could invalidate reported model metrics.
Review order
- Data loading
- Split creation
- Preprocessing fit and transform
- Model initialization
- Loss and optimizer
- Training mode and eval mode
- Metric computation
- Checkpoint selection
- Test set use
- Logging and reproducibility
High-risk bugs
Immediately flag:
- fitting scalers, tokenizers, imputers, or feature selectors on validation or test data
- using test data for early stopping
- computing metrics on logits when probabilities or labels are required
- forgetting
model.eval()during evaluation - forgetting
torch.no_grad()during evaluation - using different preprocessing at train and eval time
- reporting the best validation run as test performance
- changing seeds until a good result appears
Required output
| Area | Finding | Severity | Evidence | Fix | | --- | --- | --- | --- | --- |
Severity must be one of:
blockerhighmediumlow
Rules
- Do not claim the model is fixed until a rerun is available.
- Do not rewrite the whole project when a small patch explains the issue.
- If logs are missing, ask for logs or mark the conclusion as provisional.
Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- Author: maximussthegreat
- Source: maximussthegreat/ml-researcher-os
- License: MIT
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet — be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.