Hf Jobs

Discipline for HF cloud compute. The hf-jobs executable is the only submission path; a PreToolUse guard prices every run and a background monitor reports state changes and failures back to you, so submitted jobs never need manual polling loops.

Authenticated namespace: !"${CLAUDE_PLUGIN_ROOT}/bin/hf-whoami" --brief

Current hardware catalog (USD per hour):

```! "${CLAUDEPLUGINROOT}/bin/hf-jobs" hardware 2>/dev/null || echo '{"ok": false, "error": "catalog unavailable, run hf-jobs hardware manually"}'

## Submission contract

`hf-jobs run` refuses to submit unless all of these hold, so do them in order:

1. `--timeout` is explicit. Set it from model size, minimum 2h for any training. The 30 minute default has killed real training runs; that is why the flag is mandatory.
2. `--confirm-cost` matches or exceeds the estimate for billable hardware. Get the number first: `hf-jobs estimate --flavor a10g-large --timeout 4h`.
3. The script contains no placeholder sentinels (``, ``, and friends). Resolve the real namespace before writing the script.

Never pass a local machine path as the job script reference. Jobs run in a fresh cloud environment where local files do not exist. Submit inline source (`--inline` reads stdin), a `--script` whose content is uploaded, or a public `--url`.

For a job that needs a specific container rather than a UV script, swap the source for `--image  --command ""`; the same `--timeout` and `--confirm-cost` rules still apply, and it works for immediate and scheduled runs alike.

## Exact-source workflow

For any non-trivial job script:
1. Write the script locally with Write.
2. Validate syntax and imports.
3. Smoke test the exact script: one training step plus one eval step when evaluation is part of the workflow. GPU code paths (CUDA, bf16/fp16, quantization, flash attention kernels, torch.compile, model loading) need a GPU smoke test: `hf-sandbox create --hardware t4-small` minimum, Ampere or newer (never T4) when a flash-attention kernel is involved. GPU sandboxes are billable, so `hf-sandbox create` refuses billable hardware without `--confirm-cost ` at or above the estimate, exactly like `hf-jobs run`; free `cpu-basic` needs no confirmation. CPU sandboxes cannot test GPU paths.
4. Submit the exact tested source. Do not reconstruct a similar script from memory.

If you skip the GPU smoke test, state why before running the job.

Every training script fails fast before expensive work:
- print package versions for torch, transformers, trl, accelerate, datasets, trackio, and kernels when used
- assert required dataset columns exist
- assert `push_to_hub=True` and `hub_model_id` is set with no placeholder
- include every imported third-party package in `--dep`, plus the core stack: torch, transformers, trl, accelerate, datasets, trackio, and `kernels~=0.12.0` when using Hub kernels

## Preflight, stated before every run

- Reference implementation: [which example this is based on]
- Dataset format verified: [columns confirmed via hf-dataset-inspect]
- GPU sandbox smoke test: [hardware and result, or why not applicable]
- push_to_hub and hub_model_id: [set, no placeholders]
- timeout: [value] based on [model size] on [hardware]
- estimated cost: [from hf-jobs estimate]
- trackio: [space id, public dashboard]

Cannot fill an item: stop and complete it first.

## Hardware sizing

- 1-3B params: a10g-largex2
- 7-13B params: a100-large
- 30B+ params: l40sx4 or a100x4
- 70B+ params: a100x8

a10g-small and a10g-large have the same 24GB GPU memory; the difference is CPU and RAM only.

## Batches, scheduled jobs, failures

Ablations and sweeps: submit ONE job, watch the monitor confirm it reaches training, then submit the rest. Prefer one sweep script over ten manual submissions.

Scheduled jobs always require human approval; their recurring cost is unbounded. Auto permission modes refuse them by design; ask the user to approve in default mode.

On failure the monitor prints the log tail. Diagnose from `hf-jobs logs `, fix the actual cause, and follow the OOM ladder from /ml-intern rather than shrinking the task. Cancelling: `hf-jobs cancel `; the ledger reconciles actual cost when the terminal state lands.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [boeschj](https://github.com/boeschj)
- **Source:** [boeschj/ml-intern-plugin](https://github.com/boeschj/ml-intern-plugin)
- **License:** Apache-2.0
- **Homepage:** https://github.com/boeschj/ml-intern-plugin#quickstart

Install and usage instructions live in the source repository linked above.