# Managing Jobs

> Submit, monitor, cancel, array, and chain Slurm jobs on the Yale SOM HPC cluster. TRIGGER when running, submitting, or scheduling any job or analysis on the cluster (even phrased plainly, e.g. "run my regression on the cluster"), writing sbatch scripts, choosing partitions/resources, or using job arrays, dependencies, or sacct/squeue/scancel.

- **Type:** Skill
- **Install:** `agentstack add skill-yale-som-hpc-claude-code-marketplace-managing-jobs`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [yale-som-hpc](https://agentstack.voostack.com/s/yale-som-hpc)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Unlicense
- **Upstream author:** [yale-som-hpc](https://github.com/yale-som-hpc)
- **Source:** https://github.com/yale-som-hpc/claude-code-marketplace/tree/main/plugins/hpc/skills/managing-jobs

## Install

```sh
agentstack add skill-yale-som-hpc-claude-code-marketplace-managing-jobs
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Managing Jobs

Rule: test small, request explicitly, monitor results, then scale.

Before you submit a job — or recommend one to a human — ask **"am I being a good citizen?"** and answer it: right partition for the work, requested only what it uses (right-size `--mem`/`--cpus`/`--time`), not stranding idle GPUs with a CPU/RAM reservation, not storming GPFS with tiny files, and cleaning up after. The concrete how-to-answer for each is the [citizenship self-check in overview](../overview/SKILL.md#citizenship-self-check-run-this-before-every-job-and-every-recommendation). When you advise rather than submit, tell the human which checks you ran.

## Basic commands

```bash
sbatch job.sh                    # submit batch job
squeue -u $USER                  # current jobs
sacct -j 12345                   # completed job accounting
scancel 12345                    # cancel your job
scontrol show job 12345          # detailed job state
sinfo -s                         # partition summary
```

Common job states: `R` is running, `PD` is pending, and `CG` is completing. If a job stays pending unusually long, inspect the reason instead of just waiting; the request may not fit available nodes, memory, GPUs, partition limits, or time limits.

```bash
squeue --me --start
scontrol show job JOBID
```

## Choosing a partition

`default_queue` is the Slurm default, but it is a *short, small* queue (1h default / 4h max, only 6 nodes) and those nodes are shared with — and outranked by — the restricted `mbacourse` partition. So a `default_queue` job can pend with *"Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions"* **even when the cluster is mostly idle**, because the idle cores are on partitions it doesn't target.

Before assuming "the cluster is full," look:

```bash
squeue --me -o "%.10i %.9P %.2t %.10M %r"   # %r = pending reason — read it
sinfo -o "%R %C"                            # idle cores per partition (A/I/O/T)
```

Pick the partition that fits the work:

- **Quick test jobs (≤4h):** `default_queue` is fine.
- **Real / long / large CPU work:** `cpunormal` or `gpunormal` — these are the "normal" production queues with **no time limit**. `gpunormal` is the largest pool and, despite the name, takes CPU-only jobs (just omit `--gres`). When `default_queue`/`cpunormal` show 0 idle cores, a CPU job submitted to `gpunormal` typically starts immediately.
- **GPU work:** `gpunormal` (`--gres=gpu:1`), or `h100` for H100s. See [using GPUs](../using-gpus/SKILL.md).

### Running CPU work on `gpunormal` without stranding GPUs

Most of the cluster's CPU cores live on the GPU nodes (the CPU-only partitions are just a handful of nodes), so substantial CPU work often *has* to run on `gpunormal`. The hazard: **Slurm reserves the CPU and RAM you request whether or not you use them**, and a GPU job needs CPU + RAM *alongside* its GPU. If your CPU-only job doesn't leave a GPU job's worth of headroom per still-idle GPU on that node, those GPUs become unschedulable — scarce hardware sits idle.

This happens for real: a CPU-only job reserving ~900 GB on a 1 TB, 3-GPU node leaves only ~120 GB — room for one GPU job, stranding the other two GPUs, even though `nvidia-smi` shows them idle and the OS shows the RAM physically free. Reserved-but-unused is just as blocking as used.

So when you must run CPU-only work on `gpunormal`:

- **Right-size `--mem` and `--cpus-per-task` from `seff` — never pad "just in case."** This is the single biggest cause of accidental stranding.
- **Leave a GPU's share free.** A GPU job here typically needs roughly **8 CPUs and ~120 GB per GPU** (check live with `squeue -p gpunormal -t R -O NumCPUs,MinMemory,tres-per-node`). On the 3-GPU nodes, a rule of thumb is to keep a CPU-only job under about **one-third** of a node's CPU and RAM so the other GPUs stay usable.
- **Keep big CPU jobs off the scarcest GPU nodes** — prefer `cpunormal`/`default_queue`, or the RTX 8000 / 40 GB A100 nodes, over the 80 GB A100 and H100 nodes.
- **Check what you'd be sitting next to:** `sinfo -N -p gpunormal -O NodeHost,CPUsState,FreeMem,Gres,GresUsed` shows nodes with idle GPUs (`GresUsed`  slurm/manifest.txt
n=$(wc -l < slurm/manifest.txt)
echo "manifest has $n tasks; submit with --array=1-$n%50"
```

```bash
# In the array job script:
input=$(sed -n "${SLURM_ARRAY_TASK_ID}p" slurm/manifest.txt)
srun .venv/bin/python src/run_task.py --input "$input"
```

Commit `slurm/manifest.txt` so reruns and collaborators see the same task → input mapping.

### Per-job temp directory

For high-I/O work, stage onto compute-node `/tmp` and copy results back. The recipe lives in [using the filesystem](../using-the-filesystem/SKILL.md). The short version: `mktemp -d "${TMPDIR:-/tmp}/job_${SLURM_JOB_ID:-local}.XXXXXX"`, set `TMPDIR`, `trap` to clean.

## Dependencies

```bash
prep=$(sbatch --parsable slurm/01_prepare.sh)
est=$(sbatch --parsable --dependency=afterok:${prep} slurm/02_estimate.sh)
sbatch --dependency=afterok:${est} slurm/03_tables.sh
```

Use `afterany` for cleanup or restart logic that should run even after failure.

## Time limits

Shorter jobs often schedule faster because Slurm can backfill them into idle slots between bigger jobs. Multi-day jobs queue behind everyone. If work is resumable, prefer 2–4 hour chunks; with skip-if-exists outputs, a killed job picks up where it left off on resubmit.

## Right-size before submitting

Do not pad requests "just in case." Over-requesting blocks scheduling for everyone else on a shared cluster (there are no per-user caps — it runs on courtesy). The right-sizing loop:

1. Submit a 10-minute test job with a small input.
2. Run `seff JOBID` after it finishes.
3. Set the real job's `--mem` to ~1.5–2× the test's `MaxRSS`, not 10×.
4. Set `--cpus-per-task` to what your code actually parallelizes over (`SLURM_CPUS_PER_TASK` controls BLAS, multiprocessing, `setDTthreads`, `set processors`).
5. Set `--time` from a sample-data extrapolation, not from "what if it takes a week."

See [self-diagnosing resource use](../self-diagnosing-resource-use/SKILL.md) for the post-job checks that drive this loop.

## Before scaling up

```bash
sbatch slurm/test.sh
squeue -u $USER
sacct -j JOBID --format=JobID,Elapsed,MaxRSS,AllocCPUS,TotalCPU,State
```

## Report back without being asked

When a real job finishes, do not stop at "it ran." Proactively run `seff JOBID` and tell the user, in plain language, whether the job used what it asked for — e.g. "it used 6 GB of the 64 GB requested and 1 of 4 CPUs, so next time request `--mem=12G --cpus-per-task=1`." Most researchers will not think to ask; surfacing waste is part of the job, not an extra. See [self-diagnosing resource use](../self-diagnosing-resource-use/SKILL.md).

## Checklist

- [ ] Job starts with a small test.
- [ ] `--time`, `--mem`, and `--cpus-per-task` are explicit.
- [ ] Thread variables are set with `${SLURM_CPUS_PER_TASK:-1}`; `PYTHONUNBUFFERED=1` for Python jobs.
- [ ] Long jobs launch with `srun .venv/bin/python ...` (or `srun Rscript ...`), not `uv run python ...`.
- [ ] Arrays use a concurrency throttle like `%50` and index into a stable manifest, not glob order.
- [ ] Output paths include job IDs or task IDs.
- [ ] Job script is LF-terminated (no CRLF) so sbatch does not fail with `$'\r'`.
- [ ] Resource usage is checked after completion.

## Further reading

- [Slurm sbatch reference](https://slurm.schedmd.com/sbatch.html) — every `#SBATCH` directive, output filename patterns, signal handling.
- [Slurm job arrays](https://slurm.schedmd.com/job_array.html) — `--array` syntax, throttling (`%N`), `SLURM_ARRAY_*` env vars.
- [Slurm squeue](https://slurm.schedmd.com/squeue.html) and [sacct](https://slurm.schedmd.com/sacct.html) — format strings, state codes.
- [Slurm quickstart](https://slurm.schedmd.com/quickstart.html) — sbatch/srun/sacct big picture.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [yale-som-hpc](https://github.com/yale-som-hpc)
- **Source:** [yale-som-hpc/claude-code-marketplace](https://github.com/yale-som-hpc/claude-code-marketplace)
- **License:** Unlicense

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-yale-som-hpc-claude-code-marketplace-managing-jobs
- Seller: https://agentstack.voostack.com/s/yale-som-hpc
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.