Managing Jobs

Managing Jobs

Rule: test small, request explicitly, monitor results, then scale.

Before you submit a job — or recommend one to a human — ask "am I being a good citizen?" and answer it: right partition for the work, requested only what it uses (right-size --mem/--cpus/--time), not stranding idle GPUs with a CPU/RAM reservation, not storming GPFS with tiny files, and cleaning up after. The concrete how-to-answer for each is the [citizenship self-check in overview](../overview/SKILL.md#citizenship-self-check-run-this-before-every-job-and-every-recommendation). When you advise rather than submit, tell the human which checks you ran.

Basic commands

sbatch job.sh                    # submit batch job
squeue -u $USER                  # current jobs
sacct -j 12345                   # completed job accounting
scancel 12345                    # cancel your job
scontrol show job 12345          # detailed job state
sinfo -s                         # partition summary

Common job states: R is running, PD is pending, and CG is completing. If a job stays pending unusually long, inspect the reason instead of just waiting; the request may not fit available nodes, memory, GPUs, partition limits, or time limits.

squeue --me --start
scontrol show job JOBID

Choosing a partition

default_queue is the Slurm default, but it is a short, small queue (1h default / 4h max, only 6 nodes) and those nodes are shared with — and outranked by — the restricted mbacourse partition. So a default_queue job can pend with "Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions" even when the cluster is mostly idle, because the idle cores are on partitions it doesn't target.

Before assuming "the cluster is full," look:

squeue --me -o "%.10i %.9P %.2t %.10M %r"   # %r = pending reason — read it
sinfo -o "%R %C"                            # idle cores per partition (A/I/O/T)

Pick the partition that fits the work:

• Quick test jobs (≤4h): default_queue is fine.
• Real / long / large CPU work: cpunormal or gpunormal — these are the "normal" production queues with no time limit. gpunormal is the largest pool and, despite the name, takes CPU-only jobs (just omit --gres). When default_queue/cpunormal show 0 idle cores, a CPU job submitted to gpunormal typically starts immediately.
• GPU work: gpunormal (--gres=gpu:1), or h100 for H100s. See [using GPUs](../using-gpus/SKILL.md).

Running CPU work on gpunormal without stranding GPUs

Most of the cluster's CPU cores live on the GPU nodes (the CPU-only partitions are just a handful of nodes), so substantial CPU work often has to run on gpunormal. The hazard: Slurm reserves the CPU and RAM you request whether or not you use them, and a GPU job needs CPU + RAM alongside its GPU. If your CPU-only job doesn't leave a GPU job's worth of headroom per still-idle GPU on that node, those GPUs become unschedulable — scarce hardware sits idle.

This happens for real: a CPU-only job reserving ~900 GB on a 1 TB, 3-GPU node leaves only ~120 GB — room for one GPU job, stranding the other two GPUs, even though nvidia-smi shows them idle and the OS shows the RAM physically free. Reserved-but-unused is just as blocking as used.

So when you must run CPU-only work on gpunormal:

• Right-size --mem and --cpus-per-task from seff — never pad "just in case." This is the single biggest cause of accidental stranding.
• Leave a GPU's share free. A GPU job here typically needs roughly 8 CPUs and ~120 GB per GPU (check live with squeue -p gpunormal -t R -O NumCPUs,MinMemory,tres-per-node). On the 3-GPU nodes, a rule of thumb is to keep a CPU-only job under about one-third of a node's CPU and RAM so the other GPUs stay usable.
• Keep big CPU jobs off the scarcest GPU nodes — prefer cpunormal/default_queue, or the RTX 8000 / 40 GB A100 nodes, over the 80 GB A100 and H100 nodes.
• Check what you'd be sitting next to: sinfo -N -p gpunormal -O NodeHost,CPUsState,FreeMem,Gres,GresUsed shows nodes with idle GPUs (GresUsed slurm/manifest.txt

n=$(wc -l < slurm/manifest.txt) echo "manifest has $n tasks; submit with --array=1-$n%50"

```bash
# In the array job script:
input=$(sed -n "${SLURM_ARRAY_TASK_ID}p" slurm/manifest.txt)
srun .venv/bin/python src/run_task.py --input "$input"

Commit slurm/manifest.txt so reruns and collaborators see the same task → input mapping.

Per-job temp directory

For high-I/O work, stage onto compute-node /tmp and copy results back. The recipe lives in [using the filesystem](../using-the-filesystem/SKILL.md). The short version: mktemp -d "${TMPDIR:-/tmp}/job_${SLURM_JOB_ID:-local}.XXXXXX", set TMPDIR, trap to clean.

Dependencies

prep=$(sbatch --parsable slurm/01_prepare.sh)
est=$(sbatch --parsable --dependency=afterok:${prep} slurm/02_estimate.sh)
sbatch --dependency=afterok:${est} slurm/03_tables.sh

Use afterany for cleanup or restart logic that should run even after failure.

Time limits

Shorter jobs often schedule faster because Slurm can backfill them into idle slots between bigger jobs. Multi-day jobs queue behind everyone. If work is resumable, prefer 2–4 hour chunks; with skip-if-exists outputs, a killed job picks up where it left off on resubmit.

Right-size before submitting

Do not pad requests "just in case." Over-requesting blocks scheduling for everyone else on a shared cluster (there are no per-user caps — it runs on courtesy). The right-sizing loop:

1. Submit a 10-minute test job with a small input.
2. Run seff JOBID after it finishes.
3. Set the real job's --mem to ~1.5–2× the test's MaxRSS, not 10×.
4. Set --cpus-per-task to what your code actually parallelizes over (SLURM_CPUS_PER_TASK controls BLAS, multiprocessing, setDTthreads, set processors).
5. Set --time from a sample-data extrapolation, not from "what if it takes a week."

See [self-diagnosing resource use](../self-diagnosing-resource-use/SKILL.md) for the post-job checks that drive this loop.

Before scaling up

sbatch slurm/test.sh
squeue -u $USER
sacct -j JOBID --format=JobID,Elapsed,MaxRSS,AllocCPUS,TotalCPU,State

Report back without being asked

When a real job finishes, do not stop at "it ran." Proactively run seff JOBID and tell the user, in plain language, whether the job used what it asked for — e.g. "it used 6 GB of the 64 GB requested and 1 of 4 CPUs, so next time request --mem=12G --cpus-per-task=1." Most researchers will not think to ask; surfacing waste is part of the job, not an extra. See [self-diagnosing resource use](../self-diagnosing-resource-use/SKILL.md).

Checklist

• [ ] Job starts with a small test.
• [ ] --time, --mem, and --cpus-per-task are explicit.
• [ ] Thread variables are set with ${SLURM_CPUS_PER_TASK:-1}; PYTHONUNBUFFERED=1 for Python jobs.
• [ ] Long jobs launch with srun .venv/bin/python ... (or srun Rscript ...), not uv run python ....
• [ ] Arrays use a concurrency throttle like %50 and index into a stable manifest, not glob order.
• [ ] Output paths include job IDs or task IDs.
• [ ] Job script is LF-terminated (no CRLF) so sbatch does not fail with $'\r'.
• [ ] Resource usage is checked after completion.

Further reading

• Slurm sbatch reference — every #SBATCH directive, output filename patterns, signal handling.
• Slurm job arrays — --array syntax, throttling (%N), SLURM_ARRAY_* env vars.
• Slurm squeue and sacct — format strings, state codes.
• Slurm quickstart — sbatch/srun/sacct big picture.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yale-som-hpc
• Source: yale-som-hpc/claude-code-marketplace
• License: Unlicense

Install and usage instructions live in the source repository linked above.