Slurm

Slurm

Overview

Slurm (sbatch/squeue/sacct/scontrol) schedules jobs on shared HPC and GPU clusters. Core principle: describe the smallest resource box your job fits in (time, cpus, mem, gpus), throttle your own fan-out so you stay a good neighbor, and infer outcomes from logs/accounting instead of babysitting.

This skill is cluster-agnostic. For a specific site (account format, partition names, GPU models) read that site's docs or run the discovery commands below. For the Digital Research Alliance of Canada, use the digital-research-alliance skill.

First: discover the cluster

Never assume partitions, QoS, or GPU types. Detect them:

sinfo -o "%P %.10l %.6D %G"        # partitions, timelimit, nodes, GRES (GPU types)
sinfo -O partition,gres:40,statelong   # GPU types + node states per partition
sacctmgr show qos format=Name,MaxWall,MaxTRESPU,Priority   # QoS limits
sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES  # what YOU can use
sshare -U                          # your fairshare / current usage
scontrol show config | grep -iE "MaxArraySize|MaxJobCount|SchedulerType"

sbatch script anatomy

#!/bin/bash
#SBATCH --job-name=myrun
#SBATCH --account=def-pi          # billing account (site-specific; required on many clusters)
#SBATCH --partition=gpu           # see `sinfo`
#SBATCH --qos=normal              # see `sacctmgr show qos`
#SBATCH --time=01:00:00           # WALLTIME — shorter = scheduled sooner (backfill)
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G                 # or --mem-per-cpu=8G ; do NOT also set both
#SBATCH --gres=gpu:1              # one GPU (see GPU section)
#SBATCH --output=logs/%x_%j.out   # %x=jobname %j=jobid (see Logging)
#SBATCH --error=logs/%x_%j.err
#SBATCH --mail-type=END,FAIL      # see Email alerts
#SBATCH --mail-user=you@example.com
#SBATCH --signal=B:TERM@120       # SIGTERM 120s before kill → checkpoint gracefully

set -euo pipefail
mkdir -p logs
srun python train.py             # `srun` inside sbatch = proper task launch + accounting

Submit: sbatch job.sh. Print the job id and do not block on it — poll later.

Requesting GPUs — whole, fraction, and shard

| Goal | Flag | |---|---| | 1 GPU (any type on partition) | --gres=gpu:1 or --gpus-per-node=1 | | Typed GPU | --gres=gpu:a100:1, --gres=gpu:h100:2, --gres=gpu:v100l:1 | | N GPUs total across nodes | --gpus=4 (newer Slurm) | | MIG fraction of an A100/H100 | --gres=gpu:a100_3g.20gb:1 (profile name is site-specific — sinfo -o %G) | | Shard (time-share 1 GPU) | --gres=shard:4 (only if admin enabled gres/shard) |

• A MIG slice is a hardware-isolated fraction of one GPU (e.g. 1g.5gb,

2g.10gb, 3g.20gb). Use it for dev, inference, or small jobs to stop wasting a whole A100/H100. Profile names vary — discover with sinfo -o "%P %G".

• A shard oversubscribes one physical GPU to several jobs by memory; weaker

isolation than MIG. Only available if the site configured it.

• On bundle-ratio clusters a single GPU comes with a fixed CPU/mem share —

requesting 1 GPU but 32 cores will queue forever. Match the ratio.

• nvidia-smi, nvtop (interactive), and seff (after) tell you if you

actually used the GPU. A job that requested a GPU and ran at 0% util is the #1 waste — check it.

Throttling: arrays, concurrency, dependencies

Array jobs run the same script over an index range — the right tool for sweeps:

#SBATCH --array=0-99%8     # 100 tasks, AT MOST 8 running at once  ← the %N throttle

sbatch --array=0-99%8 run.sh           # set throttle at submit
scontrol update ArrayTaskThrottle=4 JobId=12345   # change throttle LIVE on a running array

• %N is the single most important courtesy knob: it caps your concurrent

footprint so a big sweep doesn't starve other users (or your own cheap/urgent jobs) against a QoS MaxSubmitJobs/MaxTRESPU cap.

• Dependencies serialize stages: sbatch --dependency=afterok:12345 next.sh

(also afterany, afternotok, singleton = one job per name at a time).

• Inside the script, read $SLURM_ARRAY_TASK_ID to pick the work item.

See references/sbatch-template.sh for a full manifest-driven array template.

Pipelines & DAGs (submit the whole graph, then walk away)

The fastest, most hands-off pattern: in ONE session, submit every stage with dependencies wired up, capture the job ids with --parsable, and let Slurm run the DAG without you. Stages start the instant their parents succeed.

# --parsable prints just the jobid (chainable). Build the DAG bottom-up:
prep=$(sbatch --parsable prep.sh)
train=$(sbatch --parsable --dependency=afterok:$prep --array=0-9%4 train.sh)
eval=$(sbatch  --parsable --dependency=afterok:$train eval.sh)        # waits for WHOLE train array
sbatch --dependency=afterok:$eval --mail-type=END report.sh

Dependency types — pick the right edge:

| Spec | Meaning | |---|---| | afterok: | start only if parent succeeded (exit 0) — the default DAG edge | | afternotok: | start only if parent failed — recovery/cleanup branches | | afterany: | start when parent ends, success or not | | aftercorr: | array element-wise: task i of this array waits for task i of the parent array (preprocess[i]→train[i]) — not the whole array | | singleton | one job of this --job-name+user at a time — serialize a named chain | | after:+5 | start 5 min after parent starts (not finishes) |

• Combine edges: --dependency=afterok:$a:$b (AND, wait for both); ? separates OR.
• --kill-on-invalid-dep=yes so a child auto-cancels if its parent fails (instead

of sitting DependencyNeverSatisfied forever).

• aftercorr + two arrays is the elegant fan-out/fan-in: --array=0-99 prep, then

--array=0-99 --dependency=aftercorr:$prep train — 100 independent chains, each link starting as soon as its own parent is done. No barrier.

• Give cheap stages a GPU fraction (MIG/shard) and reserve whole GPUs for the

heavy stage, so the DAG packs onto fewer nodes (see GPU section).

Go/no-go gates — stop a sweep before it wastes GPUs

The highest-value DAG pattern: gate an expensive sweep behind a cheap gate experiment that answers "is this even worth running?". Slurm edges key on exit code, so make the gate exit 1 on no-go — the sweep is then auto-cancelled, zero GPU spent.

# Gate: one cheap run that validates a precondition. It must `exit 1` if NO-GO.
#   e.g. gate.sh runs a tiny probe and:  python check.py || exit 1   (nonzero = no-go)
gate=$(sbatch --parsable --job-name=gate --time=00:20:00 --gres=gpu:1 --mem=32G gate.sh)

# Expensive sweep runs ONLY if the gate passed. --kill-on-invalid-dep cancels it
# the instant the gate fails, instead of leaving 100 tasks pending forever.
sbatch --dependency=afterok:$gate --kill-on-invalid-dep=yes \
       --array=0-99%8 --gres=gpu:1 sweep.sh

# Optional: a cheap "why did it fail" report on the no-go branch.
sbatch --dependency=afternotok:$gate --time=00:05:00 nogo_report.sh

The gate's job is to fail fast and cheap. Make check.py exit nonzero when the result says stop — e.g. a baseline that must recover before downstream is valid, a benign run that must train before you spend GPUs attacking it, a smoke metric below threshold, a NaN/divergence check. Chain gates into an escalation ladder (tiny probe → medium probe → full sweep), each afterok the previous, so compute only flows toward configs that keep clearing the bar.

references/dag-pipeline.sh is a complete, runnable DAG submitter (gate → sweep → eval, with an afternotok recovery branch) you adapt and run once over SSH.

Off-cluster autonomous operation

Goal: touch the cluster as little as possible. Pattern:

1. One submission burst. SSH in, run the DAG submitter, capture all ids

(--parsable), write a job-id → stage map to a file you own, exit the SSH.

1. Survive preemption (matters on AI/spot partitions): add --requeue,

--signal=B:TERM@120 (catch SIGTERM, checkpoint, exit), --open-mode=append (logs survive requeue), and make run.py resume from the latest checkpoint. Now a requeued job continues instead of restarting.

1. Poll, don't babysit. From your laptop/agent: `ssh cluster 'sacct -j

--format=JobID,State,Elapsed,ExitCode'. Use the loss-curve rule to scancel revealed-answer jobs. Schedule the next check on an interval (agent: ScheduleWakeup / the loop` skill — short while rebalancing, long while everything just runs).

1. Self-healing edges. Wire afternotok cleanup/retry stages into the DAG so a

failure handles itself without you logging in.

1. Notify on the boundaries only: --mail-type=END,FAIL,TIME_LIMIT_80 on the

final stage and on FAIL of any stage — silence otherwise.

A whole multi-stage sweep can run start-to-finish with a single SSH to submit and a few sacct polls. For fanning the same DAG across several clusters at once, see the digital-research-alliance skill's multi-cluster section.

Email alerts

#SBATCH --mail-type=BEGIN,END,FAIL,REQUEUE,TIME_LIMIT_80
#SBATCH --mail-user=you@example.com
#SBATCH --mail-type=ARRAY_TASKS    # ADD this to get one mail PER array task (else only the array as a whole)

FAIL + TIME_LIMIT_80 (mail at 80% of walltime) is the useful pair: you hear about crashes and about jobs that will hit the wall, without inbox spam. Omit ARRAY_TASKS for big arrays or you self-DOS your inbox.

Logging

| Pattern | Expands to | |---|---| | %j | job id | %x | job name | %a | array task id | | %A | array master id | %N | first node | %u | user |

#SBATCH --output=logs/%x_%A_%a.out   # array: one file per task
#SBATCH --error=logs/%x_%A_%a.err    # separate stderr (drop to merge into .out)

Best practice: mkdir -p logs in-script (Slurm will NOT create the dir and the job dies silently if it's missing), log the resolved config + git rev-parse HEAD

• hostname/nvidia-smi at the top of every run for reproducibility.

Monitoring (infer the outcome, don't babysit)

| Command | Use | |---|---| | squeue --me | your queued/running jobs (-t RUNNING, --start = est. start time) | | squeue --me -o "%.18i %.9P %.30j %.2t %.10M %.6D %R" | readable, with reason | | sacct -j --format=JobID,State,Elapsed,MaxRSS,ReqTRES,ExitCode | what happened (works after job ends) | | seff | quick CPU/mem/GPU efficiency of a finished job | | sstat -j --format=JobID,MaxRSS,AveCPU | live stats of a RUNNING job | | scontrol show job | full live job record (why pending: see Reason=) | | sinfo -p -t idle,mix | free capacity right now |

Pending Reason= decoded: Priority/Resources = waiting your turn (fine); QOSMaxJobsPerUserLimit = throttle yourself; ReqNodeNotAvail/PartitionConfig = bad request, fix it; AssocGrpGRES = over your GPU allocation.

Loss-curve rule for ML runs: a run that should train and is stuck high, or a run that should be blocked and keeps dropping, has already revealed its answer — scancel it and iterate instead of waiting for walltime.

Control

scancel             scancel _       scancel -u $USER --state=PENDING
scancel --name=myrun    scancel -u $USER -t PENDING -p gpu   # targeted cancels
scontrol hold       scontrol release     # pause/unpause in queue
scontrol requeue    # restart (use with --requeue + checkpointing)
scontrol update JobId= TimeLimit=02:00:00    # extend walltime (if allowed)
scontrol update JobId= Partition=other       # move a PENDING job

Smoke tests before the big run

Never launch a 100-job sweep untested. In order of cost:

sbatch --test-only job.sh           # validates request + prints est. start; submits NOTHING
salloc --gres=gpu:1 --time=0:30:0   # interactive node: debug live, then exit
srun --gres=gpu:1 --pty bash        # interactive shell on a compute node
sbatch --array=0-0 run.sh           # run ONE array element end-to-end first

Run one element to completion (does it write output? right path? GPU used?) before unleashing --array=0-99%8.

Best-practice repo layout

project/
  slurm/
    job.sbatch              # parameterized; reads $1/env, not hardcoded
    logs/                   # gitignored
  configs/sweep.jsonl       # one JSON per line = one array task
  make_manifest.py          # writes configs/*.jsonl
  run.py                    # reads config by index, idempotent (skip if output exists)

• One sbatch script, many configs via a manifest + array index — not 50 copies.
• Make runs idempotent: check for the output file and exit 0 if it exists, so

requeue/resubmit is safe.

• Keep walltime honest and tight: backfill schedules short jobs into gaps sooner.
• export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True for PyTorch GPU jobs.

Common mistakes

| Mistake | Fix | |---|---| | --output dir doesn't exist | mkdir -p logs in-script; job dies silently otherwise | | Both --mem and --mem-per-cpu | pick one | | 1 GPU + many cores on a bundle cluster | match the node's CPU:GPU ratio | | Huge array with no %N | add --array=...%8; you're starving the QoS | | ARRAY_TASKS mail on 500-task array | remove it — inbox flood | | Walltime set to max "to be safe" | tight walltime backfills sooner | | Blocking/polling squeue in a tight loop | poll on an interval; use seff/sacct after | | GPU job at 0% util | wrong device/data pipeline — check seff/nvidia-smi |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: domenicrosati
• Source: domenicrosati/claude-hpc-skills
• License: MIT

Install and usage instructions live in the source repository linked above.