# Slurm

> Use when submitting, monitoring, throttling, or debugging jobs on a Slurm HPC/GPU cluster — writing sbatch scripts, requesting GPUs (whole, MIG fraction, or shard), array job throttling, email alerts, partitions, QoS, fairshare, logging, scancel/scontrol control, smoke tests with salloc/srun/--test-only, and agent-driven monitoring loops.

- **Type:** Skill
- **Install:** `agentstack add skill-domenicrosati-claude-hpc-skills-slurm`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [domenicrosati](https://agentstack.voostack.com/s/domenicrosati)
- **Installs:** 0
- **Category:** [Communication](https://agentstack.voostack.com/c/communication)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [domenicrosati](https://github.com/domenicrosati)
- **Source:** https://github.com/domenicrosati/claude-hpc-skills/tree/main/skills/slurm

## Install

```sh
agentstack add skill-domenicrosati-claude-hpc-skills-slurm
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Slurm

## Overview

Slurm (`sbatch`/`squeue`/`sacct`/`scontrol`) schedules jobs on shared HPC and GPU
clusters. Core principle: **describe the smallest resource box your job fits in
(time, cpus, mem, gpus), throttle your own fan-out so you stay a good neighbor,
and infer outcomes from logs/accounting instead of babysitting.**

This skill is cluster-agnostic. For a specific site (account format, partition
names, GPU models) read that site's docs or run the discovery commands below.
For the Digital Research Alliance of Canada, use the `digital-research-alliance` skill.

## First: discover the cluster

Never assume partitions, QoS, or GPU types. Detect them:

```bash
sinfo -o "%P %.10l %.6D %G"        # partitions, timelimit, nodes, GRES (GPU types)
sinfo -O partition,gres:40,statelong   # GPU types + node states per partition
sacctmgr show qos format=Name,MaxWall,MaxTRESPU,Priority   # QoS limits
sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES  # what YOU can use
sshare -U                          # your fairshare / current usage
scontrol show config | grep -iE "MaxArraySize|MaxJobCount|SchedulerType"
```

## sbatch script anatomy

```bash
#!/bin/bash
#SBATCH --job-name=myrun
#SBATCH --account=def-pi          # billing account (site-specific; required on many clusters)
#SBATCH --partition=gpu           # see `sinfo`
#SBATCH --qos=normal              # see `sacctmgr show qos`
#SBATCH --time=01:00:00           # WALLTIME — shorter = scheduled sooner (backfill)
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G                 # or --mem-per-cpu=8G ; do NOT also set both
#SBATCH --gres=gpu:1              # one GPU (see GPU section)
#SBATCH --output=logs/%x_%j.out   # %x=jobname %j=jobid (see Logging)
#SBATCH --error=logs/%x_%j.err
#SBATCH --mail-type=END,FAIL      # see Email alerts
#SBATCH --mail-user=you@example.com
#SBATCH --signal=B:TERM@120       # SIGTERM 120s before kill → checkpoint gracefully

set -euo pipefail
mkdir -p logs
srun python train.py             # `srun` inside sbatch = proper task launch + accounting
```

Submit: `sbatch job.sh`. Print the job id and **do not block on it** — poll later.

## Requesting GPUs — whole, fraction, and shard

| Goal | Flag |
|---|---|
| 1 GPU (any type on partition) | `--gres=gpu:1` or `--gpus-per-node=1` |
| Typed GPU | `--gres=gpu:a100:1`, `--gres=gpu:h100:2`, `--gres=gpu:v100l:1` |
| N GPUs total across nodes | `--gpus=4` (newer Slurm) |
| **MIG fraction** of an A100/H100 | `--gres=gpu:a100_3g.20gb:1` (profile name is site-specific — `sinfo -o %G`) |
| **Shard** (time-share 1 GPU) | `--gres=shard:4` (only if admin enabled `gres/shard`) |

- A MIG slice is a hardware-isolated *fraction* of one GPU (e.g. `1g.5gb`,
  `2g.10gb`, `3g.20gb`). Use it for dev, inference, or small jobs to stop wasting
  a whole A100/H100. Profile names vary — discover with `sinfo -o "%P %G"`.
- A *shard* oversubscribes one physical GPU to several jobs by memory; weaker
  isolation than MIG. Only available if the site configured it.
- On bundle-ratio clusters a single GPU comes with a fixed CPU/mem share —
  requesting 1 GPU but 32 cores will queue forever. Match the ratio.
- `nvidia-smi`, `nvtop` (interactive), and `seff ` (after) tell you if you
  actually used the GPU. A job that requested a GPU and ran at 0% util is the
  #1 waste — check it.

## Throttling: arrays, concurrency, dependencies

**Array jobs** run the same script over an index range — the right tool for sweeps:

```bash
#SBATCH --array=0-99%8     # 100 tasks, AT MOST 8 running at once  ← the %N throttle
```

```bash
sbatch --array=0-99%8 run.sh           # set throttle at submit
scontrol update ArrayTaskThrottle=4 JobId=12345   # change throttle LIVE on a running array
```

- `%N` is the single most important courtesy knob: it caps your concurrent
  footprint so a big sweep doesn't starve other users (or your own cheap/urgent
  jobs) against a QoS `MaxSubmitJobs`/`MaxTRESPU` cap.
- **Dependencies** serialize stages: `sbatch --dependency=afterok:12345 next.sh`
  (also `afterany`, `afternotok`, `singleton` = one job per name at a time).
- Inside the script, read `$SLURM_ARRAY_TASK_ID` to pick the work item.

See `references/sbatch-template.sh` for a full manifest-driven array template.

## Pipelines & DAGs (submit the whole graph, then walk away)

The fastest, most hands-off pattern: in ONE session, submit every stage with
dependencies wired up, capture the job ids with `--parsable`, and let Slurm run the
DAG without you. Stages start the instant their parents succeed.

```bash
# --parsable prints just the jobid (chainable). Build the DAG bottom-up:
prep=$(sbatch --parsable prep.sh)
train=$(sbatch --parsable --dependency=afterok:$prep --array=0-9%4 train.sh)
eval=$(sbatch  --parsable --dependency=afterok:$train eval.sh)        # waits for WHOLE train array
sbatch --dependency=afterok:$eval --mail-type=END report.sh
```

Dependency types — pick the right edge:

| Spec | Meaning |
|---|---|
| `afterok:` | start only if parent **succeeded** (exit 0) — the default DAG edge |
| `afternotok:` | start only if parent **failed** — recovery/cleanup branches |
| `afterany:` | start when parent ends, success or not |
| **`aftercorr:`** | **array element-wise:** task `i` of this array waits for task `i` of the parent array (preprocess[i]→train[i]) — not the whole array |
| `singleton` | one job of this `--job-name`+user at a time — serialize a named chain |
| `after:+5` | start 5 min after parent *starts* (not finishes) |

- Combine edges: `--dependency=afterok:$a:$b` (AND, wait for both); `?` separates OR.
- **`--kill-on-invalid-dep=yes`** so a child auto-cancels if its parent fails (instead
  of sitting `DependencyNeverSatisfied` forever).
- `aftercorr` + two arrays is the elegant fan-out/fan-in: `--array=0-99` prep, then
  `--array=0-99 --dependency=aftercorr:$prep` train — 100 independent chains, each
  link starting as soon as its own parent is done. No barrier.
- Give cheap stages a **GPU fraction** (MIG/shard) and reserve whole GPUs for the
  heavy stage, so the DAG packs onto fewer nodes (see GPU section).

### Go/no-go gates — stop a sweep before it wastes GPUs

The highest-value DAG pattern: gate an expensive sweep behind a **cheap gate
experiment** that answers "is this even worth running?". Slurm edges key on **exit
code**, so make the gate `exit 1` on no-go — the sweep is then auto-cancelled, zero
GPU spent.

```bash
# Gate: one cheap run that validates a precondition. It must `exit 1` if NO-GO.
#   e.g. gate.sh runs a tiny probe and:  python check.py || exit 1   (nonzero = no-go)
gate=$(sbatch --parsable --job-name=gate --time=00:20:00 --gres=gpu:1 --mem=32G gate.sh)

# Expensive sweep runs ONLY if the gate passed. --kill-on-invalid-dep cancels it
# the instant the gate fails, instead of leaving 100 tasks pending forever.
sbatch --dependency=afterok:$gate --kill-on-invalid-dep=yes \
       --array=0-99%8 --gres=gpu:1 sweep.sh

# Optional: a cheap "why did it fail" report on the no-go branch.
sbatch --dependency=afternotok:$gate --time=00:05:00 nogo_report.sh
```

The gate's job is to **fail fast and cheap**. Make `check.py` exit nonzero when the
result says stop — e.g. a baseline that must recover before downstream is valid, a
benign run that must train before you spend GPUs attacking it, a smoke metric below
threshold, a NaN/divergence check. Chain gates into an **escalation ladder** (tiny
probe → medium probe → full sweep), each `afterok` the previous, so compute only
flows toward configs that keep clearing the bar.

`references/dag-pipeline.sh` is a complete, runnable DAG submitter (gate → sweep →
eval, with an `afternotok` recovery branch) you adapt and run once over SSH.

## Off-cluster autonomous operation

Goal: touch the cluster as little as possible. Pattern:

1. **One submission burst.** SSH in, run the DAG submitter, capture all ids
   (`--parsable`), write a `job-id → stage` map to a file you own, exit the SSH.
2. **Survive preemption** (matters on AI/spot partitions): add `--requeue`,
   `--signal=B:TERM@120` (catch SIGTERM, checkpoint, exit), `--open-mode=append` (logs
   survive requeue), and make `run.py` resume from the latest checkpoint. Now a
   requeued job continues instead of restarting.
3. **Poll, don't babysit.** From your laptop/agent: `ssh cluster 'sacct -j 
   --format=JobID,State,Elapsed,ExitCode'`. Use the loss-curve rule to `scancel`
   revealed-answer jobs. Schedule the next check on an interval (agent: `ScheduleWakeup`
   / the `loop` skill — short while rebalancing, long while everything just runs).
4. **Self-healing edges.** Wire `afternotok` cleanup/retry stages into the DAG so a
   failure handles itself without you logging in.
5. **Notify on the boundaries only:** `--mail-type=END,FAIL,TIME_LIMIT_80` on the
   final stage and on `FAIL` of any stage — silence otherwise.

A whole multi-stage sweep can run start-to-finish with a single SSH to submit and a
few `sacct` polls. For fanning the same DAG across several clusters at once, see the
`digital-research-alliance` skill's multi-cluster section.

## Email alerts

```bash
#SBATCH --mail-type=BEGIN,END,FAIL,REQUEUE,TIME_LIMIT_80
#SBATCH --mail-user=you@example.com
#SBATCH --mail-type=ARRAY_TASKS    # ADD this to get one mail PER array task (else only the array as a whole)
```

`FAIL` + `TIME_LIMIT_80` (mail at 80% of walltime) is the useful pair: you hear
about crashes and about jobs that will hit the wall, without inbox spam. Omit
`ARRAY_TASKS` for big arrays or you self-DOS your inbox.

## Logging

| Pattern | Expands to |
|---|---|
| `%j` | job id | `%x` | job name | `%a` | array task id |
| `%A` | array master id | `%N` | first node | `%u` | user |

```bash
#SBATCH --output=logs/%x_%A_%a.out   # array: one file per task
#SBATCH --error=logs/%x_%A_%a.err    # separate stderr (drop to merge into .out)
```

Best practice: `mkdir -p logs` in-script (Slurm will NOT create the dir and the
job dies silently if it's missing), log the resolved config + `git rev-parse HEAD`
+ `hostname`/`nvidia-smi` at the top of every run for reproducibility.

## Monitoring (infer the outcome, don't babysit)

| Command | Use |
|---|---|
| `squeue --me` | your queued/running jobs (`-t RUNNING`, `--start` = est. start time) |
| `squeue --me -o "%.18i %.9P %.30j %.2t %.10M %.6D %R"` | readable, with reason |
| `sacct -j  --format=JobID,State,Elapsed,MaxRSS,ReqTRES,ExitCode` | what happened (works after job ends) |
| `seff ` | quick CPU/mem/GPU **efficiency** of a finished job |
| `sstat -j  --format=JobID,MaxRSS,AveCPU` | live stats of a RUNNING job |
| `scontrol show job ` | full live job record (why pending: see `Reason=`) |
| `sinfo -p  -t idle,mix` | free capacity right now |

Pending `Reason=` decoded: `Priority`/`Resources` = waiting your turn (fine);
`QOSMaxJobsPerUserLimit` = throttle yourself; `ReqNodeNotAvail`/`PartitionConfig`
= bad request, fix it; `AssocGrpGRES` = over your GPU allocation.

**Loss-curve rule for ML runs:** a run that *should* train and is stuck high, or
a run that *should* be blocked and keeps dropping, has already revealed its answer
— `scancel` it and iterate instead of waiting for walltime.

## Control

```bash
scancel             scancel _       scancel -u $USER --state=PENDING
scancel --name=myrun    scancel -u $USER -t PENDING -p gpu   # targeted cancels
scontrol hold       scontrol release     # pause/unpause in queue
scontrol requeue    # restart (use with --requeue + checkpointing)
scontrol update JobId= TimeLimit=02:00:00    # extend walltime (if allowed)
scontrol update JobId= Partition=other       # move a PENDING job
```

## Smoke tests before the big run

Never launch a 100-job sweep untested. In order of cost:

```bash
sbatch --test-only job.sh           # validates request + prints est. start; submits NOTHING
salloc --gres=gpu:1 --time=0:30:0   # interactive node: debug live, then exit
srun --gres=gpu:1 --pty bash        # interactive shell on a compute node
sbatch --array=0-0 run.sh           # run ONE array element end-to-end first
```

Run one element to completion (does it write output? right path? GPU used?)
*before* unleashing `--array=0-99%8`.

## Best-practice repo layout

```
project/
  slurm/
    job.sbatch              # parameterized; reads $1/env, not hardcoded
    logs/                   # gitignored
  configs/sweep.jsonl       # one JSON per line = one array task
  make_manifest.py          # writes configs/*.jsonl
  run.py                    # reads config by index, idempotent (skip if output exists)
```

- One sbatch script, many configs via a manifest + array index — not 50 copies.
- Make runs **idempotent**: check for the output file and exit 0 if it exists, so
  requeue/resubmit is safe.
- Keep walltime honest and tight: backfill schedules short jobs into gaps sooner.
- `export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for PyTorch GPU jobs.

## Common mistakes

| Mistake | Fix |
|---|---|
| `--output` dir doesn't exist | `mkdir -p logs` in-script; job dies silently otherwise |
| Both `--mem` and `--mem-per-cpu` | pick one |
| 1 GPU + many cores on a bundle cluster | match the node's CPU:GPU ratio |
| Huge array with no `%N` | add `--array=...%8`; you're starving the QoS |
| `ARRAY_TASKS` mail on 500-task array | remove it — inbox flood |
| Walltime set to max "to be safe" | tight walltime backfills sooner |
| Blocking/polling `squeue` in a tight loop | poll on an interval; use `seff`/`sacct` after |
| GPU job at 0% util | wrong device/data pipeline — check `seff`/`nvidia-smi` |

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [domenicrosati](https://github.com/domenicrosati)
- **Source:** [domenicrosati/claude-hpc-skills](https://github.com/domenicrosati/claude-hpc-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-domenicrosati-claude-hpc-skills-slurm
- Seller: https://agentstack.voostack.com/s/domenicrosati
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.