Alliance Slurm

Alliance (DRAC) Slurm

Alliance clusters run Slurm with a custom Lua submission layer that auto-routes jobs and a prolog that injects environment variables. Hardware varies by cluster — always discover before assuming. Auth is managed via CCDB; software modules come from CVMFS.

For cluster policies, storage quotas, and account questions, check alliance-docs first.

Discovery — never hardcode, always list

Run these first on any new cluster. Never guess account names, GPU types, partition names, or QOS values.

# Partitions and time limits
sinfo -s                                            # TIMELIMIT column is authoritative

# GPU GRES inventory (look for composite entries, e.g. gpu:l40s:4,shard:l40s:16)
sinfo -h -o "%G" --Node | sort -u
sinfo -h -o "%N %G %f" --Node | sort -u            # adds features (softmig, etc.)

# Full GRES detail on a specific node
scontrol show node  | grep -E 'Gres=|CfgTRES='

# Accounts available to you
sacctmgr show assoc user=$USER format=Account,DefaultAccount,QOS -p

# QOS limits (MaxWall / MaxTRES are often blank; prefer sinfo -s for walltime)
sacctmgr show qos format=Name,MaxWall,MaxTRES,Priority -p

# Fairshare / software
sshare -U
module avail && module spider 

Always run GPU discovery before writing --gres flags. GPU type names (v100l, a100, l40s, h100, …) differ by cluster and node.

Partitions — usually omit

A Lua plugin auto-routes jobs based on --time. Omit --partition unless you have a concrete reason (e.g. forcing CPU-only). Setting it with a mismatched --time causes instant rejection.

• Time tiers go up to a 7-day maximum; requests over 7 days are rejected.
• srun/salloc always route to the interactive partition. Check actual limits with

sinfo -s — GPU interactive is often longer than CPU interactive (e.g. 8 h vs 3 h).

Key flags

| Flag | Notes | |------|-------| | --account= | Required when you have multiple accounts or must charge a specific project. Check with sacctmgr show assoc user=$USER; omit only if your default account is correct. | | --time=HH:MM:SS | Default is 60 min. Drives partition routing. Always set. | | --mem=... | Default is very low, cgroup-enforced. Always set. | | --gres=gpu::N | Use exact type string from sinfo discovery. Bare gpu:N is unreliable and is rejected on general-purpose clusters (fir, nibi, rorqual, and similar). Always use the full type string. | | --cpus-per-task=N | Default is 1. |

Requesting GPUs

--gres=gpu::1     # one GPU
--gres=gpu::4     # multiple (verify node topology first)

MIG, fractional / soft-MIG GPU (cluster-dependent)

Standard MIG (most common)

MIG-capable nodes (e.g. A100) expose named slice GRES entries. Discover them the same way as regular GPUs:

sinfo -h -o "%G" --Node | sort -u
# MIG slices appear as e.g.: gpu:a100.1g.10gb:7,gpu:a100:4

Request a slice using the exact string from the inventory:

--gres=gpu:a100.1g.10gb:1    # 1/7th slice (10 GB)
--gres=gpu:a100.3g.40gb:1    # 3/7th slice (40 GB)

Count must always be 1. Use the exact slice name — do not guess.

Soft-MIG / fractional (less common)

Support is indicated by composite GRES entries, not by MIG strings in sinfo. Check the %G column:

sinfo -h -o "%G" --Node | sort -u
# A softmig-capable node shows e.g.: gpu:l40s:4,shard:l40s:16
# A plain GPU node shows:            gpu:l40s:4

shard:: alongside gpu: means fractional requests are supported. You may also see softmig in the features column (%f). For full detail:

scontrol show node  | grep -E 'Gres=|CfgTRES='

The fractional request syntax uses dot notation — these strings appear in your job flags, not as inventory lines in sinfo:

--gres=gpu:.:1   # e.g. gpu:l40s.4:1 for a 1/4 slice

Common denominators: 2, 3, 4. Count must always be 1. Do not attempt fractional requests on nodes where %G shows no shard: entry.

Submitting

# Batch
sbatch job.sh
sbatch --account= --gres=gpu::1 --mem=32G --time=1:00:00 \
       --wrap="python train.py"

# Interactive shell
srun --account= --gres=gpu::1 --cpus-per-task=4 --mem=32G \
     --time=2:00:00 --pty bash

# Allocation only
salloc --account= --gres=gpu::2 --mem=64G --time=2:00:00

Job scripts must start with #!/bin/bash — the Lua plugin checks this.

Auto-set environment (prolog-injected)

SLURMTMPDIR=/localscratch/ # or /tmp on Vulcan; always use $SLURMTMPDIR httpproxy / httpsproxy = http://squid:3128 XDGCACHEHOME=$SLURM_TMPDIR/cache

Override cache paths in every job that downloads models or large assets:

export HF_HOME=$SCRATCH/hf_cache
export TRANSFORMERS_CACHE=$SCRATCH/hf_cache
export XDG_CACHE_HOME=$SCRATCH/cache

$SLURM_TMPDIR is wiped at job end. Do not use /tmp directly — its path varies by cluster (/localscratch on most, /tmp on Vulcan).

On most clusters, compute nodes route HTTP/HTTPS through the squid proxy and have no direct internet access. Login nodes have full internet. Use https:// git URLs in jobs — SSH git and arbitrary TCP may be blocked.

Storage

/home/$USER — code/configs, persistent, not for heavy I/O. Job scripts must not be submitted from /home on general-purpose clusters (fir, nibi, rorqual, and similar) — move scripts and set your working directory under /scratch or /project before calling sbatch. /scratch/$USER — datasets/checkpoints, fast, large, not backed up. /project/ — persistent group space.

Monitoring & accounting

squeue -u $USER -o "%i %r %R"            # queue + pending reason
scontrol show job 
seff                               # CPU/mem efficiency (post-completion)
sattach .0                         # attach to running stdio (if available)
sacct -u $USER -X --starttime=now-7days \
      --format=JobID,JobName,State,Elapsed,AllocCPUS,ReqMem,MaxRSS
sprio -u $USER && sshare -U              # priority and fairshare

Job script templates

Single GPU

#!/bin/bash
#SBATCH --job-name=train
#SBATCH --account=
#SBATCH --gres=gpu::1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=4:00:00
#SBATCH --output=%x_%j.out

export HF_HOME=$SCRATCH/hf_cache
export XDG_CACHE_HOME=$SCRATCH/cache

module --force purge
module load StdEnv/2023 cuda/12.x python/3.x
source $SCRATCH/venvs/myenv/bin/activate
python train.py

Multi-GPU / array (flag patterns)

# Single node 4 GPUs
#SBATCH --gres=gpu::4
torchrun --nproc_per_node=4 train_ddp.py

# Multi-node: add --nodes=2 --ntasks-per-node=4; use scontrol show hostnames for master node
# Array: --array=0-9 (or 1-100%5 to throttle); use $SLURM_ARRAY_TASK_ID in script

Common pitfalls

| Symptom | Cause | Fix | |---------|-------|-----| | "partition does not exist or cannot fit" | --partition with mismatched --time | Drop --partition; let Lua route | | OUT_OF_MEMORY within seconds | Default mem very low | Always set --mem | | Wrong/invalid GPU gres string | Guessed type name | sinfo -h -o "%G" --Node \| sort -u | | Fractional GPU rejected | No shard GRES on node, or wrong syntax | Check %G for shard: entries; confirm dot-notation with site docs or scontrol show node | | $SLURM_TMPDIR fills up | XDG_CACHE_HOME=$SLURM_TMPDIR/cache from prolog | Export HF_HOME and XDG_CACHE_HOME to /scratch | | git clone / pip network error | SSH git / raw TCP blocked | Use HTTPS URLs | | Job exits at 60 min | No --time set | Always set --time | | Job rejected at submission | Script or CWD is under /home on a GP cluster (fir, nibi, rorqual) | cd /scratch/$USER/... and resubmit from there |

Quick reference

# Discover
sinfo -s                                             # partitions + time limits
sinfo -h -o "%G" --Node | sort -u                   # GPU GRES (look for shard: for fractional)
sinfo -h -o "%N %G %f" --Node | sort -u             # add features (softmig, etc.)
scontrol show node  | grep -E 'Gres=|CfgTRES='
sacctmgr show assoc user=$USER format=Account,DefaultAccount,QOS -p

# Submit / interactive
sbatch job.sh
srun --account= --gres=gpu::1 --mem=32G --time=1:00:00 --pty bash

# Watch / inspect / manage
squeue -u $USER
scontrol show job 
seff 
scancel 
scontrol update JobId= TimeLimit=4:00:00

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ualberta-rcg
• Source: ualberta-rcg/drac-agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.