# Alliance Slurm

> Submit, monitor, and troubleshoot Slurm jobs on Alliance (DRAC) HPC clusters. Covers Lua auto-routing, GPU/GRES discovery, MIG and soft-MIG, prolog-injected environment, proxy, cache redirection, and storage rules. Use `alliance-cvmfs` for software and modules, `alliance-docs` for cluster policies.

- **Type:** Skill
- **Install:** `agentstack add skill-ualberta-rcg-drac-agent-skills-alliance-slurm`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [ualberta-rcg](https://agentstack.voostack.com/s/ualberta-rcg)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [ualberta-rcg](https://github.com/ualberta-rcg)
- **Source:** https://github.com/ualberta-rcg/drac-agent-skills/tree/main/skills/alliance-slurm

## Install

```sh
agentstack add skill-ualberta-rcg-drac-agent-skills-alliance-slurm
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Alliance (DRAC) Slurm

Alliance clusters run Slurm with a custom Lua submission layer that auto-routes jobs
and a prolog that injects environment variables. Hardware varies by cluster — always
discover before assuming. Auth is managed via CCDB; software modules come from CVMFS.

For cluster policies, storage quotas, and account questions, check `alliance-docs` first.

## Discovery — never hardcode, always list

Run these first on any new cluster. Never guess account names, GPU types, partition
names, or QOS values.

```bash
# Partitions and time limits
sinfo -s                                            # TIMELIMIT column is authoritative

# GPU GRES inventory (look for composite entries, e.g. gpu:l40s:4,shard:l40s:16)
sinfo -h -o "%G" --Node | sort -u
sinfo -h -o "%N %G %f" --Node | sort -u            # adds features (softmig, etc.)

# Full GRES detail on a specific node
scontrol show node  | grep -E 'Gres=|CfgTRES='

# Accounts available to you
sacctmgr show assoc user=$USER format=Account,DefaultAccount,QOS -p

# QOS limits (MaxWall / MaxTRES are often blank; prefer sinfo -s for walltime)
sacctmgr show qos format=Name,MaxWall,MaxTRES,Priority -p

# Fairshare / software
sshare -U
module avail && module spider 
```

**Always run GPU discovery before writing `--gres` flags.** GPU type names (`v100l`,
`a100`, `l40s`, `h100`, …) differ by cluster and node.

## Partitions — usually omit

A Lua plugin auto-routes jobs based on `--time`. **Omit `--partition` unless you have
a concrete reason** (e.g. forcing CPU-only). Setting it with a mismatched `--time`
causes instant rejection.

- Time tiers go up to a 7-day maximum; requests over 7 days are rejected.
- `srun`/`salloc` always route to the interactive partition. Check actual limits with
  `sinfo -s` — GPU interactive is often longer than CPU interactive (e.g. 8 h vs 3 h).

## Key flags

| Flag | Notes |
|------|-------|
| `--account=` | Required when you have multiple accounts or must charge a specific project. Check with `sacctmgr show assoc user=$USER`; omit only if your default account is correct. |
| `--time=HH:MM:SS` | **Default is 60 min.** Drives partition routing. Always set. |
| `--mem=...` | **Default is very low**, cgroup-enforced. Always set. |
| `--gres=gpu::N` | Use exact type string from `sinfo` discovery. Bare gpu:N is unreliable and is rejected on general-purpose clusters (fir, nibi, rorqual, and similar). Always use the full type string. |
| `--cpus-per-task=N` | Default is 1. |

## Requesting GPUs

```bash
--gres=gpu::1     # one GPU
--gres=gpu::4     # multiple (verify node topology first)
```

### MIG, fractional / soft-MIG GPU (cluster-dependent)

#### Standard MIG (most common)
MIG-capable nodes (e.g. A100) expose named slice GRES entries. Discover them the same
way as regular GPUs:

```bash
sinfo -h -o "%G" --Node | sort -u
# MIG slices appear as e.g.: gpu:a100.1g.10gb:7,gpu:a100:4
```

Request a slice using the exact string from the inventory:

```bash
--gres=gpu:a100.1g.10gb:1    # 1/7th slice (10 GB)
--gres=gpu:a100.3g.40gb:1    # 3/7th slice (40 GB)
```

Count must always be `1`. Use the exact slice name — do not guess.

#### Soft-MIG / fractional (less common)

Support is indicated by composite GRES entries, **not** by MIG strings in `sinfo`.
Check the `%G` column:

```bash
sinfo -h -o "%G" --Node | sort -u
# A softmig-capable node shows e.g.: gpu:l40s:4,shard:l40s:16
# A plain GPU node shows:            gpu:l40s:4
```

`shard::` alongside `gpu:` means fractional requests are supported. You may
also see `softmig` in the features column (`%f`). For full detail:

```bash
scontrol show node  | grep -E 'Gres=|CfgTRES='
```

The fractional request syntax uses dot notation — these strings appear in your job
flags, **not** as inventory lines in `sinfo`:

```bash
--gres=gpu:.:1   # e.g. gpu:l40s.4:1 for a 1/4 slice
```

Common denominators: `2`, `3`, `4`. Count must always be `1`. Do not attempt
fractional requests on nodes where `%G` shows no `shard:` entry.

## Submitting

```bash
# Batch
sbatch job.sh
sbatch --account= --gres=gpu::1 --mem=32G --time=1:00:00 \
       --wrap="python train.py"

# Interactive shell
srun --account= --gres=gpu::1 --cpus-per-task=4 --mem=32G \
     --time=2:00:00 --pty bash

# Allocation only
salloc --account= --gres=gpu::2 --mem=64G --time=2:00:00
```

Job scripts must start with `#!/bin/bash` — the Lua plugin checks this.

## Auto-set environment (prolog-injected)

SLURM_TMPDIR=/localscratch/   # or /tmp on Vulcan; always use $SLURM_TMPDIR
http_proxy / https_proxy = http://squid:3128
XDG_CACHE_HOME=$SLURM_TMPDIR/cache

Override cache paths in every job that downloads models or large assets:

```bash
export HF_HOME=$SCRATCH/hf_cache
export TRANSFORMERS_CACHE=$SCRATCH/hf_cache
export XDG_CACHE_HOME=$SCRATCH/cache
```

`$SLURM_TMPDIR` is wiped at job end. Do not use `/tmp` directly — its path varies by cluster (`/localscratch` on most, `/tmp` on Vulcan).

On most clusters, compute nodes route HTTP/HTTPS through the squid proxy and have no direct internet access. Login nodes have full internet. Use `https://` git URLs in jobs — SSH git and arbitrary TCP may be blocked.

## Storage

`/home/$USER` — code/configs, persistent, not for heavy I/O. **Job scripts must not be submitted from `/home` on general-purpose clusters (fir, nibi, rorqual, and similar)** — move scripts and set your working directory under `/scratch` or `/project` before calling `sbatch`. `/scratch/$USER` — datasets/checkpoints, fast, large, **not backed up**. `/project/` — persistent group space.

## Monitoring & accounting

```bash
squeue -u $USER -o "%i %r %R"            # queue + pending reason
scontrol show job 
seff                               # CPU/mem efficiency (post-completion)
sattach .0                         # attach to running stdio (if available)
sacct -u $USER -X --starttime=now-7days \
      --format=JobID,JobName,State,Elapsed,AllocCPUS,ReqMem,MaxRSS
sprio -u $USER && sshare -U              # priority and fairshare
```

## Job script templates

### Single GPU
```bash
#!/bin/bash
#SBATCH --job-name=train
#SBATCH --account=
#SBATCH --gres=gpu::1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=4:00:00
#SBATCH --output=%x_%j.out

export HF_HOME=$SCRATCH/hf_cache
export XDG_CACHE_HOME=$SCRATCH/cache

module --force purge
module load StdEnv/2023 cuda/12.x python/3.x
source $SCRATCH/venvs/myenv/bin/activate
python train.py
```

### Multi-GPU / array (flag patterns)
```bash
# Single node 4 GPUs
#SBATCH --gres=gpu::4
torchrun --nproc_per_node=4 train_ddp.py

# Multi-node: add --nodes=2 --ntasks-per-node=4; use scontrol show hostnames for master node
# Array: --array=0-9 (or 1-100%5 to throttle); use $SLURM_ARRAY_TASK_ID in script
```

## Common pitfalls

| Symptom | Cause | Fix |
|---------|-------|-----|
| "partition does not exist or cannot fit" | `--partition` with mismatched `--time` | Drop `--partition`; let Lua route |
| `OUT_OF_MEMORY` within seconds | Default mem very low | Always set `--mem` |
| Wrong/invalid GPU gres string | Guessed type name | `sinfo -h -o "%G" --Node \| sort -u` |
| Fractional GPU rejected | No shard GRES on node, or wrong syntax | Check `%G` for `shard:` entries; confirm dot-notation with site docs or `scontrol show node` |
| `$SLURM_TMPDIR` fills up | `XDG_CACHE_HOME=$SLURM_TMPDIR/cache` from prolog | Export `HF_HOME` and `XDG_CACHE_HOME` to `/scratch` |
| `git clone` / pip network error | SSH git / raw TCP blocked | Use HTTPS URLs |
| Job exits at 60 min | No `--time` set | Always set `--time` |
| Job rejected at submission | Script or CWD is under `/home` on a GP cluster (fir, nibi, rorqual) | `cd /scratch/$USER/...` and resubmit from there |

## Quick reference

```bash
# Discover
sinfo -s                                             # partitions + time limits
sinfo -h -o "%G" --Node | sort -u                   # GPU GRES (look for shard: for fractional)
sinfo -h -o "%N %G %f" --Node | sort -u             # add features (softmig, etc.)
scontrol show node  | grep -E 'Gres=|CfgTRES='
sacctmgr show assoc user=$USER format=Account,DefaultAccount,QOS -p

# Submit / interactive
sbatch job.sh
srun --account= --gres=gpu::1 --mem=32G --time=1:00:00 --pty bash

# Watch / inspect / manage
squeue -u $USER
scontrol show job 
seff 
scancel 
scontrol update JobId= TimeLimit=4:00:00
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ualberta-rcg](https://github.com/ualberta-rcg)
- **Source:** [ualberta-rcg/drac-agent-skills](https://github.com/ualberta-rcg/drac-agent-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-ualberta-rcg-drac-agent-skills-alliance-slurm
- Seller: https://agentstack.voostack.com/s/ualberta-rcg
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.