Hkustgz Hpc2

HKUST-GZ HPC2 (Slurm) Cheat Sheet

HPC Phase 2 uses Slurm. Phase 1 uses LSF (jsub / #BSUB) — different system, don't mix.

Official docs: https://docs.hpc.hkust-gz.edu.cn/docs/hpc12/

Rule: confirm every parameter before submitting

When the user asks you to run something on this cluster, list every sbatch/srun parameter and get explicit approval before invoking it. No silent defaults — applies even on the free debug partition.

Always present, then wait:

• -p partition
• -t walltime
• -n / -c CPU count
• --mem memory
• --gres=gpu:N GPU count (+ partition implies type)
• The actual command(s) to run
• module load … / env activation
• -o / -e output paths
• -D working directory if it matters

Prior approval is scoped to that one submission — don't carry it forward to the next run. Use AskUserQuestion when there's a real choice to make (partition tier, walltime budget).

Connect

ssh hkustgz-hpc2          # via ~/.ssh/config alias
# or
ssh @hpc2login.hpc.hkust-gz.edu.cn

Home directory: /hpc2hdd/home/

Login node is for editing/compiling/submitting only — never run heavy compute there. Use srun for interactive work.

Load Slurm before any sbatch/srun/squeue

The login node's /usr/bin/sbatch etc. exist but are misconfigured (slurm.conf not at default path). You must load the Slurm module first or every Slurm command fails with Unable to process configuration file:

module load slurm
which sinfo   # → /opt/slurm/bin/sinfo  (correct one)

Make it permanent by appending to ~/.bashrc:

echo 'module load slurm' >> ~/.bashrc

Compute nodes inside a job don't need this — they have a working local Slurm. The module is only for the login node where you submit/monitor from.

Submit a batch job

Write a script, e.g. run.sh:

#!/bin/bash
#SBATCH -J myjob              # job name
#SBATCH -p i64m512u           # partition (see table below)
#SBATCH -o output_%j.txt      # stdout, %j = jobid
#SBATCH -e err_%j.txt         # stderr
#SBATCH -n 8                  # total CPU cores
#SBATCH --mem=16G             # memory
#SBATCH -t 02:00:00           # walltime HH:MM:SS

echo "Job started at $(date) on $(hostname)"
python your_script.py
echo "Job ended at $(date)"

Submit:

sbatch run.sh                 # → "Submitted batch job "

GPU job

#SBATCH -p i64m1tga800u       # A800 partition
#SBATCH --gres=gpu:1          # 1 GPU; use gpu:2 etc.
#SBATCH -n 8                  # CPU cores to pair with GPU
#SBATCH --mem=32G
#SBATCH -t 04:00:00

A800 nodes support up to 16 GPUs/node, A40 up to 8.

Array job

#SBATCH -o output_%A_%a.txt   # %A=array id, %a=task id
#SBATCH --array=1-10          # or 1-10%3 to cap concurrency at 3
PARAM=$SLURM_ARRAY_TASK_ID
python your_script.py $PARAM

Interactive session

srun -p i64m512u -n 4 --mem=8G -t 01:00:00 --pty bash
srun -p i64m1tga800u -n 8 --mem=32G --gres=gpu:1 -t 01:00:00 --pty bash

Land in a shell on a compute node. Exit with exit to release the allocation.

Check status / monitor

squeue -u $USER                       # my running + pending jobs
squeue -j                      # one job
scontrol show job              # full details: paths, resources, why pending
sacct -u $USER --starttime=today      # today's history
sacct -j  --format=JobID,State,Elapsed,MaxRSS,ReqMem,AllocCPUS,AllocGRES
                                      # post-mortem: how much was actually used
tail -f output_.txt            # live stdout from login node
ssh  top                    # `scontrol show job` shows NodeList; peek with top

State codes from squeue: R=running, PD=pending, CG=completing, F=failed, CD=completed.

If a job is stuck in PD, the REASON column tells you why (Resources, Priority, QOSMaxJobsPerUserLimit, etc.).

Cancel / extend

scancel                        # cancel one
scancel -u $USER                      # cancel all mine
scontrol suspend               # pause (rarely needed)
scontrol resume 

Extending walltime is not a CLI op here — go to the platform portal → "My Jobs" → "Extend Time". Max +7 days per request, hard ceiling 14 days. For longer, file a ticket.

Partitions (HPC2)

| Partition | Type | Cores/Node limit | Wall | Use for | |---|---|---|---|---| | i64m512u | CPU shared | up to 1024 | 7d | normal CPU jobs | | i64m512ue | CPU exclusive | 128 | 7d | full-node CPU | | long_cpu | CPU shared | 1024 | 14d | long runs | | emergency_cpu | CPU high-pri | 512 | — | rush (premium) | | a128m512u | AMD EPYC | 256 | 7d | AMD-optimized | | i64m1tga800u(e) | GPU A800 | up to 16 GPU | 7d | training/big models | | i64m1tga40u(e) | GPU A40 | up to 8 GPU | 7d | smaller GPU jobs | | i96m3tu(e) | Large-mem | 192c / 3TB | 7d | memory-hungry | | debug | Debug | 16c / 2 GPU | 0.5h | free — for script validation |

u = shared (lower priority, lower rate), ue = exclusive node (higher).

Priority = the partition you pick

You don't set priority with a flag. The partition's built-in tier decides both scheduling priority and price:

| Tier | Partitions | Use | |---:|---|---| | 100 (low) | *u, long_*, i64m512r | default — cheap, shared | | 200 (med) | *ue, i64m512re | exclusive node, faster scheduling | | 300 (high) | emergency_cpu, emergency_gpu, emergency_gpua40 | rush, premium rate |

HARD RULE: only ever propose low-tier (*u) partitions. Medium (*ue) or high (emergency_*) are off-limits unless the user explicitly names that tier in their instruction for this specific job (e.g. "use i64m512ue" or "this is urgent, put it on emergency"). "Faster", "more reliable", "important" do not count as authorization — still propose *u and let the user override. This will rarely happen; assume *u by default.

Don't add --qos=… or --priority=… — this cluster doesn't use them (all QOS priorities are 0).

Live view on the cluster: sinfo (standard Slurm) — shows real-time node state per partition. (jqueues is HPC1/LSF only, don't use on HPC2.)

Storage

Two GPFS pools:

| Mount | Type | Size | Use | |---|---|---:|---| | /hpc2hdd | HDD | 3.9 PB (often >90% full) | bulk, long-term, slower | | /hpc2ssd | SSD | smaller | fast scratch I/O |

Home is /hpc2hdd/home/. Inside it, several symlinks map portal-managed areas — pay attention to which point at SSD vs HDD:

| Symlink | Target | What it really is | |---|---|---| | ~/jhspoolers | /hpc2ssd/JH_DATA/spooler/ | SSD scratch — the only fast-I/O area you get | | ~/jhupload | /hpc2ssd/JH_DATA/spooler//.upload | portal upload landing on SSD | | ~/jhaidata | /hpc2hdd/JH_DATA/jhai_data/ | AI data area (HDD), large datasets | | ~/.jhshares | /hpc2hdd/JH_DATA/share/ | shared with your group | | ~/.filedownload | /hpc2hdd/JH_DATA/filedownload/ | portal download staging |

Where to put what

| Content | Path | Why | |---|---|---| | Code, configs, scripts, conda env | ~/ (home, HDD) | small, persistent | | Read-only datasets (long-term) | ~/jhaidata | big, HDD is fine | | Training intermediates, checkpoints, unpacked dataset cache | ~/jhspoolers (SSD) | hot-path I/O | | Group-shared data | ~/.jhshares | visible to your group |

Using SSD scratch in a job

mkdir -p ~/jhspoolers/job-$SLURM_JOB_ID
cd ~/jhspoolers/job-$SLURM_JOB_ID
cp ~/jhaidata/big_dataset.tar.gz . && tar xf big_dataset.tar.gz
python ~/code/train.py --data ./extracted --ckpt-dir ./ckpts
# afterwards: archive final results back to HDD, then clean up SSD
cp ckpts/final.pt ~/results/

Rule of thumb: heavy random/repeat-read I/O on SSD, archival on HDD.

Quotas

quota / lfs quota / mmlsquota are not installed on the login node — quota is only visible via the portal. Each pool (HDD/SSD) and each portal-managed area is counted separately. If a write fails with Disk quota exceeded, check the portal, not the filesystem.

Baseline principle: validate before sbatch

Compute time is billed once the job starts. Burning a 4-hour GPU slot on a typo is avoidable. Before sbatch:

1. Dry-run the script logic locally or in srun --pty with tiny inputs.
2. Smoke-test in debug partition first — it's free and capped at 30 min. Submit the real script there with a small --time to confirm modules load, paths resolve, GPU is visible (nvidia-smi).
3. Right-size walltime and resources. Easier to extend (portal) or rerun than to overpay. After the first real run, use sacct -j --format=...MaxRSS,Elapsed to calibrate future jobs.
4. Sanity-check the submit script: bash -n run.sh (syntax), confirm output/error dirs exist, confirm --gres=gpu:N only on GPU partitions.

This isn't a separate workflow — it's just the default before any non-trivial submission.

Walltime: how to pick -t

It's an upper bound, not a fixed duration — script exits early, resources release immediately. Two forces pull opposite ways:

• Too low → TIMEOUT kills the job mid-run.
• Too high → lower scheduling priority, longer pending.

Billing is on actual elapsed time, not requested walltime, so overestimating doesn't cost more — just slower to start. Aim for ~1.5–2× your real estimate.

First-time estimation:

1. Time the smallest unit on debug — 1 epoch / 1 iter / small slice:

``bash srun -p debug -n 8 --gres=gpu:1 -t 00:30:00 time python train.py --epochs 1 ` Then total ≈ unit_time × units × 1.5`.

1. Calibrate from local: A800 ≈ 1.5–2× a 4090; A40 ≈ 0.5–0.8× a 4090; HPC2 CPU ≈ 5–20× a laptop (per-core × core count).
2. Always checkpoint in long jobs (torch.save per epoch, etc.) — a TIMEOUT only costs the unsaved tail.
3. After the first real run, sacct -j --format=JobID,Elapsed,Timelimit shows actual vs requested → tighten next time.

Format: -t 30 (30 min) · -t 02:00:00 (HH:MM:SS) · -t 1-12:00:00 (D-HH:MM:SS). Use a dash, not space, between days and time.

Common gotchas

• Wrong partition for GPU: --gres=gpu:1 on a CPU partition → job fails or pends forever. Match partition to resource.
• -n vs -c: -n = number of tasks (MPI ranks), -c/--cpus-per-task = threads per task. For a single multi-threaded process use -n 1 -c 8, not -n 8.
• Output file path not writable: -o/-e paths are evaluated at submission time. Relative paths resolve against the submission directory unless -D overrides.
• Modules not loaded inside the script: login shell ≠ batch shell. module load … (or source ~/.bashrc) inside the script if you need it.
• Walltime too short: job gets TIMEOUT killed mid-run. Bump it, but not absurdly — pending priority depends on requested resources.
• squeue shows nothing after sbatch: the job already finished (or failed instantly). Check sacct -j and the error file.

When NOT to use this skill

• On HPC Phase 1 (hpc1login) — that's LSF, commands are jsub / jjobs / jctrl / jhist with #BSUB directives.
• For the K8s / AI container workflow on HPC2 — that's a separate path (/docs/hpc12/k8s/...), not Slurm.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: isPANN
• Source: isPANN/hkustgz-hpc-skills
• License: MIT

Install and usage instructions live in the source repository linked above.