Garnatxa Hpc

Garnatxa HPC Cluster — I2SysBio (UV/CSIC)

Garnatxa is a shared SLURM cluster at the Institute for Integrative Systems Biology (I2SysBio, joint UV/CSIC, Valencia). Use this skill to give the user concrete, copy-pasteable answers grounded in how Garnatxa is actually configured — not generic SLURM advice.

Cluster facts at a glance

• Login host: garnatxa.srv.cpd (SSH only). Prompt after login: [USERNAME@master ~]$.
• Scheduler: SLURM. Module manager: Lmod.
• External access: blocked. Connect from inside the UV network or via VPN

(i2sysbio.ovpn with Garnatxa credentials, or vpn_uv_es.ovpn for UV staff).

• Tape access host: merlot (you ssh merlot from the login node).
• Self-hosted GitLab: (LDAP tab, Garnatxa creds).
• Support / tickets: ; email i2sysbiohpc@uv.es.
• Compute: 14 compute nodes (cn00-cn13), 1232 hardware threads, mixed 64/80/128-CPU

generations, ~18 TB RAM, ~20.5 TFLOPS. No GPUs — Garnatxa is a CPU-only cluster.

• Storage: CEPH (4.1 PB raw, ~3.5 PB usable); tape library is LTO-9 16 TB tapes

(~720 TB online; /tape2// is only readable by the owning group).

Partitions and the QoS table — pick the right one or jobs sit forever

| Partition | Use for | Time | Default mem | Notes | |-----------|---------|------|-------------|-------| | interactive | Interactive shells, light work | up to 1 d | 4 GB | Use interactive command. Max 30 GB, 20 CPUs. Runs on nodes merlot, subirat. | | global | All sbatch jobs | up to 15 d | 2 GB | Pick QoS from table below. 14 compute nodes cn00-cn13 (1232 hardware threads, mixed 64/80/128-CPU). | | tape | Tape archive ops (from merlot) | up to 7 d | – | tapecopy submits to this — you don't usually submit here yourself. |

| QoS | Max time | Max CPU (user) | Max RAM (user) | Priority | |-----|----------|----------------|----------------|----------| | short | 1 d | 200 | 1300 GB | 1000 | | medium | 7 d | 150 | 700 GB | 750 | | long | 15 d | 100 | 360 GB | 500 | | long-mem | 15 d | 80 | 1300 GB | 250 | | extra | 15 d | 400 | 2800 GB | 1000 (request via ticket only — same priority as short/interactive) |

Per-user totals across QoS: 1000 running jobs, 5000 array tasks max. Per-QoS CPU/RAM caps are enforced (e.g. running on short caps you at 200 CPUs and 1300 GB).

Job-priority formula: AGE + FAIRSHARE + JOB SIZE + QOS PRIORITY. Heavy recent users get demoted by fairshare. extra is reserved for justified urgent jobs.

How to use this skill

This skill turns you into an operator on the cluster, not just an info source. If the user has SSH access configured locally (ssh garnatxa works without a password), reach into the cluster directly to diagnose and act, rather than asking the user to run commands and paste output back.

Canonical operator guide: read [references/operator-loop.md](references/operator-loop.md) first when you're asked to do anything on the cluster. It covers preflight, the read-only diagnostic loop, the submit/wait/diagnose pattern, pipeline mode, when to confirm, and recovery from common failures.

Default operating mode (summary)

1. Preflight every session using [assets/preflight.sh](assets/preflight.sh)

— one SSH probe that verifies VPN, key auth, Garnatxa-only tooling, and the merlot hop. Map the failure mode to a fix instead of looping retries.

1. Reach into the cluster directly when it would help. Read-only

diagnostics are safe to run without asking: squeue -u $USER, squeue_, sacct, sacct_, sinfo, sshare, sprio, scontrol show job, reading slurm-*.out/.err, module avail.

1. Confirm before destructive or shared-impact actions — scancel,

rm -rf outside /tmp, large sbatch submissions, tape writes, shell config edits, GitLab pushes, anything on someone else's /storage/.

1. Read only the reference file(s) you need. The references are

self-contained deep dives; don't load them all eagerly.

1. Answer with concrete Garnatxa-specific commands and outputs — right

hostnames, right partitions/QoS, right module load names — not generic SLURM/Linux. When writing a job script, start from an asset in assets/.

1. Push the user toward good citizenship: realistic resource requests

(≥ 75 % CPU and memory efficiency), no heavy work on the login node, no Git pushes > 10 MB or with data files, no Docker (use Singularity), and the required acknowledgment in papers.

Operator-mode gotchas you must internalise

These come from running the skill against the real cluster — they bite agents who don't know about them:

• module is NOT available in non-interactive SSH. Wrap module calls in

bash -lc: ssh garnatxa 'bash -lc "module load anaconda && mamba env list"'. Inside an sbatch script module works directly.

• /scr is Ceph-shared (not node-local) and is never auto-cleaned. Use

/scr/$USER/ and clean up at end of job. $SCRATCH and $XDG_CACHE_HOME are both empty inside jobs — don't rely on them.

• /tape2 is only mounted on merlot. Tape work always starts with

ssh merlot. tapecopy -l works for any user; ls /tape2/ only for the owning group.

• **Compute-node SSH (ssh cn07 …) only works while you have a live job

there.** Nodes self-identify as osd00..osd13 (alias of cn00..cn13); resolve via scontrol show job | grep NodeList=.

• **scontrol show job only finds running or very recently finished

jobs.** For historical jobs use sacct -j -P --format=workdir%80.

• Log filenames are user-defined. Don't assume slurm-.out — read

the sbatch script's --output= or find near the WorkDir.

• **sacct_ and squeue_ print harmless tput warnings under

non-interactive SSH.** Prepend TERM=xterm for clean output.

• **source .../conda.sh; conda activate X 2>/dev/null || true is a

trap.** If X doesn't exist, activation silently fails and python3 quietly resolves to base (no error, just a ModuleNotFoundError later for packages you "know" are installed). Drop the || true, or verify the env exists first with conda env list.

• bash -lc poisons conda envs via the auto-loaded gcc module. The

login shell auto-loads an old gcc module (e.g. gcc/9.4.0) that prepends /opt/ohpc/.../gcc/9.4.0/lib64 to LD_LIBRARY_PATH. Its ancient libstdc++.so.6 then shadows the newer one your conda env ships, and compiled extensions blow up with ImportError: ... libstdc++.so.6: version GLIBCXX3.4.29' not found (numpy ≥2, torch, scipy, …). It is **not** a broken install — the same env imports fine in a real job. To run / verify a conda env, call its python by **full path with NO login shell**: ssh garnatxa '/storage//envs/myenv/bin/python -c "import numpy"' — not ssh garnatxa 'bash -lc "...python..."'. sbatch scripts are already safe (they call the env python directly and load no modules). If you genuinely need a login shell, module purge first, or export LDLIBRARYPATH=/lib:$LDLIBRARY_PATH. Corollary: only use bash -lc over SSH when you actually need module`; never wrap conda-env python in it.

• Lmod module names are case-sensitive. module avail SAMTOOLS

prints samtools/1.21 (case-insensitive substring match), but module load SAMTOOLS errors with "unknown module" — copy the name exactly as module avail printed it. The install dir /storage/apps/SAMTOOLS/ being uppercase is unrelated to the modulefile name.

Typical diagnostic loop (read-only, run freely)

ssh garnatxa '
  echo "=== queue ===";            squeue -u $USER --long
  echo "=== live efficiency ===";  squeue_ -u $USER
  echo "=== recent (7d) ===";      sacct -u $USER --starttime=$(date -d "-7 days" +%F) \
                                          --format=jobid,jobname%20,state,elapsed,reqcpu,reqmem,maxrss,exitcode
  echo "=== finished efficiency ==="; sacct_ -b -u $USER
  echo "=== fairshare ===";        sshare -u $USER
'

From that one round-trip you can diagnose: stuck PD jobs (look at REASON), under-utilization (CPU/MEM efficiency srun your_command

For OpenMP/threaded code, raise `--cpus-per-task` (keep `--ntasks=1`) and set
`export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK`. For MPI, raise `--ntasks` and
use `mpirun -np $SLURM_NTASKS …` after `module load openmpi4`.

### Interactive session

```bash
interactive                       # 2 CPUs, 4 GB, 12 h
interactive -c 6 -m 30G -t 24:00:00   # 6 CPUs, 30 GB, 24 h

Mamba env

module load anaconda
mamba create -n myenv python=3
mamba activate myenv
mamba install -c bioconda samtools

> Never mix module load and mamba activate inside the same > sbatch job — pick one source of binaries.

Move data on/off the cluster

# Local -> Garnatxa
scp -O ./file.txt USER@garnatxa.srv.cpd:./
rsync --inplace --progress --partial --append -av ./mydir USER@garnatxa.srv.cpd:.

# Garnatxa -> local
scp -O USER@garnatxa.srv.cpd:./mydir/file.txt .

Archive to tape (LTO-9)

ssh merlot                       # tape ops happen on merlot, not master
tapecopy -l                      # list tapes and available space
tapecopy path/to/dir             # submits a SLURM job; check jobtape_*.out
tapecopy "out/*.sai"             # quote wildcards
# Recall:
cp /tape2/XXX006L9/home/user/test/out/file.sai /home/user/test/out

Communication style

• The audience is mostly researchers, not sysadmins. Use plain language, but

keep commands exact — they will be copy-pasted.

• When asked "how do I run X?", produce a complete sbatch script, not a

fragment, because users are often new to SLURM.

• When a user describes a problem (a stuck job, an OOM, a wrong output dir),

diagnose with the Garnatxa-specific tooling first: squeue_ and sacct_ for efficiency, scontrol show job for stuck PDs, plotjob for over time.

• When unsure about a detail, point the user to the live docs at

or to a support ticket — better than guessing.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: TianYuan-Liu
• Source: TianYuan-Liu/garnatxa-hpc-skill
• License: MIT

Install and usage instructions live in the source repository linked above.