Self Diagnosing Resource Use

Self-Diagnosing Resource Use

Rule: after every serious job, check what you actually used and right-size the next job. Do this on your own initiative and report the result in plain language — the user is usually a researcher who will not think to ask "was that wasteful?" Translate the numbers ("used 6 of 64 GB; drop --mem to 12G next time"), don't just paste seff output.

Completed job accounting

sacct -j JOBID --format=JobID,JobName,Elapsed,AllocCPUS,TotalCPU,MaxRSS,State --units=G

Note: MaxRSS is blank on the parent job row — it only populates on the .batch and step (.0, .1, …) rows. Read the .batch line for actual peak memory, not the summary line.

For a friendlier summary:

seff JOBID

Caveat: seff samples coarsely and reports CPU Utilized: 00:00:00 / 0.00% for sub-minute jobs even when they did real work (verified on this cluster). Ignore the CPU-efficiency numbers on jobs shorter than ~1–2 minutes — they reflect the sampling miss, not waste. Trust them only on jobs long enough to be sampled.

Interpret CPU use

Approximate CPU efficiency:

TotalCPU / (Elapsed × AllocCPUS)

Rules of thumb:

• >50%: reasonable for CPU-bound jobs.
• 10–50%: maybe I/O-bound or over-requested.
• logs/gpu${SLURMJOB_ID}.csv &

Rules of thumb:

- 0 MB used: your process is not using the GPU.
- Low GPU utilization and high VRAM: model loaded but waiting on data/CPU/network.
- Sustained /dev/null
find /gpfs/scratch60/$USER -mtime +30 -size +1G -ls 2>/dev/null | head
find output -type f | wc -l

If a job creates thousands of files, strongly consider redesigning the output storage. See [using the filesystem](../using-the-filesystem/SKILL.md) for Parquet, JSONL, zip, local /tmp, and atomic-write patterns.

Agent-friendly checkup script

#!/bin/bash
set -euo pipefail

jobid=${1:?usage: ./checkup.sh JOBID}

echo "=== Job accounting ==="
sacct -j "$jobid" --format=JobID,JobName,Elapsed,AllocCPUS,TotalCPU,MaxRSS,State

echo ""
echo "=== seff ==="
seff "$jobid" 2>/dev/null || true

echo ""
echo "=== Current jobs ==="
squeue -u "$USER" -o "%.8i %.9P %.20j %.2t %.10M %.6D %.4C %.10m %R"

echo ""
echo "=== Scratch usage ==="
du -sh "/gpfs/scratch60/$USER" 2>/dev/null || true
find "/gpfs/scratch60/$USER" -mtime +30 -size +1G -ls 2>/dev/null | head || true

echo ""
echo "=== GPU status, if on GPU node ==="
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv 2>/dev/null || true

Quick rules table

| Metric | Good | Wasteful | Fix | |---|---:|---:|---| | CPU efficiency | >50% | 25% requested | 30% training | 10,000/job | Parquet/HDF5/zip | | Time used | close to request | tiny fraction | request shorter time |

Checklist

• [ ] Checked sacct or seff after the job.
• [ ] Reduced future CPU/memory/time requests if over-requested.
• [ ] Checked nvidia-smi for GPU jobs.
• [ ] Checked scratch usage and old files.
• [ ] Confirmed output file count is reasonable.
• [ ] Made the next job shorter or more resumable where possible.

Further reading

• Slurm sacct — format strings, fields like MaxRSS, TotalCPU, Elapsed.
• Slurm squeue — format strings and reason codes for pending jobs.
• nvidia-smi reference — --query-gpu, logging utilization, MIG.
• py-spy — py-spy dump --pid PID for stuck Python processes.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yale-som-hpc
• Source: yale-som-hpc/claude-code-marketplace
• License: Unlicense

Install and usage instructions live in the source repository linked above.