Molgpu

MolGPU Cluster Reference

Shared GPU cluster for the group. All nodes on private network (use MIT VPN to connect off campus).

Machines

| Node | CPUs | GPUs | VRAM/GPU | RAM | Storage | |------|------|------|----------|-----|---------| | molgpu01 | 64 cores (2x AMD 3970X) | 2x RTX 3090 Ti | 24 GB | — | — | | molgpu02 | 128 cores (2x AMD 3995WX) | 4x RTX A5000 | 24 GB | — | — | | molgpu03 | 64 cores (AMD 3995WX) | 4x RTX A5000 | 24 GB | 512 GB | 2x 2TB SSD | | molgpu04 | 64 cores (AMD 3995WX) | 3x RTX 3090 | 24 GB | 512 GB | 1TB + 2TB SSD (encrypted at /data) | | molgpu05 | 32 cores (AMD 5975WX) | 2x RTX 4090 | 24 GB | 256 GB | 4TB SSD | | molgpu06 | 32 cores (AMD 5975WX) | 2x RTX 4090 | 24 GB | 256 GB | 4TB SSD | | molgpu07 | 128 cores (AMD 5995WX) | 4x RTX 4090 | 24 GB | 512 GB | 2TB SSD | | molgpu08 | 52 cores (2x Xeon 6230R) | 8x RTX 2080 Ti | 11 GB | 512 GB | 1TB SSD | | moldata01 | — (Synology DS3622xs) | — | — | — | 100TB HDD + 800GB SSD cache |

Quick Selection Guide

• 4x GPU (fastest): molgpu07 (4x 4090, best single-GPU perf + most cores)
• 4x GPU (fallback): molgpu02 or molgpu03 (4x A5000)
• 2x GPU: molgpu05 or molgpu06 (2x 4090)
• Many small jobs: molgpu08 (8x 2080 Ti, only 11 GB each)

Access & SSH Setup

Full hostnames are molgpu01.mit.edu through molgpu08.mit.edu. Same credentials across all nodes. SSH keys set up on one node work on all others (shared home dir).

Add this to ~/.ssh/config for easy access:

Host *
  AddKeysToAgent yes
  UseKeychain yes   # macOS only — saves key password in keychain
  ForwardAgent yes

Host molgpu0?
  Hostname %h.mit.edu
  User 
  ForwardAgent yes
  IdentityFile 

This lets you ssh molgpu07 instead of ssh user@molgpu07.mit.edu -i /path/to/key. ForwardAgent means you don't need to store your private key on the server to use GitHub via SSH.

To add access from a new client, append your public key to ~/.ssh/authorized_keys on any molgpu node.

Change password with passwd (applies to all nodes, must be mixed case + numeric, 8+ chars).

Shared Filesystem

All molgpu machines mount the same NFS home directory at /mnt/home/ (81.8 TB total). Files written on one machine are immediately visible on all others — no rsync needed between molgpus. Only rsync from your local machine to any one node.

Sensitive data goes on the encrypted partition at /mnt/encrypted/ (16.4 TB). Protect sensitive folders with chmod 700.

Important: Avoid reading/writing a large number of files in parallel — this slows NFS for everyone. Reduce parallel I/O workers if NFS feels slow.

Always back up code and important checkpoints. There is no guarantee storage will never fail.

Conda / Mamba

Global conda is available on molgpu02–06 (no need to install your own):

/opt/miniconda3/bin/conda init

Global mamba (faster drop-in replacement for conda) on molgpu02:

# after conda init:
mamba init

uv is also available at ~/.local/bin/uv if installed per-user.

Dashboard

View current load across all machines: http://molgpu01:8088/

Common Commands

# transfer files from local machine (only need to hit one node, shared fs does the rest)
rsync -az --exclude '.venv*' --exclude '__pycache__' --exclude '.git' \
  . molgpu07:~/project-name/

# check GPU availability on a node
ssh molgpu07 nvidia-smi --query-gpu=index,name,memory.used,memory.free --format=csv,noheader

# scan all machines at once
for h in molgpu0{1..8}; do echo "=== $h ===" && ssh $h nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader 2>/dev/null; done

# always use tmux for long-running jobs (nohup is NOT enough — child processes get SIGHUP)
ssh molgpu07
tmux new-session -s myjob
# ... launch job ...
# detach: Ctrl-B then D
# reattach later: tmux attach -t myjob

# ONE-TIME per user per node: enable linger so the tmux server survives
# when your login session ends. without this, systemd-logind kills the
# whole user slice (including detached tmux sessions) when your last
# ssh closes.
loginctl enable-linger
loginctl show-user $USER --property=Linger  # should read Linger=yes

# always redirect stderr (some frameworks log errors only to stderr)
python train.py > /tmp/train.log 2>&1

# kill orphaned GPU processes after a crash
nvidia-smi  # find PIDs
kill -9 

Known Issues

• NFS stale file handles. Caches stored in home dir (~/.triton/cache, ~/.cache/torch/) can go stale when switching between machines. Fix: use local disk for caches (export TRITON_CACHE_DIR=/tmp/triton_cache) or nuke the cache (rm -rf ~/.triton/cache).
• All GPUs are 24 GB (except molgpu08 at 11 GB).
• Torch / CUDA version mismatch. NVIDIA driver on molgpu nodes is CUDA 12.4 (as of 2026-04). Recent PyTorch (2.10+) ships only cu130 wheels and will silently fall back to CPU with a driver-too-old warning. If torch.cuda.is_available() is False after a fresh install, pin torch to a cu124 build:

``bash ~/.local/bin/uv pip install --force-reinstall torch==2.5.1 \ --index-url https://download.pytorch.org/whl/cu124 ` Verify with python -c "import torch; print(torch.cuda.is_available())"`.

LLM/VLM-Specific Notes

Multi-GPU Training (DDP)

# set HF_HUB_OFFLINE=1 to avoid HF Hub race conditions during multi-GPU load
# (download model first, then set this)
HF_HUB_OFFLINE=1 accelerate launch --num_processes=4 train.py

# QLoRA + DDP: use device_map={"": local_rank} (not None or "auto")

vLLM Inference on 24 GB GPUs

VLMs are especially memory-hungry. If OOM during warmup or generation, tune:

• gpu_memory_utilization: lower from 0.9 to 0.35–0.7
• max_num_seqs: lower from 256 to 4–16
• enforce_eager=True: skips CUDA graphs, saves ~2 GB VRAM
• tensor_parallel_size: split model across GPUs

Model Size Reference

| Size | bf16 VRAM | QLoRA NF4 | Min GPUs (inference) | |------|-----------|-----------|---------------------| | 4B | ~8 GB | ~5 GB | 1 | | 8B | ~16 GB | ~8-10 GB | 1 | | 12B | ~24 GB | ~12-14 GB | 2 | | 70B | ~140 GB | ~42 GB | 8+ |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mcox3406
• Source: mcox3406/claude-comp-chem-skills
• License: MIT

Install and usage instructions live in the source repository linked above.