Mila Run Jobs

Running Jobs on the Mila Cluster

This skill guides users through running jobs on the Mila cluster: either interactive development sessions using mila code (VSCode on a compute node), or batch jobs submitted with sbatch.

Base policies

At the start of each response, use the Skill tool with skill: "mila-base" to load and apply all shared policies before proceeding with the workflow below.

Reference documentation

Interactive development: https://docs.mila.quebec/gettingstarted/myfirst_job/

Batch training: https://docs.mila.quebec/gettingstarted/trainfirst_model/

Discover documentation

Use the WebSearch tool with this query to find the current URLs of the pages above:

site:docs.mila.quebec "__skill-mila-run-jobs"

The query should return two pages (interactive and batch). Use those URLs in the WebFetch steps below. If the search returns no results, fall back to the hardcoded URLs in "Reference documentation".

Workflow

Step 1: Identify what the user wants

Determine which mode the user needs:

• Interactive / exploratory — they want to write and run code with an

editor, debug interactively, or check GPU availability. → Use mila code.

• Batch / training — they want to submit a job that runs unattended,

train a model overnight, or use sbatch. → Use sbatch.

If unclear, ask: "Do you want to work interactively with VSCode on a compute node, or submit a batch job that runs on its own?"

Step 2: Fetch the documentation

• For interactive: use the WebFetch tool to fetch the interactive URL discovered

above (fallback: https://docs.mila.quebec/gettingstarted/myfirst_job/)

• For batch: use the WebFetch tool to fetch the batch URL discovered above

(fallback: https://docs.mila.quebec/gettingstarted/trainfirst_model/)

• If the user asks about both, use the WebFetch tool to fetch both pages.

Step 3a: Guide through interactive development (mila code)

1. From the local machine, create the project directory on the cluster:

``bash ssh mila 'mkdir -p CODE/my_first_job' ``

1. Start VSCode on a GPU compute node:

``bash mila code CODE/my_first_job --alloc --gres=gpu:1 --cpus-per-task=2 --mem=16G --time=01:00:00 ` Everything after --alloc` is passed to Slurm. Adjust resources as needed.

1. Wait for the allocation to be granted and VSCode to open, connected to

the compute node.

1. In VSCode, create pyproject.toml and main.py in the project folder.
2. Open the VSCode integrated terminal (Terminal → New Terminal) — this

terminal runs on the compute node — and run: ``bash uv run python main.py ``

1. When done, close VSCode and press Ctrl+C in the terminal where

mila code is running to end the session and release the allocation.

Key points:

• mila code requires milatools and a working SSH connection (see the

mila-connect-cluster skill).

• VSCode must be installed locally; Cursor is also supported.
• Adjust --gres=gpu:1, --mem, and --time for the actual workload.

Step 3b: Guide through batch job submission (sbatch)

1. From the local machine, create the project directory on the cluster:

``bash ssh mila 'mkdir -p CODE/train_first_model' ``

1. Open a CPU node for editing (faster to allocate than a GPU node):

``bash mila code CODE/train_first_model --alloc --cpus-per-task=2 --mem=16G --time=01:00:00 ``

1. In VSCode, create three files: job.sh, pyproject.toml, and main.py.

job.sh does three things:

• #SBATCH directives — request resources (GPU, CPUs, memory, time). Typical

values: --ntasks=1, --cpus-per-task=4, --gpus-per-task=:1. Use --mem-per-gpu (not --mem) when requesting GPUs.

• Data staging — copy the dataset from /network/datasets/ into

$SLURM_TMPDIR/data/ before training. Example: ``bash mkdir -p $SLURM_TMPDIR/data cp /network/datasets/cifar10/cifar-10-python.tar.gz $SLURM_TMPDIR/data/ ` Compute nodes read from $SLURM_TMPDIR` much faster than from network storage.

• Run the training script — srun uv run python main.py.

1. Submit the job from the VSCode terminal:

``bash sbatch job.sh ``

1. Monitor the job:

• Queue status: squeue --me
• Output: once running, watch slurm-.out for logs.
• ReqNodeNotAvail status: means no matching node is available

right now; the job waits in the queue automatically. If waiting too long, resubmit requesting a different GPU type.

Key points:

• $SLURM_TMPDIR is fast local storage on the compute node, available

only during the job. Always stage datasets there before training.

• srun inside job.sh runs the command within the allocated resources.
• The Mila CIFAR-10 dataset is at /network/datasets/cifar10/.

Step 4: Answer follow-up questions

Common questions:

• "How do I check if my job is still running?" — squeue --me
• "How do I cancel a job?" — scancel
• "How do I request multiple GPUs?" — --gres=gpu:2 (or more) in the

mila code command or #SBATCH directive.

• "Can I use Cursor instead of VSCode?" — Yes. mila code supports

Cursor and other compatible editors.

• "What is $SLURM_TMPDIR?" — Fast local storage on the compute node,

unique to each job. Use it to stage datasets for fast I/O during training.

• "How do I see job output?" — Open slurm-.out in the project

directory once the job starts.

Step 5: Point to further resources

For more complex job patterns (multi-GPU, distributed training, environment variables, partitions):

• https://docs.mila.quebec/Userguiderunningcode/
• https://docs.mila.quebec/Userguide_multigpu/

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mila-iqia
• Source: mila-iqia/skills
• License: MIT

Install and usage instructions live in the source repository linked above.