Python in Containers#

On some of our clusters, such as AISurrey, containers are required to run jobs. On other systems like Eureka2, containers are optional but useful for advanced use cases.

We use Apptainer (formerly Singularity), which is available on all compute nodes and compatible with Docker. For general usage instructions, see Containerisation: Docker & Apptainer.

Containers: What, Why, and When#

A container is a self-contained Linux environment that bundles:

  • A specific Python version

  • Any libraries or dependencies your code needs

  • Optional OS-level tools or system libraries (e.g. CUDA)

It behaves like a lightweight virtual machine, but runs faster and without root permissions.

Examples: Python in containers with Slurm#

Below are some examples of Slurm submission scripts that show common patterns for running Python inside containers.

Run a Python script from the current directory

Use this when your script (e.g. my_script.py) is in the same directory you submit the job from.

#!/bin/bash
#SBATCH --job-name="basic-python"
#SBATCH --partition=2080ti
#SBATCH --gpus=1
#SBATCH --mem=16G
#SBATCH --time=00-00:20:00
#SBATCH -o slurm.%N.%j.out
#SBATCH -e slurm.%N.%j.err

# This pulls a Python 3.11 image from DockerHub and runs your script
apptainer exec docker://python:3.11 python my_script.py

Explanation:

  • The container runs in a temporary environment and sees only certain directories by default.

  • Data written by the script is saved if it’s in a directory visible to the container, see next example.

Access files using --bind

Use --bind to mount your working directory inside the container.

apptainer exec --bind $PWD:/app \
   docker://python:3.11 python /app/my_script.py

Explanation:

  • --bind $PWD:/app makes your current directory ($PWD) available inside the container at /app.

  • Your Python script and data will be visible and accessible inside the container.

  • Output files written to /app are stored in your original working directory (not inside the container).

  • Without --bind, the container may not have access to your code or data. Any files created may be lost when the container exits, unless explicitly written to a bound directory.

Use an internal container from our own registry

Example using a PyTorch container from our own registry:

apptainer exec --bind $PWD:/app \
   oras://container-registry.surrey.ac.uk/shared-containers/pytorch:latest \
   python /app/my_script.py

Explanation:

  • The oras://... syntax pulls the container from our internal registry.

  • To learn more about searching for or adding containers to the registry, see Container registry.

Passing parameters or training with checkpointing

A Slurm array job where you pass parameters to your Python script (e.g. learning rate, output dir).

#!/bin/bash
#SBATCH --job-name="param-sweep"
#SBATCH --partition=2080ti
#SBATCH --gpus=1
#SBATCH --array=0-2
#SBATCH --mem=10G
#SBATCH --time=00-00:10:00
#SBATCH -o slurm.%N.%j.out
#SBATCH -e slurm.%N.%j.err

lrs=("0.001" "0.01" "0.1")
lr=${lrs[$SLURM_ARRAY_TASK_ID]}
ckpt_dir=$PWD/models_$lr
script=$PWD/my_script.py

apptainer exec --bind $PWD:/app \
   oras://container-registry.surrey.ac.uk/shared-containers/pytorch:latest \
   python /app/my_script.py \
   --ckpt-dir /app/models_$lr --batch-size 32 --epochs 2 --lr $lr --resume-training

Explanation:

  • Uses a Slurm job array to iterate over parameters (learning rate).

  • Each job has its own checkpoint/output directory.

  • All files are accessible inside the container thanks to --bind $PWD:/app.

  • Arguments are passed normally to the Python script as in any CLI run.

  • This example assumes that the script has checkpointing capability.

Note that additional examples can be found on our Gitlab instance, which requires a University of Surrey login.

Quick Tips#