Python

Python on Deucalion

Much of the programming for Artificial Intelligence is done in Python. On HPC systems, this often involves using either virtual environments or Conda environments. However, these environments place a significant burden on shared filesystems because they contain thousands of small files. A typical Python virtual environment consumes a large number of inodes while taking up relatively little disk space — the opposite of what HPC filesystems are usually optimized for, as they are designed for a small number of large files.

Singularity containers offer an effective workaround. A container can be as large as needed, but still encapsulate the many small files that make up a Python environment, thereby reducing inode usage and easing pressure on the shared filesystem.

Python environments usually default to write the packages at the home folder. However, Deucalion's home offers less performance than the projects folder (/projects) and so it's recommended that these environments go into that folder.

This page documents the process to create simple singularity containers for python workflows or in alternative create python environments (either cuda or pip) in the appropriate folders.

Singularity Containers

To create a singularity container, you must run singularity build --fakeroot test.sif test.def, where test.sif is the name you want to give the container and test.def is your definitions file, telling the instructions to create the container application. In this case, since we want a python container you can follow this definitions file, that bootstraps your container from a pre-existing python container from Dockerhub:

test.def (for pip environments)

Bootstrap: docker
From: python:3.9
Stage: build
%environment
   export LC_ALL=C
%post
  pip install --upgrade pip
  pip install -r /path/to/requirements.txt
%test
  python -V

In this case we are installing every package in /path/to/requirements.txt. To get this file for your specific workload you must take an environment that you are already using and run pip freeze > requirements.txt. After doing so, you can transfer the requirements.txt file to Deucalion and then run the singularity build command described above. Because Deucalion is an heterogenous system, you must build the container in the same architecture where you are planning to run it. For example, if you want to use it in the ARM partition you must start a session a node in the dev-arm partition and run the build command (if you do it in the login nodes the container will not run in the ARM partition).

Pip virtual environments

Virtual environments must also be built in the same architecture where they will run. You need to load the python module you want to run first (e.g. module load Python/3.12.3-GCCcore-13.3.0). You should change directory to your project folder. To check the ones you have access to run quotaprojects. After being there you can run python -m venv <venv_name>, where you should substitute <venv_name> with the name you want. We recommend adding a suffix to identify the architecture, such as venv_x86 or venv_arm. After that, activate that environment using source /path/to/<venv_name>/bin/activate and do all pip install commands. Any time you want to use that environment you just need to put source /path/to/<venv_name>/bin/activate in your jobscript (no need to load modules).

Conda Environments

Conda environments can also exist in Deucalion. However, by default conda will install all packages in your home folder. To bystep this, please rewrite the ~/.condarc file to look like this:

~/.condarc

channels:
  - defaults
  - conda-forge
  - pytorch
  - nvidia
pip_interop_enabled: true
channel_priority: strict

envs_dirs:
    - /projects/<your_project>/.conda/conda_envs 
pkgs_dirs:
    - /projects/<your_project>/.conda/conda_pkgs

You will have to substitute <your_project> for your project folder. After that you can use conda as usual.