Conda#

Conda is a tool that helps you manage different Python versions and packages in isolated environments. We recommend it for most users working on our HPC systems.

This approach makes it easier to:

  • Avoid conflicts between projects

  • Use modern and consistent Python packages

  • Follow research best practices around reproducibility and portability

  • Share your environment across machines or with collaborators

1. Setting up Conda#

Before using Conda, e.g. on Eureka2, look for and load one of the Anaconda3 modules:

Load Anaconda#
[ab1234@login1 ~]$ module spider Anaconda3

------------------------------------------------------------------------------------
 Anaconda3:
------------------------------------------------------------------------------------
  Description:
     Built to complement the rich, open source Python community, the Anaconda platform
     provides an enterprise-ready data analytics platform that empowers companies to
     adopt a modern open data science analytics architecture.

  Versions:
       Anaconda3/2021.05
       Anaconda3/2021.11
       Anaconda3/2022.10
       Anaconda3/2024.02-1


[ab1234@login1 ~]$ module load Anaconda3/2024.02-1

This gives you access to the conda command.

Tip

You need to load the module each time you log in or open a new terminal session.

For more about modules, see Environment modules (Lmod).

Finalising the setup#

To set up conda in your shell properly, it needs to be initialised once:

[ab1234@login1 ~]$ conda init bash

Then log out and log back in (re-load the Anaconda module if you are using the Lmod system). Your shell will now support Conda properly.

You’ll see something like:

(base) [ab1234@login1 ~]$

This shows you are in the default Conda environment (base).

2. Creating and using environments#

We recommend creating your own environments, rather than using the base one.

Here, we create an environment myenv using Python 3.11:

Create and use a basic environment#
conda create -n myenv python=3.11
conda activate myenv

# Search for a package
conda search matplotlib

# Install with conda
conda install matplotlib

# Or install with pip
conda install pip
pip install seaborn

# Deactivate when done
conda deactivate

Note

By default, packages are installed from the main channel. We recommend switching to conda-forge for most use cases, see tips below.

To install specific version, impose version number constraints, or install packages or whole environments from files, see the tabs below:

Install specific versions (examples)#
conda create -n myenv python=3.12 numpy=2.0 matplotlib>=3.0

3. Tips and best practices#

Project-specific environments#

Use one environment per project to avoid conflicts, to easily move your workloads across compute environments, and to reproduce your work or share it later.

Use conda-forge packages#

The conda-forge channel is a community-maintained collection of packages. It is more up-to-date and avoids Anaconda’s license restrictions (see more).

Configure Conda to use the conda-forge channel#
conda config --add channels conda-forge
conda config --set channel_priority strict

List environments#

You can view all your conda environments with:

conda env list

Remove environments#

Free up disk space by deleting unused environments:

conda remove -n myenv --all

Export / import environments#

To save your environment or share it with others, export it to a .yml file:

conda env export > environment.yml

You (or someone else) can recreate the environment later:

conda env create -f environment.yml

Manage storage usage#

By default, conda stores environments in your home directory under ~/.conda/envs. If you create many environments or install large packages, this can quickly use up your home directory quota.

Custom storage location#

To store environments in another location, like your scratch directory, e.g. on Eureka2 (replace ab1234 with your user name):

Configure Conda to save environments to a custom directory#
cd /parallel_scratch/ab1234/
mkdir conda_envs
conda config --append envs_dirs /parallel_scratch/ab1234/conda_envs

Check where conda looks for environments with:

conda config --show envs_dirs

Note

You can manually copy environments to a new location, but it is usually safer to recreate them from your environment.yml file.

Clear Cached data#

To free up disk space, you can delete Conda’s package cache, which stores downloaded .tar.bz2 and .conda files:

conda clean --all

This removes unused packages and caches. You can also preview what will be deleted first with:

conda clean --all --dry-run

Use recent Python versions#

We recommend selecting Python versions for your projects that offer long term security and performance updates, see the Status of Python Versions page for details.

Further reading#