4. HPC software
4.1. Environment modules (Lmod)
All Software in HPC is managed by, and made accessible with, Environment modules (Lmod). It enables dynamic modification of your environment to make software, libraries and applications available for you to use.
4.1.1. Listing available software
To see all available modules, run module avail
[abc123@login7(eureka) ~]$ module avail
------------------------------------ /opt/pkg/modulefiles ------------------------------------------------------
R/3.5.2 (D) freesurfer/6.0.0 (D)
anaconda2/4.3.1 glib/2.12.1
anaconda3/4.3.1 hdf5/1.10.1+intel-xe_2017_3
castep/16.1.1/gcc-4.8.5+intel-xe_2015_6+openmpi-1.10 lammps/2017/mpi+intel-xe_2016_4
castep/17.2.1/gcc-4.8.5+intel-xe_2016_4+openmpi-1.10+avx lammps/2018/mpi+intel-xe_2017_3
castep/17.2.1/gcc-4.8.5+intel-xe_2016_4+openmpi-1.10 (D) platformmpi/9.1
freesurfer/5.3.0
————————————————————————————————— /opt/proprietary-apps/modulefiles ----------------------------------
ansys/18.0 lsdyna/hybrid-double-9.1+platformmpi-9.1 lsdyna/smp-single-10.1 (D)
gamess-uk/8.2 lsdyna/hybrid-single-9.1+platformmpi-9.1 lsopt/5.2.1
gaussian/09 lsdyna/hybrid-single-10.0+platformmpi-9.1 matlab/R2012a-CR
intel/Intel_Parallel_Suite/xe_2015_6 lsdyna/mpp-double-latest.dev+platformmpi-9.1 matlab/R2017a (D)
intel/Intel_Parallel_Suite/xe_2016_4 lsdyna/mpp-double-10.0+platformmpi-9.1 vasp/5.4.4-intel2017a
intel/Intel_Parallel_Suite/xe_2017_3 (D) lsdyna/mpp-single-10.0+platformmpi-9.1 vasp/5.4.4 (D)
-------------------------------- /opt/eb-pkg/ebadmin/modules/all ------------------------------------------------
ACTC/1.1-intel-2016a damageproto/1.2.1-intel-2016a
Autoconf/2.69-foss-2016b eudev/3.1.5-intel-2016a
Autoconf/2.69-GCCcore-6.3.0 expat/2.1.0-intel-2016a
Autoconf/2.69-intel-2016a expat/2.1.1-intel-2016a
Autoconf/2.69 (D) expat/2.2.0-foss-2016b
Automake/1.15-foss-2016b expat/2.2.0-GCCcore-6.3.0
Automake/1.15-GCCcore-6.3.0 expat/2.2.0-intel-2016a (D)
4.1.2. Using a software package
4.1.2.1. Loading an environment module
To use a particular software package you will need to load its environment module with the module load <modulename> command.
# Loading up the Intel_Parallel_Suite/xe_2017_3 compiler module.
[abc123@login7(eureka) ~]$ module load intel/Intel_Parallel_Suite/xe_2017_3
# Loading up the Lammps 2018.
[abc123@login7(eureka) ~]$ module load lammps/2018/mpi+intel-xe_2017_3
# Loading up cp2k version 6.1 openmp.
[abc123@login7(eureka) ~]$ module load cp2k/6.1-openmp
4.1.2.2. Show currently loaded modules
To see what modules are loaded, run module list.
[abc123@login7(eureka) ~]$ module list
Currently Loaded Modules:
1) lammps/2018/mpi+intel-xe_2017_3 2) intel/Intel_Parallel_Suite/xe_2017_3 3) cp2k/6.1-openmp
Caution
Sometimes additional modules may get loaded that you didn’t expect, these are dependencies
e.g. Loading up lammps/2018/mpi+intel-xe_2017_3 will also load up intel/Intel_Parallel_Suite/xe_2017_3
since it needs its mpi libraries available to execute.
4.1.2.3. Check a module has loaded correctly
To check a module has loaded an executable, run which <executable>.
You can check a module has loaded properly by checking if the correct executable has been made available.
[abc123@login7(eureka) ~]$ module load lammps/2018/mpi+intel-xe_2017_3
[abc123@login7(eureka) ~]$ which lmp_intel_cpu_intelmpi
/opt/pkg/apps/lammps/2018-mpi/src/lmp_intel_cpu_intelmpi
[abc123@login7(eureka) ~]$ module load intel/Intel_Parallel_Suite/xe_2017_3
[abc123@login7(eureka) ~]$ which icc
/opt/proprietary-apps/Intel_Parallel_Suite/xe_2017_3/compilers_and_libraries_2017.3.191/linux/bin/intel64/icc
4.1.2.4. Unloading a module
To unload a particular module, run module unload <modulename>.
# Unloading Intel_Parallel_Suite/xe_2017_3.
[abc123@login7(eureka) ~]$ module unload intel/Intel_Parallel_Suite/xe_2017_3
# Unloading Lammps 2018.
[abc123@login7(eureka) ~]$ module unload lammps/2018/mpi+intel-xe_2017_3
# Unloading cp2k version 6.1 openmp.
[abc123@login7(eureka) ~]$ module unload cp2k/6.1-openmp
4.1.2.5. Unloading all modules
To unload ALL modules loaded, run module purge.
[abc123@login7(eureka) ~]$ module list
Currently Loaded Modules:
1) lammps/2018/mpi+intel-xe_2017_3 2) intel/Intel_Parallel_Suite/xe_2017_3 3) cp2k/6.1-openmp
[abc123@login7(eureka) ~]$ module purge
[abc123@login7(eureka) ~]$ module list
No modules loaded
4.1.2.6. Check the status of a module
To show what a module is doing, run module show <modulename>.
[abc123@login7(eureka) ~]$ module show lammps/2018/mpi+intel-xe_2017_3
--------------------------------------------------------------------------------------------------
/opt/pkg/modulefiles/lammps/2018/mpi+intel-xe_2017_3:
--------------------------------------------------------------------------------------------------
whatis(" Adds lammps (mpi) version 2018 to your environment variables ")
prepend_path("PATH","/opt/pkg/apps/lammps/2018-mpi")
prepend_path("PATH","/opt/pkg/apps/lammps/2018-mpi/src")
load("intel/Intel_Parallel_Suite/xe_2017_3")
help([[ Adds lammps (mpi) version 2018 to your environment variables
]])
4.1.3. Using module “collections”
Module collections are used to automatically load a set of modules while also preserving the order in which they are loaded.
To save a “collection” of modules in their correct “load order” run the commands below. This will create a “default” collection.
First load all the necessary modules in the right order, then run:
module saveTo restore the “default” collection in future and auto-load in the right order, run:
module restore
You can also create “non-default” module collections with a custom names.
Load all the necessary modules in the right order, then run:
module save <collection_name>To restore the above created module collection run:
module restore <collection_name>
List all the modules in a collection with:
module describe <collection_name>
List all of your module collections with:
module savelist
4.2. Compilers and libraries
When using compilers and libraries, consistency is required. Random libraries, compilers and software cannot be put together.
For example when using GCC compilers and using a library with it, the library must also be compiled by the same GCC compiler version. It cannot be used with a library compiled with another compiler such as intel or GCC of a different version.
So in the example below, a gfortran compiler can compile code against the HDF5 library which was compiled with gfortran. However, the Intel compiler cannot compile code against the HDF5 library which was compiled with GCC.
If the in HDF5 library was compiled with intel, the intel compiler would be able to compile code against the HDF5 library which was compiled with Intel.
4.3. Requesting new software
You may find that the software you want to use is not currently available on the HPC and in this case you will need to raise a support ticket to request it be installed.
for details on how to do this, see Request new software.
4.4. Using Python on HPC
For those who utilise and develop Python software, we recommend using virtual environments where possible. Virtual environments allow you have multiple different python environments, where you can install and manage whatever packages/libraries you require and they are installed in your own directory. They additionally allow you to load them as and when you need them.
There are two ways to create python environments on Eureka, Python virtual environments using Virtualenvs and Python virtual environments using Conda envs.
These both are via the anaconda3 module. Anaconda is a distribution of the Python and R for scientific computing.
Note
for information on modules, see Environment modules (Lmod).
4.4.1. Python virtual environments using Virtualenvs
Virtualenv is one way to create a python virtual environments. It can be accessed via the anaconda3 module.
First thing to do is load up the anaconda3 module:
[abc123@login7(eureka) ~]$ module load anaconda3
4.4.1.1. Creating a python virtualenv
Then you can create a virtual python environment using the command virtualenv <env name>.
This will create the virtual environment in the same directory you execute the command.
Once created you should unload the the anaconda3 module.
These steps are shown in the example below creating a virtualenv called example_env.
[abc123@login7(eureka) ~]$ virtualenv example_env
Using base prefix '/opt/pkg/apps/Anaconda3/2019.03'
New python executable in /users/abc123/TEST_PYTHON/example_env/bin/python
copying /opt/pkg/apps/Anaconda3/2019.03/bin/python => /users/abc123/TEST_PYTHON/example_env/bin/python
Installing setuptools, pip, wheel...done.
[abc123@login7(eureka) ~]$ ls
example_env
[abc123@login7(eureka) ~]$ module unload anaconda3/2019.03
4.4.1.2. Loading a python virtualenv
Now you can change into the directory for the virtualenv and then change into the bin directory when you need to source the activate file to activate your virtual environment. Once loaded, your command prompt will change to indicate the virtual environment is loaded.
[abc123@login7(eureka) ~]$ ls
example_env
[abc123@login7(eureka) ~]$ cd example_env/
[abc123@login7(eureka) example_env]$ ls
bin include lib pip-selfcheck.json
[abc123@login7(eureka) example_env]$ cd bin/
[abc123@login7(eureka) bin]$ source activate
(example_env) [abc123@login7(eureka) bin]$
4.4.1.3. Customising your python virtualenv
Once activated you can begin installing any package you wish your environment using pip,
all packages installed when inside the virtual environment will be installed to the example_env in your space.
For example we could install the mpi4py python package to the environment.
(example_env) [abc123@login7(eureka) bin]$ pip install mpi4py
Processing /users/abc123/.cache/pip/wheels/18/e0/86/2b713dd5121e59818871d6/mpi4py-3.0.3-cp37-cp37m-linux_x86_64.whl
Installing collected packages: mpi4py
Successfully installed mpi4py-3.0.3
4.4.1.4. De-activating a python virtualenv
To close a virtualenv you simply need to use the command deactivate.
This will unload the environment and it will be indicated by the removal of the name of the virtualenv you had loaded from the command prompt.
(example_env) [abc123@login7(eureka) ~]$ deactivate
[abc123@login7(eureka) ~]$
Tip
You can create and delete as many python environments as you like, each with their own purposes, or versions of packages, libraries and applications.
4.4.2. Python virtual environments using Conda envs
An alternative to virtualenvs is Conda envs, which operate in a very similar way and can be accessed via the anaconda3 module.
Note
for information on modules, see Environment modules (Lmod).
First thing to do is load up anaconda3 module:
[abc123@login7(eureka) ~]$ module load anaconda3
4.4.2.1. Creating a Conda env
If its your first time, using Conda envs you will need to run conda init bash then log out and then back in to the terminal.
This will add the Conda environment permanently and is indicated by the (base) next to your command prompt.
Then you can create a conda virtual python environment using the command conda create -n <env name> pip.
This will create the conda env when you execute the command and makes pip available in the environment too.
These steps are shown in the example below creating a conda env called myenv:
(base)[abc123@login7(eureka) envs]$ conda create -n myenv pip
Collecting package metadata (repodata.json): done
Solving environment: done
.....
.....
.....
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base)[abc123@login7(eureka) envs]$
4.4.2.2. Loading a Conda env
Once created we can then activate the Conda env using conda activate <env name>.
Once activated, your command prompt will change to indicate the virtual environment is loaded.
(base) [abc123@login7(eureka) ~]$ conda activate myenv
(myenv) [abc123@login7(eureka) ~]$
4.4.2.3. Customising your Conda env
Once activated you can begin installing any package you wish your environment using pip or conda,
all packages installed when inside the conda env will be installed to the directory /users/<user name>/.conda in your space.
For example, we could install the numpy python package to the environment using pip, as shown below:
(myenv) [abc123@login7(eureka) ~]$ pip install numpy
Collecting numpy
Downloading https://files.pythonhosted.org/packages/41/38/b278d96baebe9b83ed5/numpy-1.18.1-cp38-cp38-manylinux1_x86_64.whl (20.6MB)
|████████████████████████████████| 20.6MB 11.7MB/s
Installing collected packages: numpy
Successfully installed numpy-1.18.1
or you could install packages using conda with the command conda install -n <env name> <package name>.
(myenv) [abc123@login7(eureka) ~]$ conda install -n myenv scipy
Collecting package metadata (repodata.json): done
Solving environment: done
.....
.....
The following packages will be downloaded:
package | build
---------------------------|-----------------
scipy-1.3.2 | py38h7c811a0_0 18.2 MB
------------------------------------------------------------
Total: 18.2 MB
.....
.....
.....
Downloading and Extracting Packages
scipy-1.3.2 | 18.2 MB | ############################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Note
When using Conda envs its advisable to exclusively use either conda or pip to install packages.
4.4.2.4. De-activating a Conda env
To deactivate your Conda env you simply need to use the command conda deactivate. This will unload the environment and it will be indicated by the removal of the name of the Conda env you had loaded from the command prompt.
(myenv) [abc123@login7(eureka) ~]$ conda deactivate
(base) [abc123@login7(eureka) ~]$
More information can be found from Anaconda’s Website on Conda Envs.
Tip
You can create and delete as many Conda python environments as you like, each with their own purposes, or versions of packages, libraries and applications.
Note
We will not install any Python packages to the system’s Python on EUREKA
4.5. Using R on HPC
For those who utilise and develop R software, we recommend using virtual environments where possible. Virtual environments allow you have multiple different R environments, where you can install and manage whatever packages/libraries you require and they are installed in your own directory. They additionally allow you to load them as and when you need them.
4.5.1. R virtual environments
Conda envs provide a way to have R virtual environments. It can be accessed via the anaconda3 module.
Note
for information on modules, see Environment modules (Lmod).
Start by loading the anaconda3 module:
[abc123@login7(eureka) ~]$ module load anaconda3
4.5.1.1. creating a Conda env
If its your first time, using conda envs you will to run conda init bash then log out and in back to the terminal.
This will add the Conda environment permanently and is indicated by the (base) next to your command prompt.
Then you can create a Conda virtual R environment using the command conda create -n <env name> r-essentials r-base.
These steps are shown in the example below creating a Conda env called my_r_env:
(base) [abc123@login7(eureka) ~]$ conda create -n my_r_env r-essentials r-base
Collecting package metadata (repodata.json): done
Solving environment: done
.....
.....
.....
.....
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate my_r_env
#
# To deactivate an active environment, use
#
# $ conda deactivate
4.5.1.2. Activating a Conda env
Once created we can then activate the Conda Env using conda activate <env name>.
Once activated, your command prompt will change to indicate the virtual environment is loaded.
(base) [abc123@login7(eureka) ~]$ conda activate my_r_env
(my_r_env) [abc123@login7(eureka) ~]$
4.5.1.3. Customising a Conda env
Once activated you’ll have access to the R and Rscript commands, and you can begin installing any package you wish in your environment using conda.
All packages installed when inside the Conda env will be installed to the directory /users/<user name>/.conda in your home directory.
For example, we could install the rstudio package to the environment using conda install -c r <package name>, as shown below:
(my_r_env) [abc123@login7(eureka) ~]$ conda install -c r rstudio
Collecting package metadata (repodata.json): done
Solving environment: done
.....
.....
.....
r-rsconnect-0.8.13 | 550 KB | ############################################################ | 100%
r-bit-1.1_14 | 243 KB | ############################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
4.5.1.4. De-activating a conda env
To deactivate your Conda env you simply need to use the command conda deactivate. This will unload the environment and it will be indicated by the removal of the name of the Conda env you had loaded from the command prompt.
(my_r_env) [abc123@login7(eureka) ~]$ conda deactivate
(base) [abc123@login7(eureka) ~]$
More information can be found from Anaconda’s Website on Conda Envs.
Tip
You can create and delete as many Conda R environments as you like, each with their own purpose, versions of packages, libraries and applications.
Note
We will not install any R packages to the system’s R on EUREKA