Distributed Environment Modules
Overview
This page covers the distributed environment modules on OSG Connect and how to use them to provide workflows with a consistent environment when running.
Introduction
Environment modules have historically been used in HPC environments to provide users with an easy way to access different versions of software and to access various libraries, compilers, and software. OSG Connect uses environment modules for similar reasons as well as to provide consistent environments to test and run workflows. OSG Connect uses an implementation of the module system called Lmod to provide the typical module services as well as to allow modules to record dependencies. All modules and binaries for applications and libraries are published on OASIS, a distributed filesystem that is available on most of the sites that run OSG Connect jobs, providing a consistent environment on these resources. This allows for users to test workflows on the OSG Connect login nodes and then submit the same workflows without making any changes.
Using Distributed Environment Modules on OSG Connect
Initialization
The first step to using distributed environment modules is to initialize the module system. This step consists of sourcing a shell specific file that then adds the module
command to your environment. For example, if you use bash then this can be done by running the following:
For other shells such as sh, zsh, tcsh, csh, etc., you would replace bash with the shell name (e.g. zsh).
Using Distributed Environment Modules
Once the distributed environment modules system is initialized, you can then uses module avail
to see available modules, e.g.:
In order to load a module, you should run module load [modulename]
, e.g.
Some modules require other modules to be loaded, if that occurs then you'll get an error message indicating this:
In addition, some modules such as the all-pkgs for python won't be available until the relevant packages are loaded, e.g.:
To unload a package and remove it from the environment, you should use module unload [modulename]
, e.g.:
To see currently loaded modules, you can should use module list
, e.g.:
Finally, module help
will give you more detailed information.
Available Software
A list of current modules currently available is given below:
Software | Modulename | Notes |
---|---|---|
Python 2.7 | python/2.7 | Interpreter for the python programming language |
Modules for Python 2.7 | all-pkgs | Only available after python 2.7 module loaded, requires atlas and lapack modules to be loaded. Provides Numpy, Scipy, Scikit-learn, matplotlib, Pandas, and NLTK |
Python 3.4 | python/3.4 | Interpreter for the python programming language |
Modules for Python 3.4 | all-pkgs | Only available after python 3.4 module loaded, requires atlas and lapack modules to be loaded. Provides Numpy, Scipy, Scikit-learn, matplotlib, and Pandas |
ATLAS Libraries | atlas | ATLAS 3.10.1 libraries. Libraries for linear algebra. |
LAPACK libraries | lapack | LAPACK 3.5.0 libraries. Libraries for linear algebra. |
NAMD | namd/2.9 | NAMD 2.9. Molecular dynamics simulator. |
gromacs | gromacs/4.6.5 | Gromacs molecular dynamics simulator. Requires fftw/fftw-3.3.4-gromacs, lapack, and atlas modules to be loaded. |
FFTW | fftw/fftw-3.3.4-gromacs | FFTW is a library for fast fourier transforms. FFTW 3.3.4 library compiled with gromacs specific options. |
blender | blender | Blender 3D rendering software. |
R | R/3.1.1 | The R language interpreter is a domain specific language designed for statistical analysis. |