Modules

Introduction

The Environment Modules package is a tool that simplify shell
initialization and lets users easily modify their environment during
the session with module files.

Each module file contains the information needed to configure the
shell for an application. Once the Modules package is initialized, the
environment can be modified on a per-module basis using the module
command which interprets module files. Typically module files instruct
the module command to alter or set shell environment variables such as
PATH, MANPATH, etc. Module files may be
shared by many users on a system and users may have their own
collection to supplement or replace the shared module files.

Modules can be loaded and unloaded dynamically and atomically, in
an clean fashion.

Example

By default, the compiler gcc is the one installed by the
system.

$ type gcc

gcc is /usr/bin/gcc

You can decide to use a specific one installed with a module.

$ module load compiler/gcc/10.1.0
$ type gcc

gcc is /cm/shared/modules/intel/skylake/compiler/gcc/10.1.0/bin/gcc

And decide to switch to another version.

$ module switch compiler/gcc/9.2.0
$ type gcc

gcc is /cm/shared/modules/intel/skylake/compiler/gcc/9.2.0/bin/gcc

and finally to come back to the default system compiler.

$ module unload compiler/gcc/9.2.0
$ type gcc

gcc is /usr/bin/gcc

Other module commands

  • module avail’ → list available modules on the system.
  • module list’ → list currently loaded modules.
  • module purge’ → unload all loaded modules.

Module Naming Policy

To ease module management, they are sorted according to the
architecture of the nodes, and grouped in categories

The different architectures are:

  • generic for modules which can run on all nodes
  • intel/haswell for the nodes miriel and sirocco[01-05]
  • intel/broadwell for the nodes sirocco[07-13]
  • intel/skylake for the nodes bora
  • intel/knightslanding for the nodes kona

When connecting to a node via salloc, the environment
variable MODULEPATH contains the directory
generic and the node-specific directory
manufacturer/chip

Within each architecture, modules are grouped with the following
module naming policy

<category>/<module>/<option>/<version>
the number of options being between 0 and as many as needed.

For example

  • partitioning/scotch/int32/6.0.4
  • partitioning/scotch/int64/6.0.4

Dev and Users Modules

  • Modules managed by the technical team (MPI, GCC and CUDA compilers)
    are available in /cm/shared/modules
  • User-managed modules are available in /cm/shared/dev/modules
  • All users can install modules, one only needs to be added to the Unix group
    plafrim-dev (ticket to plafrim-support AT inria.fr)

How to Create a Module

File System

  • Modules files go
    in /cm/shared/dev/modules/architecture/modulefiles by following the architecture and the naming policies.
  • Installation application files go in /cm/shared/dev/modules/architecture/apps with the same architecture
    and naming policies.
  • For example, let’s install for all nodes the version 1.1.8 of the trace generator application eztrace
    • Install your application in /cm/shared/dev/modules/generic/apps/trace/eztrace/1.1.8/
    • Create the module file /cm/shared/dev/modules/generic/modulefiles/trace/eztrace/1.1.8
proc ModulesHelp { } {
  puts stderr "\tAdds EzTrace 1.1.8 to your environment variables"
}

module-whatis "adds eztrace 1.1.8 trace generator tool to your environment variables"

set             version         1.1.8
set             prefix          /cm/shared/dev/modules/generic/apps/trace/eztrace
set             root            $prefix/$version

#path added in the beginning
prepend-path   CPATH  $root/include
prepend-path   LIBRARY_PATH  $root/lib
prepend-path   LD_LIBRARY_PATH  $root/lib
  • Set the correct permissions
    $ module load tools/module_cat
    $ module_perm /cm/shared/dev/modules/generic/apps/trace/eztrace/1.1.8
    $ module_perm /cm/shared/dev/modules/generic/modulefiles/trace/eztrace/1.1.8
    

Dependencies

If your module depends on other modules and has been compiled with different versions of this module.

First solution

  • Define a single module file /cm/shared/dev/modules/generic/modulefiles/perftools/simgrid/3.24
  • specify the dependency(ies)
    prereq compiler/gcc
    
    $ module show compiler/gcc
    ...
    setenv GCC_VER 9.3.0
    ...
    
  • and use their information
    set prefix /cm/shared/dev/modules/generic/apps/perftools/simgrid/$version/install/gcc_$env(GCC_VER)
    

    Second solution

  • Define two different module files
    • /cm/shared/dev/modules/generic/modulefiles/perftools/simgrid/3.24/gcc_8.2.0
    • /cm/shared/dev/modules/generic/modulefiles/perftools/simgrid/3.24/gcc_9.2.0
  • To avoid loading 2 versions of the same module
    conflict perftools/simgrid
    

What are the useful variables?

Path to development headers (for compiling)

 prepend-path CPATH ...
 prepend-path FPATH ...
 prepend-path INCLUDE ...
 prepend-path C_INCLUDE_PATH ...
 prepend-path CPLUS_INCLUDE_PATH ...
 prepend-path OBJC_INCLUDE_PATH ...

Path to libraries (for linking)

 prepend-path LIBRARY_PATH ...

Path to tools and libraries (for running)

 prepend-path PATH ...
 prepend-path LD_LIBRARY_PATH ...

pkg-config to simplifying build systems (point to directories with .pc files)

prepend-path PKG_CONFIG_PATH $prefix/lib/pkgconfig

Manpages

append-path  MANPATH   $man_path
append-path  MANPATH   $man_path/man1
append-path  MANPATH   $man_path/man3
append-path  MANPATH   $man_path/man7

Some modules define specific variables, likely because one random project ever decided they need them…

setenv            CUDA_INSTALL_PATH   $root
setenv            CUDA_PATH           $root
setenv            CUDA_SDK            $root
prepend-path      CUDA_INC_PATH       $root/include

setenv          HWLOC_HOME          $prefix

setenv         MPI_HOME          $prefix
setenv         MPI_RUN           $prefix/bin/mpirun
setenv         MPI_NAME          $name
setenv         MPI_VER           $version

Which version gets selected by default during load?

The default is in (reverse) alphabetical order

$ module avail formal/sage

— /cm/shared/dev/modules/generic/modulefiles —
formal/sage/7.0 formal/sage/8.9 formal/sage/9.0

$ module load formal/sage
$ module list

Currently Loaded Modulefiles:
1) formal/sage/9.0

Another default version may be enforced. Useful if your last beta
release isn’t stable or backward compatible but still needed
for some hardcore users.

$ cat /cm/shared/dev/modules/generic/modulefiles/hardware/hwloc/.version

#%Module1.0#
set ModulesVersion “2.1.0”

More on environment dependent modules

$ module_grep starpu

runtime/starpu/1.3.2/mpi
runtime/starpu/1.3.2/mpi-fxt
runtime/starpu/1.3.3/mpi
runtime/starpu/1.3.3/mpi-cuda
runtime/starpu/1.3.3/mpi-cuda-fxt
runtime/starpu/1.3.3/mpi-fxt

StarPU has different module files which ONLY differ in
the prereq commands and the prefix setting.

> prereq compiler/cuda/10.1
> prereq trace/fxt/0.3.9
< set     prefix          /cm/shared/dev/modules/generic/apps/runtime/starpu/1.3.3/gcc@9.2.0-hwloc@2.1.0-openmpi@4.0.2
> set     prefix          /cm/shared/dev/modules/generic/apps/runtime/starpu/1.3.3/gcc@8.2.0-hwloc@2.1.0-openmpi@4.0.1-cuda@10.1
> set     prefix          /cm/shared/dev/modules/generic/apps/runtime/starpu/1.3.3/gcc@8.2.0-hwloc@2.1.0-openmpi@4.0.1-cuda@10.1-fxt@0.3.9

Create a module file with all the cases

if {![ is-loaded compiler/gcc ]} {
   module load compiler/gcc
}
if {![ is-loaded hardware/hwloc ]} {perftools
   module load hardware/hwloc/2.1.0
}

conflict runtime/starpu

set cuda ""
set fxt ""
if {[ is-loaded compiler/cuda/10.1 ]} {
  set cuda -cuda@10.1
}
if {[ is-loaded trace/fxt/0.3.9 ]} {
  set fxt -fxt@0.3.9
}

set     name            starpu
set     version         1.3.3

set     prefix /cm/shared/dev/modules/generic/apps/runtime/$name/$version/gcc@$env(GCC_VER)-hwloc@2.1.0-openmpi@4.0.1${cuda}${fxt}

Different environments lead to a different version of StarPU to be used.

1st case

$ module purge
$ module load runtime/starpu/42
$ module list

Currently Loaded Modulefiles:
1) compiler/gcc/9.2.0 2) hardware/hwloc/2.1.0 3) runtime/starpu/42

$ module show runtime/starpu/42


setenv STARPU_DIR /cm/shared/dev/modules/generic/apps/runtime/starpu/1.3.3/gcc@9.2.0-hwloc@2.1.0-openmpi@4.0.1

2nd case

$ module purge
$ module load compiler/gcc/8.2.0 compiler/cuda/10.1 runtime/starpu/42
$ module show runtime/starpu/42


setenv STARPU_DIR /cm/shared/dev/modules/generic/apps/runtime/starpu/1.3.3/gcc@8.2.0-hwloc@2.1.0-openmpi@4.0.1-cuda@10.1

3rd case

$ module purge
$ module load compiler/gcc/8.2.0 compiler/cuda/10.1 trace/fxt runtime/starpu/42
$ module show runtime/starpu/42


setenv STARPU_DIR /cm/shared/dev/modules/generic/apps/runtime/starpu/1.3.3/gcc@8.2.0-hwloc@2.1.0-openmpi@4.0.1-cuda@10.1-fxt@0.3.9

More module commands

https://modules.readthedocs.io/en/latest/modulefile.html

Module tools/module_cat

The module tools/module_cat provides the following tools

  • module_list to list the existing categories
  • module_add, module_init, module_add, module_rm to modify the environment variable MODULEPATH which defines the folders in which to look for modules
  • module_perm to set the correct permissions on a given directory
  • module_search to search all modules whose name have the given string, e.g module_search hwloc