Software Environment

Modified

April 11, 2024

Abstract

The cluster group provides multiple so called Virtual Application Environments aka VAEs tailored for many use-cases among the GSI user community. The following section provides an overview to this infrastructure and illustrates how to use the Spack HPC package manager to load required application software.

Prerequisites

The following material assumes your familiarity with the Linux command-line interface (CLI) 1 and a basic understanding of a command line shell 2 environment like Bash or Zsh. If this is new to you, we recommend to read the very beginner friendly introduction to this topic Effective Shell 3 by Dave Kerr and The Linux Command Line 4 by William Shotts. Shotts book is freely available under the Creative Commons license and has been translated into multiple languages. Furthermore it is recommended to know the basic usage of a text-editor like Vim or Emacs.

Besides the already mentioned prerequisites there are more widely used tools for scientific computing environments. Your selection of these tools may vary depending on your use-case. On common denominator for most software projects in this environment is the use of the Git version control system 5. Most scientific software frameworks and libraries require the user to have a understanding of a programming language. In scientific computing Python and C++ are very popular. An excellent introduction into the methodologies and basic tools for scientific computing are available from the Software Carpentry Foundation 6. Depending on your scientific field you may find more specific introduction material from The Carpentries Community 7.

Virtual Application Environment

The Virgo cluster is build with the incentive to align its software stack with the cluster computing infrastructure build by a majority of scientific research institutions. The operating system platform is build with RockyLinux 8. Slurm is used as workload management system and supports the allocation of containers. All containerized applications use Apptainer 9 as container run-time engine. Multiple containerized application environments called Virtual Application Environment (VAE) provide a pre-build selection of software. Users access a specific VAE by login to a corresponding submit node:

Virtual Environment Support Until
VAE’22 2024/06
VAE’23 2025
VAE’24 2026

It is required to use SSH key-authentication in order to logon to a submit node of the Virgo cluster. Make sure to use a supported default-shell, otherwise SSH login will not work.

Login to an VAE will print the absolute path to the container launched for reference.

» ssh vae23.hpc.gsi.de
Slurm Cluster – Virgo 2.0 Submit Node
Container launched: /cvmfs/vae.gsi.de/vae23/containers/vae23-user_container_20230308T0124.sif

Containers will be transparently started for your login sessions as well as for your jobs executed on the cluster. Note that each login runs in a dedicated container. If you need multiple terminal sessions, which are also known as shells, it is recommend to use a terminal-multiplexer.

VAEs allow decoupling of the user application environments from the underling system software stack used to operate on the compute hardware. This allows great options for cluster administrators to utilize the most optimal software components to support the hardware and to update/upgrade the infrastructure as required. Furthermore VAEs do enable multiple distinct application environments tailored to specific use-cases to coexist on the same infrastructure. These VAEs can be continuously developed with minimal dependencies to the underlying systems software stack. Environment variables describing the VAE container:

Variable Description
APPTAINER_CONTAINER Absolute path to the container image
APPTAINER_NAME Name of the container image

Spack Packages

Software is made available on the Virgo compute cluster using the Spack 10 package manager. Spack builds software packages with optimisation for multiple computer microarchitectures 11 like AMD Zen or Intel Broadwell. List available software in a virtual application environment (VAE) using following command:

# list packages
spack find

# details infromation about a package (i.e openmpi)
spack find -v openmpi

This prints all packages with their respective microarchitecture and compiler 12, e.g. GCC from the GNU Compiler Collection 13. Print the location’s absolute path of a package:

spack location -i openmpi target=$(spack arch -t)

# note that following command evaluates to the
# microarchitecture of your current host
spack arch -t

Load a package into the current environment. If multiple versions of a specific package are available, it is required to provide a version for loading the package:

spack load openmpi target=$(spack arch -t)

# load a package including all dependencies
spack load -r openmpi target=$(spack arch -t)

# load a specific version
spack load openmpi@3.1.0 target=$(spack arch -t)

Remove a package from your environment:

# list loaded packages
spack find --loaded

# remove a package
spack unload openmpi

# remove all packages
spack unload --all

Spack unload --all removes the command-line interface to Slurm from the environment also. This is required to interact with the cluster controller using the sinfo, sbatch and similar commands. Make sure to load the Spack slurm package again if required.

Footnotes

  1. Command-line Interface, Wikipedia
    https://en.wikipedia.org/wiki/Command-line_interface↩︎

  2. Shell (Computing), Wikipedia
    https://en.wikipedia.org/wiki/Shell_(computing)↩︎

  3. Effective Shell, Dave Kerr
    https://effective-shell.com
    https://github.com/dwmkerr/effective-shell↩︎

  4. The Linux Command Line, William Shotts
    http://www.linuxcommand.org/tlcl.php↩︎

  5. Version Control, Wikipedia
    https://en.wikipedia.org/wiki/Version_control↩︎

  6. Software Carpentry Lessons
    Software Carpentry Foundation
    https://software-carpentry.org/lessons↩︎

  7. The Carpentries Community Developed Lessons
    Carpentries Communit
    https://carpentries.org/community-lessons↩︎

  8. Rocky Linux, RESF Project
    https://rockylinux.org↩︎

  9. Apptainer Documentation, Linux Foundation
    https://apptainer.org/docs/user/latest/↩︎

  10. Basic Usage, Spack Documentation
    https://spack.readthedocs.io/en/latest/basic_usage.html↩︎

  11. Computer microarchitecture, Wikipedia
    https://en.wikipedia.org/wiki/Microarchitecture↩︎

  12. Compiler, Wikipedia
    https://en.wikipedia.org/wiki/Compiler↩︎

  13. GNU Compiler Collection, GNU Project
    https://gcc.gnu.org/↩︎