OpenMP

Modified

November 10, 2023

Abstract

This section introduces the basics of executing a program using the OpenMP (Open Multi-Processing) library for shared-memory multiprocessing.

Hello World

This example uses a simple OpenMP ¹ hello-world program for illustration. The following C code exemplifies a basic program utilizing OpenMP to parallelize a code block with a directive #pragma omp parallel ²:

// omp-hello.c
#include <stdio.h>
#include <omp.h>

int main() {
  printf ( "OMP_GET_NUM_THREADS=%d\n", omp_get_num_procs ( ) );
  printf ( "OMP_GET_MAX_THREADS=%d\n", omp_get_max_threads ( ) );
  // parallel region with default number of threads
  #pragma omp parallel
  {
    // library function to return a thread ID
    int id = omp_get_thread_num();
    printf("Hello from thread %d\n", id);
  }
  // end of parallel region
}

Compile an executable using the gcc compiler, option -fopenmp enables handling of OpenMP directives #pragma omp by the gcc compiler ³.

# compile a program with OpenMP support
gcc -fopenmp $LUSTRE_HOME/src/omp-hello.c -o $LUSTRE_HOME/bin/omp-hello

Variable	Description
`OMP_NUM_THREADS`	Specifies number of threads to use in parallel regions
`OMP_PROC_BIND=TRUE`	Threads should not be moved between processors (thread affinity policy)

The above OpenMP environment variables are used for initialization:

# execute the program with two threads
OMP_NUM_THREADS=2 $LUSTRE_HOME/bin/omp-hello

Batch Job

In order to start an OpenMP program on the cluster, it is required to specify the level of parallelism using a command option to sbatch. In this particular case --cpus-per-task defines the number of cores to allocate for a job. Use one of the output environment-variables e.g. $SLURM_JOB_CPUS_PER_NODE, which is set by Slurm to initialize OpenMP. Store this code in a file called omp-wrapper. Remove the $OMP_DISPLAY_ENV in case you want less verbose output.

#!/bin/bash

#SBATCH -J omp
#SBATCH -o %j.log

if [ -n "$SLURM_JOB_CPUS_PER_NODE" ]; then
  omp_threads=$SLURM_JOB_CPUS_PER_NODE
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads
export OMP_PROC_BIND=TRUE
#  print out at runtime OpenMP specification
export OMP_DISPLAY_ENV=true

srun -- $@

Submit the OpenMP program with the wrapper script:

sbatch --partition debug \
       --job-name omp_hello \
       --chdir $LUSTRE_HOME \
       --no-requeue \
       --ntasks 1 \
       --cpus-per-task 8 \
       -- $LUSTRE_HOME/omp-wrapper \
          $LUSTRE_HOME/bin/omp-hello

You should find an output similar to the following in the log-file (cf. io-redirection):

OPENMP DISPLAY ENVIRONMENT BEGIN
  _OPENMP = '201511'
  OMP_DYNAMIC = 'FALSE'
  OMP_NESTED = 'FALSE'
  OMP_NUM_THREADS = '8'
  OMP_SCHEDULE = 'DYNAMIC'
  OMP_PROC_BIND = 'TRUE'
  OMP_PLACES = '{1},{65},{2},{66},{3},{67},{4},{68}'
  OMP_STACKSIZE = '0'
  OMP_WAIT_POLICY = 'PASSIVE'
  OMP_THREAD_LIMIT = '4294967295'
  OMP_MAX_ACTIVE_LEVELS = '2147483647'
  OMP_CANCELLATION = 'FALSE'
  OMP_DEFAULT_DEVICE = '0'
  OMP_MAX_TASK_PRIORITY = '0'
OPENMP DISPLAY ENVIRONMENT END
OMP_GET_NUM_THREADS=8
OMP_GET_MAX_THREADS=8
Hello from thread 0
Hello from thread 1
Hello from thread 2
Hello from thread 7
Hello from thread 6
Hello from thread 4
Hello from thread 3
Hello from thread 5

Footnotes

OpenMP Project
https://www.openmp.org ↩︎
OpenMP 4.5 API C/C++ Syntax Reference Guide, OpenMP Project
https://www.openmp.org/wp-content/uploads/OpenMP-4.5-1115-CPP-web.pdf ↩︎
Options Controlling C Dialect, GCC, GNU Project
https://gcc.gnu.org/onlinedocs/gcc/C-Dialect-Options.html ↩︎