Resource Allocation

Modified

November 9, 2023

Abstract

Following sections describes how users request resource like CPUs, memory of GPUs to be allocated for a compute job. Furthermore the differences between interactive jobs and batch jobs in explained.

Allocations

Users request the allocation of computing resources on behalf of their associated accounts using the salloc ¹, srun ² or sbatch ³ commands:

Command	Interactive	Blocking	Description
`salloc`	yes	yes	Allocate resources and launches a shell.
`srun`	yes	yes	Allocate resources and starts an application.
`sbatch`	no	no	Queues an application for later execution.

A resource allocation specifies a set of resources, e.g. nodes, CPUs, RAM, etc., possibly with some set of constraints, e.g. number of processors per node, maximum runtime and so on. All three commands accept the same set of parameters for resource allocation.

A significant difference is that salloc and srun are interactive & blocking. This means that both are linked to your terminal session, hence bound to your connection to the submit node. Output is directed to your interactive shell running in the terminal session. Losing the connection to the submit node might kill the job.

The sbatch command in contrast **transfers complete control to the cluster controller, and allows you to disconnect from the submit node.

Interactive

The following example uses salloc to request a set of default resources from one of the partitions. The command blocks until resources are allocated:

# start an interactive command interpreter
» salloc --partition=debug
salloc: Granted job allocation 2964352

salloc launches an interactive shell, after resources have been granted by the cluster controller:

# execute an arbitrary command
» cat /proc/cpuinfo | grep 'model name' | sort | uniq
model name      : AMD EPYC 7551 32-Core Processor

# investigate the job configuration
» scontrol show job $SLURM_JOB_ID | grep Time
   RunTime=00:00:51 TimeLimit=00:05:00 TimeMin=N/A
   SubmitTime=2020-08-27T07:23:14 EligibleTime=2020-08-27T07:23:14
   AccrueTime=Unknown
   StartTime=2020-08-27T07:23:14 EndTime=2020-08-27T07:28:14 Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0

Use the exit command to stop the interactive shell and release all allocated resources:

» exit
exit
salloc: Relinquishing job allocation 2964352

srun within a job allocation launches in parallel across all allocated nodes. The execution environment is inherited by all launched processes:

» salloc --partition=debug --nodes=2 --chdir=/tmp -- bash
salloc: Granted job allocation 2964433
# run a command on the node hosting the interactive shell
» hostname
lxbk0595
# run a command on all allocated nodes (in parallel)
» srun hostname
lxbk0595
lxbk0596
# run another command
» srun uptime
 07:37:04 up 56 days, 14:26, 27 users,  load average: 0.28, 0.18, 0.15
 07:37:04 up 56 days, 14:26,  4 users,  load average: 0.03, 0.07, 0.13
# release the resources
» exit
exit
salloc: Relinquishing job allocation 2964433

Each invocation of srun within a job is known as a step. A (compute) job consists of one or more steps, each consisting of one or more tasks, each using one or more processors.

Real Time

Using srun outside of a job allocation from salloc, requests resources specified by command options and waits until resource allocation. As soon as resource are available it automatically launches the specified application.

» srun --partition=debug --nodes=5 --chdir=/tmp -- hostname
lxbk0596
lxbk0597
lxbk0599
lxbk0598
lxbk0600

Once the specified application has finished, resources a relinquished automatically.

The command above is basically is short notation for:

» salloc --partition=debug --nodes=5 \
        srun --chdir=/tmp -- hostname
salloc: Granted job allocation 138
lxbk0596
lxbk0599
lxbk0597
lxbk0598
lxbk0600
salloc: Relinquishing job allocation 138

Batch Jobs

sbatch is used to submit a compute job to the scheduler-queue for later execution. The command exits immediately after the controller has assigned a unique JOBID, and the job is queued by the scheduler. Batch jobs wait in the queue of pending jobs until resources become available.

The compute job is copied to a compute node as soon as resources have been granted by the scheduler. The jobs application is typically launched by a batch script with the resource allocation and resource constrains specified by meta-commands.

A compute job submitted with sbatch has at least one implicit job step, the start of the executable provided as argument to the command.

Following is a very simple example of a batch job executing a simple shell-script:

# simple script executing a couple of commands
cat > $LUSTRE_HOME/sleep.sh <<EOF
#!/usr/bin/env bash
hostname ; uptime ; sleep 180 ; uname -a
EOF
# submit the script above, with the job name "sleep"
sbatch --output='%j.log' --chdir=$LUSTRE_HOME \
       --job-name=sleep -- $LUSTRE_HOME/sleep.sh

Check if the state of the job using the squeue command:

» squeue --format='%6A %8T %8N %9L %o' --name=sleep
JOBID  STATE    NODELIST TIME_LEFT COMMAND
18     RUNNING  lxbk0596 1:57:40   /lustre/hpc/vpenso/sleep.sh

# read the stdout of the job
» cat $LUSTRE_HOME/$(squeue -ho %A -n sleep).log
lxbk0596
 10:08:37 up 1 day,  2:05,  0 users,  load average: 0.00, 0.01, 0.05

Attach to Job

From a submit node you may attach an interactive debugging shell to your running job with the following command:

srun --jobid <running-jobid> [-w <hostname>] -O --pty bash

Option	Description
`--jobid=<jobid>`	Initiate a job step under an already allocated job with given id.
`-O`, `--overcommit`	The instantiated job step and task for the debugging shell do not demand additional resources from the existing allocation (which is usually already used up).
`--pty`	Execute task zero in pseudo terminal mode. Implicitly sets `--unbuffered`. Implicitly sets `--error` and `--output` to `/dev/null` for all tasks except task zero, which may cause those tasks to exit immediately (e.g. shells will typically exit immediately in that situation). This option applies to step allocations.
`-w`, `--nodelist=<hostname>`	Request a specific host. Useful if job allocation spans multiple nodes.

Footnotes

salloc manual page, SchedMD
https://slurm.schedmd.com/salloc.html ↩︎
srun manual page, SchedMD
https://slurm.schedmd.com/srun.html ↩︎
sbatch manual page, SchedMD
https://slurm.schedmd.com/sbatch.html ↩︎

--- title: Resource Allocation date-modified: 2023/11/09 abstract: > Following sections describes how users request resource like CPUs, memory of GPUs to be allocated for a compute job. Furthermore the differences between interactive jobs and batch jobs in explained. --- ## Allocations Users request the allocation of computing resources on behalf of their associated [accounts][SdD97] using the `salloc` [^4g2MI], `srun` [^yDrhe] or `sbatch` [^TOjTZ] commands: [SdD97]: accounts.html [^4g2MI]: `salloc` manual page, SchedMD <https://slurm.schedmd.com/salloc.html> [^yDrhe]: `srun` manual page, SchedMD <https://slurm.schedmd.com/srun.html> [^TOjTZ]: `sbatch` manual page, SchedMD <https://slurm.schedmd.com/sbatch.html> Command | Interactive | Blocking | Description ----------|-------------|----------|------------------------ `salloc` | yes | yes | Allocate resources and launches a shell. `srun` | yes | yes | Allocate resources and starts an application. `sbatch` | no | no | Queues an application for later execution. A resource allocation specifies a **set of resources**, e.g. nodes, CPUs, RAM, etc., possibly with some **set of constraints**, e.g. number of processors per node, maximum runtime and so on. All three commands accept the same set of parameters for resource allocation. A significant difference is that **`salloc` and `srun` are interactive & blocking**. This means that both are linked to your terminal session, hence **bound to your connection to the submit node**. Output is directed to your interactive shell running in the terminal session. Losing the connection to the submit node might kill the job. The **`sbatch`** command in contrast **transfers complete control to the cluster controller, and allows you to disconnect from the submit node. ### Interactive The following example uses `salloc` to request a set of default resources from one of the [partitions][4NXUz]. The command **blocks until resources are allocated**: [4NXUz]: partitions.html ```sh # start an interactive command interpreter » salloc --partition=debug salloc: Granted job allocation 2964352 ``` **`salloc` launches an interactive shell**, after resources have been granted by the cluster controller: ```sh # execute an arbitrary command » cat /proc/cpuinfo | grep 'model name' | sort | uniq model name : AMD EPYC 7551 32-Core Processor # investigate the job configuration » scontrol show job $SLURM_JOB_ID | grep Time RunTime=00:00:51 TimeLimit=00:05:00 TimeMin=N/A SubmitTime=2020-08-27T07:23:14 EligibleTime=2020-08-27T07:23:14 AccrueTime=Unknown StartTime=2020-08-27T07:23:14 EndTime=2020-08-27T07:28:14 Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 ``` Use the **`exit` command to stop the interactive shell** and release all allocated resources: ```sh » exit exit salloc: Relinquishing job allocation 2964352 ``` `srun` within a job allocation **launches in parallel across all allocated nodes**. The execution **environment is inherited** by all launched processes: ```sh » salloc --partition=debug --nodes=2 --chdir=/tmp -- bash salloc: Granted job allocation 2964433 # run a command on the node hosting the interactive shell » hostname lxbk0595 # run a command on all allocated nodes (in parallel) » srun hostname lxbk0595 lxbk0596 # run another command » srun uptime 07:37:04 up 56 days, 14:26, 27 users, load average: 0.28, 0.18, 0.15 07:37:04 up 56 days, 14:26, 4 users, load average: 0.03, 0.07, 0.13 # release the resources » exit exit salloc: Relinquishing job allocation 2964433 ``` ::: {.callout-note appearance="simple"} Each invocation of `srun` within a job is known as a **step**. A (compute) **job** consists of one or more steps, each consisting of one or more **tasks**, each using one or more processors. ::: ### Real Time Using `srun` outside of a job allocation from `salloc`, requests resources specified by command options and **waits until resource allocation**. As soon as resource are available it automatically **launches the specified application**. ```sh » srun --partition=debug --nodes=5 --chdir=/tmp -- hostname lxbk0596 lxbk0597 lxbk0599 lxbk0598 lxbk0600 ``` Once the specified application has finished, **resources a relinquished automatically**. The command above is basically is short notation for: ```sh » salloc --partition=debug --nodes=5 \ srun --chdir=/tmp -- hostname salloc: Granted job allocation 138 lxbk0596 lxbk0599 lxbk0597 lxbk0598 lxbk0600 salloc: Relinquishing job allocation 138 ``` ## Batch Jobs `sbatch` is used to submit a compute job to the scheduler-queue for later execution. The command exits immediately after the controller has assigned a unique JOBID, and the job is queued by the scheduler. Batch jobs **wait in the queue of pending jobs** until resources become available. The compute job is copied to a compute node as soon as resources have been granted by the scheduler. The jobs application is typically launched by a batch script with the [resource allocation][qp4C8] and [resource constrains][M28Ty] specified by [meta-commands][sR7IP]. [M28Ty]: resource-constrains.html [qp4C8]: resource-allocation.html [sR7IP]: environment.html#meta-commands ::: {.callout-note appearance="simple"} A compute job submitted with `sbatch` has at least **one implicit job step**, the start of the executable provided as argument to the command. ::: Following is a very simple example of a batch job executing a simple shell-script: ```sh # simple script executing a couple of commands cat > $LUSTRE_HOME/sleep.sh <<EOF #!/usr/bin/env bash hostname ; uptime ; sleep 180 ; uname -a EOF # submit the script above, with the job name "sleep" sbatch --output='%j.log' --chdir=$LUSTRE_HOME \ --job-name=sleep -- $LUSTRE_HOME/sleep.sh ``` Check if the state of the job using the `squeue` command: ```sh » squeue --format='%6A %8T %8N %9L %o' --name=sleep JOBID STATE NODELIST TIME_LEFT COMMAND 18 RUNNING lxbk0596 1:57:40 /lustre/hpc/vpenso/sleep.sh # read the stdout of the job » cat $LUSTRE_HOME/$(squeue -ho %A -n sleep).log lxbk0596 10:08:37 up 1 day, 2:05, 0 users, load average: 0.00, 0.01, 0.05 ``` ## Attach to Job From a submit node you may attach an interactive debugging shell to your running job with the following command: ```sh srun --jobid <running-jobid> [-w <hostname>] -O --pty bash ``` Option | Description ---------------------|------------------------ `--jobid=<jobid>` | Initiate a job step under an already allocated job with given id. `-O`, `--overcommit` | The instantiated job step and task for the debugging shell do not demand additional resources from the existing allocation (which is usually already used up). `--pty` | Execute task zero in pseudo terminal mode. Implicitly sets `--unbuffered`. Implicitly sets `--error` and `--output` to `/dev/null` for all tasks except task zero, which may cause those tasks to exit immediately (e.g. shells will typically exit immediately in that situation). This option applies to step allocations. `-w`, `--nodelist=<hostname>` | Request a specific host. Useful if job allocation spans multiple nodes.