Submit Nodes

Modified

August 7, 2024

Abstract

This section overviews how to access the HPC infrastructure at GSI Darmstadt. It covers details about login to submit nodes using the SSH (Secure SHell) command-line tools.

Overview

The HPC infrastructure is only accessible from within the GSIs internal networks.

Virgo is located in a dedicated high-speed InfiniBand network shielded from external access by a firewall and network gateways. Login to a cluster submit node from outside the GSI networks is facilitated by a pool of login nodes used as SSH proxies. Submit nodes in the Virgo cluster support different software environments described in more details in the Virtual Application Environment (VAE) section. Login to the cluster uses the SSH application. Password authentication is disabled on submit nodes, use of SSH key-authentication is required.

# Example of access to a VAE using the login pool as SSH proxy 
ssh -J lxlogin.gsi.de vae24.hpc.gsi.de

Following is a list of login addresses to the Virgo cluster. Note that these addresses provide access over a load-balancer service, which redirects to a pool of submit nodes:

Submit Nodes	VAE	Support EOF
`virgo.hpc.gsi.de`	No	-
`vae23.hpc.gsi.de`	Yes	2025
`vae24.hpc.gsi.de`	Yes	2026

Login to one of the VAEs using the corresponding host alias vae*.hpc.gsi.de
The host alias virgo.hpc.gsi.de provides login to the bare-metal environment. These are typically used for building and using custom containers.
Individual experiments and user groups distribute software using CVMFS. Consult the list of software repositories for further information.

Submit nodes are not used as workstation!

Submit node are not intended to be used as continues interactive working environment. We strongly discourage the execution of an application directly on submit nodes for a long period of time (e.g. several hours). The expected usage of submit nodes is code compilation, a short test for execution and subsequent job submission.

Security Advice

A fully functional access to an Linux user login on a large computer cluster over SSH is a very valuable resource for malicious activities. Therefore, it is very important to pay attention to security. Unfortunately, most of the security-settings cannot be technically enforced or only at a great cost to usability. Therefore your engagement is important!

Please make sure to comply with following recommendation:

Use strong passwords for your Linux account (16+ random characters) and your SSH private key (cf. key-authentication).
Never use unencrypted private keys!
If possible do not use SSH agent-forwarding, since this exposes your credentials on a remote computer. Instead use the mechanism described in proxy jump.
Only use X11-forwarding if necessary.

Secure SHell

Access the infrastructure using a Secure SHell (SSH)¹. The following list includes all commands and configuration files we recommend to get familiar with:

ssh                       # login to a remote server
scp                       # copy files from/to a remote server
rsync                     # ^^ (delta sync.)
ssh-keygen                # generate an ssh public/private key pair
ssh-copy-id               # copy a public key to a server
ssh-agent                 # daemon temporarily storing the private key
ssh-add                   # add a private key to an SSH agent
ssh-keyscan               # manage host fingerprints
sshuttle                  # private network tunnel over SSH
sshfs                     # mount remote file-systems over SSH
~/.ssh/config             # client configuration
~/.ssh/authorized_keys    # authorized keys on a server
~/.ssh/known_hosts        # list of server fingerprint for authentication

The OpenSSH² tool is pre-installed on nearly all Linux distributions, as well as Apple Mac OS X. On Windows users have to access the SSH tools from the Windows Subsystem for Linux³ using the Windows Terminal⁴. As reference documentation we recommend the excellent manual pages from the OpenSSH⁵ project.

Login Nodes

Login Pool	Description
`lxlogin.gsi.de`	Minimal environment, supposed to be used as SSH proxy jump
`lxpool.gsi.de`	User environment includes access to the network storage for example the user home-directory (`/u/$USER`)

The GSI provides a pool of public Linux login nodes lxlogin.gsi.de that serve as an entry point to systems behind the GSI firewall. Before you continue make sure that you can access these nodes using your personal GSI Linux user name. If you do not have your user name and password yet, follow the instructions under request access. Establish a remote terminal session to the login nodes using the ssh command with following arguments:

Arguments	Description
`[user@]hostname[.domain]`	hostname of the login node for example `lxlogin` domain of GSI `gsi.de` prefixed by your Linux user name

# `$user` is your GSI Linux user name
ssh $user@lxlogin.gsi.de

The system will ask you to accept the remote computer into the list of known hosts. Type in your password before you can login. Use exit to logout from the remote computer and end the SSH session. Close hanging SSH connections using an escape sequence ~.

Host Verification

The first connection to a submit node includes a process to check for the authenticity of the remote server⁶. The user will need to compare a server fingerprint (derived from the server host key) with the fingerprints listed in the table below. The following fingerprints are used by all submit nodes include nodes providing virtual applications environments:

Type	Bits	Fingerprint
RSA	2048	`SHA256:gxQDZLUauwIusOl906+oobSUgIYG2s8jrIXiupf9aYc`
ECDSA	256	`SHA256:l8NadOR4W07iATwuzlIzsQ208za43YY22RqcnpocGq0`
ED25519	256	`SHA256:IryzZCRhtt7BhymAiyh/Scy/w/6dd2/8zl+uit2S+zc`

Accepting a fingerprint adds it to the ~/.ssh/known_hosts file on your local computer. The client automatically checks the fingerprint on each future login and warns about a divergence. Differing fingerprints indicate a potential man-in-the-middle attack, in which case you should get in contact with the cluster support by following instructions report issues.

User login into a Submit Node will use one of multiple available nodes. By default SSH clients associate a host key to a specific host IP address. This behavior may lead to a conflicting host key - IP address pair when submit nodes get added or removed from a pool. Users may configure their SSH client to not check IP addresses with the option CheckHostIP=no. The examples in the proxy-jump section illustrate this in more details.

You can display the fingerprint of a submit node using the ssh-keyscan and ssh-keygen⁷ commands in combination:

» ssh-keyscan -t rsa virgo.hpc.gsi.de | ssh-keygen -lf -
2048 SHA256:gxQDZLUauwIusOl906+oobSUgIYG2s8jrIXiupf9aYc virgo.hpc.gsi.de (RSA)

Default Shell

Supported login shells are /bin/bash and /bin/zsh.

You can determine your default shell on the login nodes. Please refer to the FAQ How to change my Linux user account configuration? for instructions how to change the default shell. Note that we recommend uses to us the default Bash shell⁸.

Copy Files

Use SSH to copy files from/to a remote computer with the scp command:

scp [[user@]host1:]file1 … [[user@]host2:]file2

Copy a local file to a remote computer:

scp /path/to/data* user@files.hpc.gsi.de:/lustre/home/path/...

The most common option to the scp command is -r to recursively copy an entire directory tree.

Relative paths start from the $HOME home-directory.
The rsync ⁹ file synchronization program is described in its dedicated documentation ¹⁰. It is much more efficient when working with incrementally changing files, since it only copies differences between a source and destination.

Footnotes

Secure Shell, Wikipedia
https://en.wikipedia.org/wiki/Secure_Shell ↩︎
OpenSSH, OpenBSD Foundation
http://www.openssh.com ↩︎
Windows Subsystem for Linux, Microsoft
https://docs.microsoft.com/en-us/windows/wsl ↩︎
Windows Terminal, Microsoft
https://docs.microsoft.com/en-us/windows/terminal ↩︎
OpenSSH Manual Pages, OpenBSD Foundation
http://www.openssh.com/manual.html ↩︎
OpenSSH remote login client - Verifying Host Keys
OpenBSD Foundation
https://man.openbsd.org/ssh#VERIFYING_HOST_KEYS ↩︎
OpenSSH Manual Pages, OpenBSD Foundation
http://www.openssh.com/manual.html ↩︎
GNU Bash Manual
https://www.gnu.org/software/bash/manual/↩︎
Rsync Project, Samba Team
https://rsync.samba.org ↩︎
Rsync Documentation, Samba Team
https://rsync.samba.org/documentation.html ↩︎

--- title: Submit Nodes date-modified: 2024/08/07 toc-expand: 3 abstract: > This section overviews how to access the HPC infrastructure at GSI Darmstadt. It covers details about login to submit nodes using the SSH (Secure SHell) command-line tools. --- ## Overview **The HPC infrastructure is only accessible from within the GSIs internal networks**. Virgo is located in a dedicated high-speed InfiniBand network shielded from external access by a firewall and network gateways. Login to a cluster submit node from outside the GSI networks is facilitated by a **pool of [login nodes][1gXlp] used as SSH [proxies][xtMZ5]**. Submit nodes in the Virgo cluster support different software environments described in more details in the [Virtual Application Environment][hB2WB] (VAE) section. Login to the cluster uses the SSH application. **Password authentication is disabled on submit nodes**, use of SSH [key-authentication][BUkT6] is required. [xtMZ5]: proxy-jump.html ```bash # Example of access to a VAE using the login pool as SSH proxy ssh -J lxlogin.gsi.de vae24.hpc.gsi.de ``` Following is a list of login addresses to the Virgo cluster. Note that these addresses provide access over a load-balancer service, which redirects to a pool of submit nodes: Submit Nodes | VAE | Support EOF -------------|-----|--------------------- `virgo.hpc.gsi.de` | No | - `vae23.hpc.gsi.de` | Yes | 2025 `vae24.hpc.gsi.de` | Yes | 2026 - Login to one of the VAEs using the corresponding host alias `vae*.hpc.gsi.de` - The host alias `virgo.hpc.gsi.de` provides login to the bare-metal environment. These are typically used for building and using [custom containers][tXBGU]. - Individual experiments and user groups distribute software using [CVMFS][6QdZR]. Consult the list of [software repositories][VPi67] for further information. [hB2WB]: ../overview/software.html#vae [1gXlp]: #login-nodes [tXBGU]: ../containers/run-time.html [6QdZR]: ../storage/cvmfs.md [VPi67]: ../storage/cvmfs.md#software-repositories ::: {.callout-warning title="Submit nodes are not used as workstation!"} Submit node are not intended to be used as continues interactive working environment. We strongly discourage the execution of an application directly on submit nodes for a long period of time (e.g. several hours). The expected usage of submit nodes is code compilation, a short test for execution and subsequent job submission. ::: [BUkT6]: ../access/key-authentication.html ## Security Advice A fully functional access to an Linux user login on a large computer cluster over SSH is a very valuable resource for malicious activities. Therefore, it is very important to pay attention to security. Unfortunately, most of the security-settings cannot be technically enforced or only at a great cost to usability. Therefore your engagement is important! Please make sure to comply with following recommendation: 1. Use **strong passwords** for your Linux account (16+ random characters) and your SSH private key (cf. [key-authentication][BUkT6]). 2. **Never use unencrypted private keys!** 3. If possible do not use SSH agent-forwarding, since this exposes your credentials on a remote computer. Instead use the mechanism described in [proxy jump][pnrNy]. 4. Only use X11-forwarding if necessary. [pnrNy]: proxy-jump.html ## Secure SHell Access the infrastructure using a Secure SHell (SSH)[^ssh]. The following list includes all commands and configuration files we recommend to get familiar with: ```sh ssh # login to a remote server scp # copy files from/to a remote server rsync # ^^ (delta sync.) ssh-keygen # generate an ssh public/private key pair ssh-copy-id # copy a public key to a server ssh-agent # daemon temporarily storing the private key ssh-add # add a private key to an SSH agent ssh-keyscan # manage host fingerprints sshuttle # private network tunnel over SSH sshfs # mount remote file-systems over SSH ~/.ssh/config # client configuration ~/.ssh/authorized_keys # authorized keys on a server ~/.ssh/known_hosts # list of server fingerprint for authentication ``` The OpenSSH[^ossh] tool is pre-installed on nearly all Linux distributions, as well as Apple Mac OS X. On Windows users have to access the SSH tools from the Windows Subsystem for Linux[^wsfl] using the Windows Terminal[^wtrm]. As reference documentation we recommend the excellent manual pages from the OpenSSH[^ossm] project. ### Login Nodes Login Pool | Description -|------------------- `lxlogin.gsi.de` | Minimal environment, supposed to be used as SSH proxy jump `lxpool.gsi.de` | User environment includes access to the [network storage][UWHmy] for example the user home-directory (`/u/$USER`) [UWHmy]: ../storage.html The GSI provides a pool of public Linux **login nodes `lxlogin.gsi.de`** that serve as an entry point to systems behind the GSI firewall. Before you continue make sure that you can access these nodes using your personal GSI Linux user name. If you do not have your user name and password yet, follow the instructions under [request access][lXhaw]. Establish a remote terminal session to the login nodes using the `ssh` command with following arguments: [lXhaw]: ../../help/faq/index.html#sec-access Arguments | Description -----------|------------- `[user@]hostname[.domain]` | hostname of the login node for example `lxlogin`<br>domain of GSI `gsi.de`<br/> prefixed by your Linux user name Login with a command similar to: ```sh # `$user` is your GSI Linux user name ssh $user@lxlogin.gsi.de ``` The system will ask you to accept the remote computer into the [list of known hosts](#host-verification). Type in your password before you can login. Use `exit` to logout from the remote computer and end the SSH session. Close hanging SSH connections using an escape sequence `~.` ### Host Verification The first connection to a submit node includes a process to check for the authenticity of the remote server[^sshhk]. The user will need to compare a server fingerprint (derived from the server host key) with the fingerprints listed in the table below. The following fingerprints are used by all submit nodes include nodes providing [virtual applications environments][5a8jq]: [5a8jq]: ../overview/software.html#vae Type | Bits | Fingerprint ----------|---------|---------------------------------------------------- RSA | 2048 | `SHA256:gxQDZLUauwIusOl906+oobSUgIYG2s8jrIXiupf9aYc` ECDSA | 256 | `SHA256:l8NadOR4W07iATwuzlIzsQ208za43YY22RqcnpocGq0` ED25519 | 256 | `SHA256:IryzZCRhtt7BhymAiyh/Scy/w/6dd2/8zl+uit2S+zc` Accepting a fingerprint adds it to the `~/.ssh/known_hosts` file on your local computer. The client automatically checks the fingerprint on each future login and warns about a divergence. Differing fingerprints indicate a potential man-in-the-middle attack, in which case you should get in contact with the cluster support by following instructions [report issues][18Pqa]. [18Pqa]: ../../help/report-issues.html ::: {.callout-note appearance="simple"} User login into a Submit Node will use one of multiple available nodes. By default SSH clients associate a host key to a specific host IP address. This behavior may lead to a conflicting host key - IP address pair when submit nodes get added or removed from a pool. Users may configure their SSH client to not check IP addresses with the option `CheckHostIP=no`. The examples in the [proxy-jump][Mjdzq] section illustrate this in more details. [Mjdzq]: proxy-jump.html ::: You can display the fingerprint of a submit node using the `ssh-keyscan` and `ssh-keygen`[^ossm] commands in combination: ```bash » ssh-keyscan -t rsa virgo.hpc.gsi.de | ssh-keygen -lf - 2048 SHA256:gxQDZLUauwIusOl906+oobSUgIYG2s8jrIXiupf9aYc virgo.hpc.gsi.de (RSA) ``` ### Default Shell **Supported login shells are `/bin/bash` and `/bin/zsh`.** You can determine your default shell on the login nodes. Please refer to the FAQ [How to change my Linux user account configuration?][wScPM] for instructions how to change the default shell. Note that we recommend uses to us the default Bash shell[^tmCtA]. [wScPM]: ../../help/faq/index.html#user [^tmCtA]:GNU Bash Manual <https://www.gnu.org/software/bash/manual/> ## Copy Files Use SSH to **copy files from/to a remote computer** with the `scp` command: ```sh scp [[user@]host1:]file1 … [[user@]host2:]file2 ``` Copy a local file to a remote computer: ```sh scp /path/to/data* user@files.hpc.gsi.de:/lustre/home/path/... ``` The most common option to the `scp` command is **`-r` to recursively copy an entire directory tree**. - Relative paths start from the `$HOME` home-directory. - The `rsync` [^rsyn] file synchronization program is described in its dedicated documentation [^rsdc]. It is much more efficient when working with incrementally changing files, since it only copies differences between a source and destination. [^sshhk]: OpenSSH remote login client - Verifying Host Keys OpenBSD Foundation <https://man.openbsd.org/ssh#VERIFYING_HOST_KEYS> [^ssh]: Secure Shell, Wikipedia <https://en.wikipedia.org/wiki/Secure_Shell> [^ossh]: OpenSSH, OpenBSD Foundation <http://www.openssh.com> [^wsfl]: Windows Subsystem for Linux, Microsoft <https://docs.microsoft.com/en-us/windows/wsl> [^ossm]: OpenSSH Manual Pages, OpenBSD Foundation <http://www.openssh.com/manual.html> [^rsyn]: Rsync Project, Samba Team <https://rsync.samba.org> [^wtrm]: Windows Terminal, Microsoft <https://docs.microsoft.com/en-us/windows/terminal> [^rsdc]: Rsync Documentation, Samba Team <https://rsync.samba.org/documentation.html>