Submit Nodes

Modified

April 11, 2024

Abstract

This section overview how to access the HPC infrastructure at GSI Darmstadt. It covers details about login to submit nodes using the SSH (Secure SHell) command-line tools.

Virgo is only accessible from within the GSIs internal networks. The cluster is located in a dedicated high-speed InfiniBand interconnect network and shielded from the outside by a firewall and network gateways. In order to access a submit node from outside the GSI networks it is required to use the pool of login nodes called lxpool.gsi.de as proxy jump. Submit nodes in the Virgo cluster support different software environments described in more details in the virtual application environment section:

Submit nodes for Virgo-2:

Submit Nodes VAE Comment
virgo.hpc.gsi.de No -
virgo3.hpc.gsi.de No Evaluate the upcoming bare-metal environment
vae22.hpc.gsi.de Yes No migrated to Virgo-3
vae23.hpc.gsi.de Yes Migrated to Virgo-3…
virgo3-vae23.hpc.gsi.de Yes …functionally identical to Virgo-2
vae24.hpc.gsi.de Yes -

Individual experiments and user groups distribute software using CVMFS. Consult the list of software repositories for further information. Login to the cluster uses the SSH application. Password authentication is disabled on the submit nodes. Use of SSH key-authentication is required.

Submit nodes are not workstations!

A submit node is not intended to be used as interactive working environment.

We strongly discourage the execution of an application directly on submit nodes for a long period of time (e.g. several hours). The expected usage of submit nodes is code compilation, a short test for execution and subsequent job submission.

Security Advice

A fully functional access to an Linux user login on a large computer cluster over SSH is a very valuable resource for malicious activities. Therefore, it is very important to pay attention to security. Unfortunately, most of the security-settings cannot be technically enforced or only at a great cost to usability. Therefore your engagement is important!

Please make sure to comply with following recommendation:

  1. Use strong passwords for your Linux account (16+ random characters) and your SSH private key (cf. key-authentication).
  2. Never use unencrypted private keys!
  3. If possible do not use SSH agent-forwarding, since this exposes your credentials on a remote computer. Instead use the mechanism described in proxy jump.
  4. Only use X11-forwarding if necessary.

Secure SHell

Access the infrastructure using a Secure SHell (SSH) 1. The following list includes all commands and configuration files we recommend to get familiar with:

ssh                       # login to a remote server
scp                       # copy files from/to a remote server
rsync                     # ^^ (delta sync.)
ssh-keygen                # generate an ssh public/private key pair
ssh-copy-id               # copy a public key to a server
ssh-agent                 # daemon temporarily storing the private key
ssh-add                   # add a private key to an SSH agent
ssh-keyscan               # manage host fingerprints
sshuttle                  # private network tunnel over SSH
sshfs                     # mount remote file-systems over SSH
~/.ssh/config             # client configuration
~/.ssh/authorized_keys    # authorized keys on a server
~/.ssh/known_hosts        # list of server fingerprint for authentication

The OpenSSH 2 tool is pre-installed on nearly all Linux distributions, as well as Apple Mac OS X. On Windows users have to access the SSH tools from the Windows Subsystem for Linux 3 using the Windows Terminal 4. As reference documentation we recommend the excellent manual pages from the OpenSSH 5 project.

Login Nodes

The GSI provides a pool of public Linux login nodes lxpool.gsi.de that serve as an entry point to systems behind the GSI firewall. Before you continue make sure that you can access these nodes using your personal GSI Linux user name. If you do not have your user name and password yet, follow the instructions under request access. Establish a remote terminal session to the login nodes using the ssh command with following arguments:

Arguments Description
[user@]hostname[.domain] hostname of the login node lxpool
domain of GSI gsi.de
prefixed by your Linux user name

Login to lxpool.gsi.de with a command similar to:

# $user is your GSI Linux user name
ssh $user@lxpool.gsi.de

The system will ask you to accept the remote computer into the list of known hosts. Type in your password before you can login. Use exit to logout from the remote computer and end the SSH session. Close hanging SSH connections using an escape sequence ~.

Host Verification

The first connection to a submit node includes a process to check for the authenticity of the remote server 6. The user will need to compare a server fingerprint (derived from the server host key) with the fingerprints listed in the table below. The following fingerprints are used by all submit nodes include nodes providing virtual applications environments:

Type Bits Fingerprint
RSA 2048 SHA256:gxQDZLUauwIusOl906+oobSUgIYG2s8jrIXiupf9aYc
ECDSA 256 SHA256:l8NadOR4W07iATwuzlIzsQ208za43YY22RqcnpocGq0
ED25519 256 SHA256:IryzZCRhtt7BhymAiyh/Scy/w/6dd2/8zl+uit2S+zc

Accepting a fingerprint adds it to the ~/.ssh/known_hosts file on your local computer. The client automatically checks the fingerprint on each future login and warns about a divergence. Differing fingerprints indicate a potential man-in-the-middle attack, in which case you should get in contact with the cluster support by following instructions report issues.

User login into a Submit Node will use one of multiple available nodes. By default SSH clients associate a host key to a specific host IP address. This behavior may lead to a conflicting host key - IP address pair when submit nodes get added or removed from a pool. Users may configure their SSH client to not check IP addresses with the option CheckHostIP=no. The examples in the proxy-jump section illustrate this in more details.

You can display the fingerprint of a submit node using the ssh-keyscan and ssh-keygen 7 commands in combination:

» ssh-keyscan -t rsa virgo.hpc.gsi.de | ssh-keygen -lf -
2048 SHA256:gxQDZLUauwIusOl906+oobSUgIYG2s8jrIXiupf9aYc virgo.hpc.gsi.de (RSA)

Default Shell

Supported login shells are /bin/bash and /bin/zsh.

You can determine your default shell on the login nodes. Please refer to the FAQ How to change my Linux user account configuration? for instructions how to change the default shell.

Copy Files

Use SSH to copy files from/to a remote computer with the scp command:

scp [[user@]host1:]file1 … [[user@]host2:]file2

Copy a local file to a remote computer:

scp /path/to/data* user@lustre.hpc.gsi.de:/lustre/home/path/...

The most common option to the scp command is -r to recursively copy an entire directory tree.

  • Relative paths start from the $HOME home-directory.
  • The rsync 8 file synchronization program is described in its dedicated documentation 9. It is much more efficient when working with incrementally changing files, since it only copies differences between a source and destination.

Footnotes

  1. Secure Shell, Wikipedia
    https://en.wikipedia.org/wiki/Secure_Shell↩︎

  2. OpenSSH, OpenBSD Foundation
    http://www.openssh.com↩︎

  3. Windows Subsystem for Linux, Microsoft
    https://docs.microsoft.com/en-us/windows/wsl↩︎

  4. Windows Terminal, Microsoft
    https://docs.microsoft.com/en-us/windows/terminal↩︎

  5. OpenSSH Manual Pages, OpenBSD Foundation
    http://www.openssh.com/manual.html↩︎

  6. OpenSSH remote login client - Verifying Host Keys
    OpenBSD Foundation
    https://man.openbsd.org/ssh#VERIFYING_HOST_KEYS↩︎

  7. OpenSSH Manual Pages, OpenBSD Foundation
    http://www.openssh.com/manual.html↩︎

  8. Rsync Project, Samba Team
    https://rsync.samba.org↩︎

  9. Rsync Documentation, Samba Team
    https://rsync.samba.org/documentation.html↩︎