HPC Basics

Here, you will learn essential skills for working with performance-optimized computing resources. We will guide you through making your first SSH connection to a remote cluster, navigating file systems, and optimizing your workflow for maximum efficiency. Do not worry if it seems overwhelming at first; we will break down each concept into manageable steps and provide hands-on examples to help you build routines.

Basic Shell Commands¶

In this section, we'll explore essential command-line interface (CLI) commands that serve as the building blocks for navigating and manipulating your system through the terminal.

Command	Description	Example
`cd`	Change into another directory	`cd -`
`~`	Path of the user's home directory	`cd ~`
`ls`	List all files and directories in current path	`ls`
`pwd`	Displays the current path	`pwd`
`mkdir`	Create a new directory	`mkdir new_directory`
`rm`	Remove files or directories	`rm file`
`rm -rf`	Force remove files or directories (handle with care)	`rm -rf file`
`cp`	Copy files	`cp file1 file2`
`mv`	Move files (can be used to rename files)	`mv file1 directory/`
`du`	Show directory contents and storage size	`du -h`
`ln -s`	Create a symbolic link	`ln -s /path/to/source /path/to/link`

Take some time to familiarize yourself with these basic CLI commands and try them in practice. They will help you efficiently navigate and work within the cluster environment.

Dartmouth Discovery Cluster

To familiarize yourself with these basic commands, navigate to your group's lab space on Dartmouth's discovery cluster. Create a symbolic link to the lab space in your home directory.

Secure Shell (SSH) Protocol¶

SSH (Secure Shell) is a network protocol that provides a secure way to access and manage remote computers by establishing an encrypted connection between client and server. To set it up on your local machine, you can go through the following steps (see also other online resources).

Setting up an SSH key locally. In a terminal window on your local machine, run the following command to set up an shh key:
```
ssh-keygen -t ed25519 -C "your_email@example.com"
```
This generates a new SSH key using the provided email address as a label. When prompted to "Enter a file in which to save the key," press Enter to accept the default file location. You can choose a passphrase to secure your SSH keys.
Using SSH keys to access servers without password. In a terminal window on your local machine, run the following command to copy your SSH key to a remote host:
```
ssh-copy-id -i ~/.ssh/mykey user@host
```
You will be prompted to enter the remote host's access data. After the succesfull ssh-key communication, future logins should not require additional password requests. However, systems may have special security requirements.

Dartmouth Discovery Cluster

Dartmouth infrastructure has some caveats when using ssh-keys for remote connections.
Create aliases for remote hosts. It can be a substantial shortcut to abbreviate remote host names and addresses. Open a terminal window in your home directory and access the following file:
```
.ssh/config
```
In this file, you can set up aliases for remote hosts. For basic functionality, use:
```
Host alias_name
    HostName host_address
    User user_name
```
You can now access the remote host by using the alias
```
ssh alias_name
```
Dartmouth Discovery Cluster

For Dartmouth infrastructure, the alias could read:
```
Host discovery
    HostName discovery.dartmouth.edu
    User netID
```

Transfering files with rsync¶

Rsync (remote synchronization) is a powerful command-line utility for efficiently transferring and synchronizing files between different locations, whether they are on your local machine or across remote systems. This tool is particularly valuable because it uses a delta-transfer algorithm that only copies the differences between source and destination files, significantly reducing the amount of data that needs to be transferred. We will explore how to use rsync's basic features and some of its most useful advanced options. The basic syntax is as follows:

rsync -arv /path/to/origin/directory path/to/destination/directory

The option -arv will use archive mode to preserve permissions (a), recursively (r) loop through sub-directories, and provide verbose (v) output. Each path can be a local or remote paths, making rsync an important tool to transfer data between clusters and our local resources. The following option provides the transfer progress for each file:

rsync -arv --progress /path/to/origin/directory path/to/destination/directory