Self help

Open a command line (an ssh client should be installed by default on both OSs) and use the following command to connect to the phoenix cluster using University of Adelaide credentials: ssh <userid>@phoenix.adelaide.edu.au

Here, <userid> refers to the University of Adelaide identification number. Once a connection is established, a prompt asking for UoA password should appear.

There are various programmes which can be used to connect to Phoenix.

The recommended software to use under Windows XP/7/8 is called PuTTY (download and install PuTTY). Once you have got your copy of PuTTY, you can follow this PuTTY set up guide to get your PuTTY configured. An alternative to PuTTY is the Bitvise SSH client.

The recommended software is still PuTTY (see previous section).

However, a default Windows 10 installation usually offers a PowerShell. An pre-release version of OpenSSH exists. If it is desired to install the package open PowerShell with Administrator rights.

1. Allow running scripts with Set-ExecutionPolicy RemoteSigned
2. Installing Chocolatey (Package manager) iwr https://chocolatey.org/install.ps1 -UseBasicParsing | iex
3. Install OpenSSH via choco install openssh

In a similar way to Linux and Mac one can now connect to Phoenix using the Powershell:  ssh.exe <userid>@phoenix.adelaide.edu.au

If you have cygwin installed, then you can open a cygwin-terminal and then follow the Linux and Mac section, if you never heard of cygwin, safely ignore this note.

For Windows users, WinSCP is an ideal place to start. Download the WinSCP program.  If you need help to set up WinSCP, you may find this Step by step WinSCP guide useful.

For Mac users, Cyberduck is an option, you can download Cyberduck here. Use SFTP to establish the connection.

Fetch is another alternative, however, you will have to register as a educational user to gain licence. Download Fetch.

If you need help to set up Cyberduck or Fetch, you may find. This step by step Cyberduck setup guide or this step by step Fetch setup guide useful.

Using the command line interface may seem like a challenge to start with, but becomes easy with practice. The two common commands for transferring files using the terminal are scp and sftp.

scp uses a similar syntax to the local cp command by specifying the file or directory to copy and the destination:
scp -r myfile.txtOrMydirectory aXXXXXXX@phoenix.adelaide.edu.au:~/fastdir/MyPhoenixFolder

Remote file locations are specified by prepending the file path with the user and hostname as in the example above. For further details you may wish to refer to this scp command Tutorial.

sftp (secure ftp) provides a similar interface to the ftp command. First, navigate to the local directory where your files reside. You can then initiate an sftp session to phoenix with the following command:
sftp aXXXXXXX@phoenix.adelaide.edu.au

Once the sftp session has started, you can use put and get to upload and download files to/from the remote computer. There are other commands available using the sftp protocol, to learn more you can look at this sftp tutorial.

If you need to transfer files from tizard to Phoenix, you need to know your tizard credentials, path to folders on tizard to copy from, and path on Phoniex to copy to.
scp  USER@TIZARD:/path/to/files $FASTDIR/ somepath

A job script is a text file that specifies the resources your code needs to run. The job scheduler then uses these resources to determine when to to run your job. Let's have a look at a simple job script example for some sequential code (that only runs on 1 CPU core):

#!/bin/bash
#SBATCH -p batch # partition (this is the queue your job will be added to)
#SBATCH -N 1 # number of nodes (use a single node)
#SBATCH -n 1 # number of cores (sequential job uses 1 core)
#SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM:SS), here set to 1 hour
#SBATCH --mem=4GB # memory pool for all cores (here set to 4 GB)

# Executing script (Example here is sequential script)
./my_sequential_program # your software with any arguments

We'll begin by explaining the purpose of each line of the script example:

The header line #!/bin/bash simply tells the scheduler which shell language is going to be used to interpret the script. The default shell on Phoenix is bash.

The next set of lines all begin with the prefix #SBATCH. This prefix is used to indicate that we are specifying a resource request for the scheduler.The scheduler divides the cluster workload into partitions, or work queues. Different partitions are used for different types of compute job. Each compute job must select a partition with the -p option. To learn more about the different partitions available on Phoenix, see <reference>.

The Phoenix cluster is a collection of compute nodes, where each node has multiple CPU cores. Each job must specify the CPU resources required by using the -N option to request nodes and the -n to request the number of cores per node required. See <reference>

Each compute job needs to specify an estimate of the amount of time it needs to complete. This is commonly referred to as the walltime, specified with the --time=HH:MM:SS option. The estimated walltime needs to be larger than the actual time needed by the job, otherwise the scheduler will terminate the job for exceeding its requested time.
Dedicated memory (RAM) is allocated for each job when it runs, and the amount of memory required per node must be specified with the --mem option.

A simple job is one in which the computational process is sequential and is carried out by a single node. (Note: If your program file does not use MPI or MPI enabled libraries, your job belongs to this category.) Depending on your computational needs, you may need to use either CPU or GPU-accelerated computing nodes. This post provides some insights about the differences between CPU and GPU-accelerated computing. If you need further support with this, please contact the team to discuss.

Below is a sample job script for simple CPU jobs. You will need to create an .sh file in your directory on Phoenix, and can copy and paste the script below into that file. Please remember you must then configure the job script to your needs. The most common fields that need modification are the number of nodes and cores you wish to use, the duration of time for which you wish to run the job, and the email address to which notifications should be sent (i.e. your email address).

#!/bin/bash
#SBATCH -p cpu # partition (this is the queue your job will be added to)
#SBATCH -N 1 # number of nodes (due to the nature of sequential processing, here uses single node)
#SBATCH -n 2 # number of cores (here uses 2)
#SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour
#SBATCH --mem=4GB # memory pool for all cores (here set to 4 GB)

# Notification configuration
#SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done)
#SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete)
#SBATCH --mail-user=firstname.lastname@adelaide.edu.au # Email to which notification will be sent

# Executing script (Example here is sequential script and you have to select suitable compiler for your case.)
bash ./my_program.sh # bash script used here for demonstration purpose, you should select proper compiler for your needs

For simple GPU jobs, the following example job script can be copied and pasted into a new .sh file in your Phoenix directory:

#!/bin/bash

# Configure the resources required
#SBATCH -p gpu # partition (this is the queue your job will be added to)
#SBATCH -n 8 # number of cores (here uses 8, up to 32 cores are permitted)
#SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour
#SBATCH --gres=gpu:4 # generic resource required (here requires 4 GPUs)
#SBATCH --mem=16GB # memory pool for all cores (here set to 16 GB)

# Configure notifications
#SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done)
#SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete)
#SBATCH --mail-user=my_email@adelaide.edu.au # Email to which notification will be sent

# Execute your script (due to sequential nature, please select proper compiler as your script corresponds to)
bash ./my_program.sh # bash script used here for demonstration purpose, you should select proper compiler for your needs

A parallel (MPI) job is one that harnesses the computational power of multiple nodes, which are networked together and will perform related calculations or processes simultaneously. This can allow highly complex computational processes to be performed in much shorter time-frames. To enable parallel computing, the program you use will need to be MPI enabled or incorporate MPI enabled library. If you do need to run a parallel job on CPUs, following job script is an example:

#!/bin/bash
#SBATCH -p cpu # partition (this is the queue your job will be added to)
#SBATCH -N 2 # number of nodes (here uses 2)
#SBATCH -n 64 # number of cores (here 64 cores requested)
#SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour
#SBATCH --mem=32GB # memory pool for all cores (here set to 32 GB)

mpirun -np 64 ./my_program

For jobs that use GPU accelerators, <my_job.sh> will look like something like the example below:

#!/bin/bash
#SBATCH -p gpu # partition (this is the queue your job will be added to)
#SBATCH -n 2 # number of cores (here 2 cores requested)
#SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour
#SBATCH --gres=gpu:1 # generic resource required (here requires 1 GPU)
#SBATCH --mem=8GB # memory pool for all cores (here set to 8 GB)

mpirun -np 8 ./my_program

More commonly used options, which can be added to #SBATCH lines includes

#SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done)
#SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete)
#SBATCH --mail-user=my_email@adelaide.edu.au # Email to which notification will be sent

SLURM is very powerful and allows detailed tailoring to fit your specific needs. If you want to explore all available SLURM parameters available, simply type following in the command line:

man sbatch

The -A association argument specifies the association from which you wish the compute time to be be debited. Note that you normally only need to specify an association if you have access to multiple allocations, otherwise the scheduler will debit the resources used from your default association.

To submit a job script to the queue use the sbatch command:

$ sbatch my_job.sh

If your job script requires additional variables you can define these with the --export option to sbatch:

$ sbatch --export=ALL,var1=val1,var2=val2 my_job.sh

Be sure to include the ALL option to --export to ensure your job runs correctly.

You can view your job's progress through the queue with the squeue command:

$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
2916 batch my_job. a1234567 PD 0:00 1 (Resources)
2915 batch my_job. a1234567 R 01:03 2 phoenixcs[1-2]
2914 batch my_job. a1234567 R 00:21 1 phoenixg1

The fifth column gives the status of the job as follows: R - running, PD - Pending, F - Failed, ST - Stopped, TO - Timeout

Running squeue without arguments will list all currently running jobs. However if the list displayed is too long for you to easily locate your job, you can limit the search to your own jobs by using -u argument like this

squeue -u aXXXXXXX

where aXXXXXXX is your UofA ID number

To cancel a job you own, use the sbatch command followed by the slum job ID:

$ scancel 2914

To cancel all jobs you own in a particular queue, use the -p argument:

$ scancel -p batch

When you first login to Phoenix, your user shell is located in your personal home directory /home/<userid> ($HOME). The /home file system is backed up, and is solely intended for the files that define your user environment and irrecoverable data such as source code. No user should have more than 5 GB of data in their $HOME directory.

• $HOME directory is located at /home/<userid>
• Don't use $HOME for launching jobs or active job data!
• You can always get back to your home directory by typing cd

The /home file system hardware is not designed to support the intensive file access generated by the many hundreds of jobs that run on the Phoenix compute cores, you should use the /fast file system for job input/output instead.

Your personal /fast directory $FASTDIR is located at /fast/users/<userid>

A symbolic link to that directory can be found in your $HOME directory, i.e. ~/fastdir.

Hence, changing into your personal /fast directory can be achieved through:

• cd ~/fastdir.
• cd /fast/users/<userid>
• cd $FASTDIR.