Cookies

We use cookies to ensure that we give you the best experience on our website. You can change your cookie settings at any time. Otherwise, we'll assume you're OK to continue.

Durham University

Advanced Research Computing

Running Jobs

Once logged into Hamilton, the Linux commands you type in at the prompt are run on one of the service's login two nodes. Although these are relatively powerful computers, they are a resource shared between all the users using Hamilton and should not be used for running demanding programs. Light interactive work, downloading and compiling software, and short test runs using a few CPU cores are all acceptable.

Care should be taken not to overload the login nodes: we reserve the right to stop programs that interfere with other people's use of the service.

The majority of the CPU cores and RAM are only available by interacting with Hamilton's queuing system and packaging your work into units called jobs.

The basic idea is that a shell script is written using a text editor such as nano, containing the commands the job will run in sequence. In addition, some specially formatted comment lines are added to the file, describing how much CPU, RAM, time, and other things that the job needs. This is called a job submission script and, once submitted to the queuing system, Hamilton will run the job as resources become free when other jobs finish. A number of example job submission scripts can be found below.

When a job script is submitted using the sbatch command, the system will provide you with a job number, or job id. This number is how the system identifies your job; it can be used to see if the job has completed running yet, to cancel it, etc. If contacting us about a problem with a job, this number is essential to diagnose it.

Using the Serial (one CPU) jobs job script (below) as an example, using the fictional user account foobar22:

[foobar22@hamilton2 ~]$ sbatch my_serial_job.sh
Submitted batch job 3141717

[foobar22@hamilton2 ~]$ squeue -u foobar22
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3141717    seq7.q my_seria   foobar22 PD       0:00      1 (Resources)

The fifth column (ST) shows what state the job is in. R means that the job is running and PD means the job is Pending, i.e. waiting for its turn in the queue. While it is pending, the NODELIST(REASON) column will show why it is not running:

  • (Resources) - normal. The job is waiting for nodes to become free and allow it to run
  • (Priority) - normal. The job is waiting in the queue as there are higher-priority jobs ahead of it
  • (PartitionNodeLimit) - job will not run. The job submission script has asked for too many resources for the queue

When the job has started running, a file called slurm-<jobid>.out will be created. This contains any output printed by the commands in your job script. If the batch scheduled has to kill your job, for example because it tried to use more time or memory than requested, this will be noted at the bottom of this file.

Once the job has finished running, it will no longer appear in the output of squeue. Details about a finished job can be obtained from the command sacct -j <jobid>

Useful commands

The core commands to interact with the batch queue system are:

  • sfree - show what resources are available
  • sbatch <jobscript> - submit a job to the queue
  • squeue -u <username> - see the status of jobs in the queue
  • scancel <jobid> - remove jobs from the queue
  • sacct -j <jobid> - show details of a job that has finished

Available queues and job limits

Compute nodes are organised into queues (also known as partitions). Hamilton currently has:

QueueDescriptionNode TypeNode QuantityJob Limits
test.q Small, short test runs Hamilton7 2 nodes (48 cores) 15 minutes, 8 CPU cores, 2 jobs per user
seq7.q Serial jobs (hamilton7 equipment) Hamilton7 13 nodes (312 cores) 3 days, 1 CPU core
par7.q Parallel jobs (hamilton7 equipment) Hamilton7 95 nodes (2280 cores) 3 days, 30 jobs per user
par6.q Parallel jobs (hamilton6 equipment) Hamilton6 120 nodes (1920 cores) 3 days, 20 jobs per user
openmp7.q Longer running, small shared memory jobs Hamilton7 2 nodes (48 cores) 4 CPU cores, 10 jobs per user

Compute node type details:

  • Hamilton7 - 24 CPU cores, 64G RAM, Omnipath 100GB/s interconnect
  • Hamilton6 - 16 CPU cores, 64G RAM, Truescale 40GB/s interconnect

The Hamilton6 nodes are slower, but people send fewer jobs to them - so jobs will start running more quickly.

Hamilton also has the following resources available that can be accessed by ssh'ing into the node from a login node:

HostnameDescriptionNode Type
cn6g01 GPU node 16 CPU cores, 128G RAM, 2x Tesla K40m
cn6g02 GPU node 16 CPU cores, 128G RAM, 2x Tesla K40m
smp4 High memory node 32 CPU cores, 3T RAM
smp3 High memory node 96 CPU cores, 3T RAM
smp2 High memory node 32 CPU cores, 512G RAM
smp1 High memory node 24 CPU cores, 512G RAM

Useful sbatch options

sbatch options can either be used using the command line (e.g. sbatch -n 1 <job_script>, or by embedding inside your job script (e.g. adding the line #SBATCH -n 1. If both are done, the command line takes precedence. Useful options include:

OptionDescription
-p <QUEUE> Submit job to <QUEUE> (also known as a partition)
--time=<TIME> Run job for a maximum time of <TIME>, e.g. 1:0:0 for 1 hour
--nodes=<NODES> Allocate <NODES> compute nodes to the job
-n <CORES> Allocate <CORES> CPU cores to the job (only for test.q, seq7.q and openmp7.q)
--mem=<MEM> Allocate <MEM> RAM to the job, e.g. 1G (only for test.q, seq7.q and openmp7.q)
--array=<START>-<END> Run job several times, from indexes <START> to <END>
--mail-user=<EMAIL> Send job notifications to email address <EMAIL>
--mail-type=<TYPE> Types of job notifications to send, e.g. BEGIN, END, FAIL, ALL (recommended: END,FAIL)

Serial (one CPU) jobs

Programs that aren't parallel, which includes most programs, are known as serial or sequential programs. They only use one CPU core at a time, and so many can run at the same time on one of Hamilton's multi-core compute nodes.

An example job script to run a program called my_serial_program would be:

#!/bin/bash

# Request resources:
#SBATCH -n 1          # 1 CPU core
#SBATCH --mem=1G      # 1 GB RAM
#SBATCH --time=6:0:0  # 6 hours (hours:minutes:seconds)

# Run on the queue for serial ("sequential") work
# (job will share node with other jobs)
#SBATCH -p seq7.q

# Commands to be run:

./my_serial_program

If saved in a file called my_serial_job.sh, this can be submitted to the queue with the command sbatch my_serial_job.sh

Shared memory (multiple CPUs on one node) jobs

Some programs can use more than one CPU core at a time, but are limited to the CPU cores on a single compute node. These typically use programming techniques such as OpenMP or threading to achieve this. We call them shared memory programs, because the parallelisation requires that all CPU cores have access to the same RAM/memory.

An example job script to run a program called my_sharedmemory_program, which can take advantage of an entire 24 core compute node, would be:

#!/bin/bash

# Request resources:
#SBATCH --nodes=1     # 1 entire node
#SBATCH --time=6:0:0  # 6 hours (hours:minutes:seconds)

# Run on Hamilton7's queue for distributed work
# (but we only ask for one node, so all cores are on the same node)
#SBATCH -p par7.q

# Tell OpenMP to use the job's CPU cores
export OMP_NUM_THREADS=$SLURM_JOB_CPUS_PER_NODE

# Commands to be run:

./my_sharedmemory_program

If saved in a file called my_shared_job.sh, this can be submitted to the queue with the command sbatch my_shared_job.sh

Alternatively, some shared memory codes have can only meaningfully take advantage of a small number of CPU cores. For this case, Hamilton has a openmp7.q queue for running small numbers of potentially long-running programs, but limited to a maximum of 4 CPU cores:

#!/bin/bash

# Request resources:
#SBATCH -n 4          # 4 CPU cores
#SBATCH --mem=1G      # 1 GB RAM
#SBATCH --time=6:0:0  # 6 hours (hours:minutes:seconds)

# Run on Hamilton7's queue for shared memory work
# (job will share node with other jobs)
#SBATCH -p openmp7.q

# Tell OpenMP to use the job's CPU cores
export OMP_NUM_THREADS=$SLURM_JOB_CPUS_PER_NODE

# Commands to be run:

./my_sharedmemory_program

Distributed memory (multiple CPUs across one or more nodes)

Programs can be written to take advantage of CPU cores and memory spread across multiple compute nodes. They typically use the low-level library called MPI (Message Passing Interface) to allow communication between many copies of the same program, each with access to its own CPU core and memory. We call this a distributed memory programming model.

An example job script to run an MPI program called my_mpi_program would be:

#!/bin/bash

# Request resources:
#SBATCH --nodes=4     # 4 compute nodes
#SBATCH --time=6:0:0  # 6 hours (hours:minutes:seconds)

# Run on Hamilton7's queue for distributed work
# (job given exclusive access to enough nodes to provide the CPUs requested)
#SBATCH -p par7.q

# Commands to be run:

module load openmpi/gcc/3.1.3

mpirun -np 96 ./my_mpi_program

If saved in a file called my_dist_job.sh, this can be submitted to the queue with the command sbatch my_dist_job.sh

Hybrid distributed and shared memory (multiple CPUs across one or more nodes)

Writers of distributed memory programs have discovered that using a mixed MPI / OpenMP model have their benefits (for example, to reduce the memory and computation dedicated to halo exchanges between different processes in grid-based codes).

For these codes, we recommend running one MPI rank per CPU socket (two MPI ranks per compute node on Hamilton). An example job script would be:

#!/bin/bash

# Request resources:
#SBATCH --nodes=4              # 4 compute nodes
#SBATCH --ntasks-per-socket=1  # 8 MPI ranks (1 rank per CPU socket, 2 sockets/node)
#SBATCH --time=6:0:0           # 6 hours (hours:minutes:seconds)

# Run on Hamilton7's queue for distributed work
# (job given exclusive access to enough nodes to provide the CPUs requested)
#SBATCH -p par7.q

# 12 cores per socket
export OMP_NUM_THREADS=12

# Commands to be run:

module load openmpi/gcc/3.1.3

mpirun -np 8 ./my_hybrid_program

If saved in a file called my_hybrid_job.sh, this can be submitted to the queue with the command sbatch my_hybrid_job.sh

Job arrays

Sometimes it is necessary to run a large number of very similar jobs. In order to avoid having to write a job script for each of these jobs, the batch queue system provides a technique called Job arrays, to allow a single job script to be used to be run many times. Each run is called a task.

This feature can be combined with any of the above examples, but here is a serial job example that runs the command ./my_program 32 times, with the arguments input_file_1.txt to input_file_32.txt

#!/bin/bash

# Request resources (per task):
#SBATCH -n 1          # 1 CPU core
#SBATCH --mem=1G      # 1 GB RAM
#SBATCH --time=6:0:0  # 6 hours (hours:minutes:seconds)

# Run on the queue for serial ("sequential") work
# (job will share node with other jobs)
#SBATCH -p seq7.q

# Specify the tasks to run:
#SBATCH --array=1-32   # Create 32 tasks, numbers 1 to 32

# Each separate task can be identified based on the SLURM_ARRAY_TASK_ID
# environment variable:

echo "I am task number $SLURM_ARRAY_TASK_ID"

# Run program:

./my_program input_file_${SLURM_ARRAY_TASK_ID}.txt