Python

Hamilton supports using the Python programming language. By default, a fairly old version of Python is available via the python command, which is provided by the underlying operating system.

In addition to this, a number of other versions of Python are made available through the module command. module avail python will show what is available.

The current version of Python we recommend on Hamilton is python/3.9.9. This module also contains a number of additional Python packages suitable for numerical work, such as numpy and scipy.

This page contains information about

• Installing python packages
• Running python jobs
• Making python faster

Installing Python packages

To supplement the python module, each user is able to install additional Python packages accessible only to them. As people will need different packages and versions of those packages, this is a useful way of giving users the flexibility to install software without waiting for the Hamilton administrators to install it for you.

To see what packages are installed:

module load python/3.9.9
pip list

To install additional packages:

module load python/3.9.9
module load $PYTHON_BUILD_MODULES
pip install --user <package_name>

This will download, build and install the package in your home directory under the directory .local . Your Python programs will be able to automatically find them using the normal import statement.

Note the use of $PYTHON_BUILD_MODULES above. The python module sets this environment variable, which contains a list of the modules used to build the python module itself. Reproducing this environment can help when installing some packages. To see which modules it lists, type:

echo $PYTHON_BUILD_MODULES

Running Python jobs

Example python program, in file my_python_program.py:

#!/usr/bin/env python

print("hello, world!")

This file needs to be marked as executable before the operating system will allow you to run it:

chmod u+x my_python_program.py

Example job script my_python_job.sh to run a python program using a single CPU core:

#!/bin/bash
 
# Request resources:
#SBATCH -c 1          # 1 CPU core
#SBATCH --mem=1G      # 1 GB RAM
#SBATCH --time=1:0:0  # 1 hour (days-hours:minutes:seconds)
 
# Run on the 'shared' queue
# (job will share node with other jobs)
#SBATCH -p shared

# Make python available:
module load python/3.9.9
 
# Commands to be run:
 
./my_python_program.py

Submit it to the queue with the command: sbatch my_python_job.sh

Making Python faster

If you need to make your Python code run faster, there are a number of things you can try. Roughly, in order of importance (most important first):

1.     Use a profiler such as the cProfile and profile packages, or benchmark using timeit to understand: where your Python program is spending most of its time; where you need to concentrate your effort; and the impact of any changes made to speed up the code.   
2. Avoid writing your own matrix or vector manipulation functions, use the numpy package.
3. Avoid native Python datatypes such as lists and dictionaries for numerical work, instead use equivalent functionality provided by the numpy and scipy packages.
4. Try using the Python package numba to automatically optimise specific functions using a compiler.
5. Try writing the most numerically intensive part of your program in another language, such as Fortran or C, and call it from Python using the Cython package. Consider parallelising the Fortran or C code using OpenMP.
6. Try using packages such as multiprocessing or mpi4py to make use of more than one CPU core.