It is possible to use gcc 9.3, 10.2 or 11.1 with openmpi/4.1.1 or mvapich_mpi/2.3.7 on cosma8.

The current recommendation is the Intel Compiler and OpenMPI:

module load intel_comp/2021.3.0 compiler

module load openmpi/4.1.1

module load ucx/1.10.1

module load fftw/3.3.9epyc parallel_hdf5/1.12.0 parmetis/4.0.3-64bit gsl/2.5

On COSMA8 with 4 ranks per node, the following sould be used when launching an mpirun for SWIFT:

-np 4 --map-by socket:PE=32

For a similiar effect with mvapich the following is required:

export MV2_CPU_BINDING_POLICY=hybrid

export MV2_HYBRID_BINDING_POLICY=linear

export OMP_NUM_THREADS=32

but note that the current module does not yet support mpirun.

MV2_ENABLE_AFFINITY=0 is required.

With gcc 11.1 the dependencies for swift are also available:

module purge

module load gnu_comp/11.1.0

module load openmpi/4.1.1 # OR module load mvapich2_mpi/2.3.6

module load parallel_hdf5/1.12.0 parmetis/4.0.3

module load gsl/2.4 fftw/3.3.9epyc

With openmpi the default binding of processes to cores seems to be wrong for SWIFT. The '--bind-to none' flag is required to use all of the cpus.

With mvapich, 'export MV2_ENABLE_AFFINITY=0' is required to use all of the cpus.