The Lustre file system
/cosma6, /cosma7 and /snap7 are LUSTRE file systems. These are parallel file systems which can write different parts of a file to different physical hard drives, thereby speeding up read and write access.
By default, Lustre is configured to write files to a single physical hard drive. However, for large files (typically >100MB), improved performance can be achieved if the file is striped across multiple disks. This must be performed when the file is first created: It is not possible to re-stripe a Lustre file.
Lustre files inherit their striping from the directory in which they are written. Therefore if you have a data directory containing only large files, the simplest method to stripe these files is to simply stripe the directory when it is created (i.e. before any files are written to it).
To stripe a directory:
- lfs setstripe -S 4M -c -1 /path/to/directory
which will stripe across all disks, in 4MB chunks.
To stripe a file before it is created (i.e. to touch it):
- lfs setstripe -S 4M -c -1 /path/to/new/file
This will create an empty file, which can then be written to.
For very large files, striping is essential if the file won’t fit on a single HDD
A C API also exists, so that striping can be performed programmatically. However, don't reinvent the wheel, and use e.g. parallel HDF5 instead (with correct settings).
If you are rsync-ing data from elsewhere onto the Lustre file system, and have files >1GB in size, the rsyncLustre module should be used. This is a patched version of rsync which will automatically stripe all files >1GB in size.