Cookies

We use cookies to ensure that we give you the best experience on our website. You can change your cookie settings at any time. Otherwise, we'll assume you're OK to continue.

Durham University

Advanced Research Computing

Storage

Internal Storage

Users have two locations on Hamilton where they can store their files:

  • Home directory (/ddn/home/<username>)

    By default, new users are provided with 5GB of storage space in their home directory and old versions of files are kept for 14 days, appearing in directory /ddn/home/.snapshot/daily-<date>.
    In addition, these areas are backed up - versions of files up to 28 days old, and sometimes longer, can be recovered on request.

  • Data directory (/ddn/data/<username>)

    New users are also provided with 85GB of storage space in their data directory. Unlike the home directories, data directories are not backed up and old versions of files are not kept.

Your current usage of these storage areas can be seen by running the quota command on hamilton. Please contact us if your research requires more disk space.

NOTE: the data directories are not backed up and are vulnerable to data loss. Please ensure you save a copy of anything you cannot easily recreate to another service, for example the CIS PRS and SRS services.

NOTE: Hamilton's compute nodes do not contain disks, so its operating system is held in RAM. Any files stored in /tmp within a job means that there is less memory available to run a job.

External Storage

In addition to the scp command described on the login page, files can be transferred between Hamilton and other Linux/Mac machines using the rsync command:

  • rsync -av <directory> <username>@hamilton.dur.ac.uk:

    Enter the command (replacing <username> with your University user name, and <directory> with the directory you wish to transfer to hamilton). Unlike scp, which transfers the file(s) specified, this command only transfers files that are different - so can speed up the process of synchronising a directory if there have been a small number of changes.

    Detailed information on how to use this command can be found by entering the command man rsync

Other graphical interfaces include:

  • Apple Mac - Cyberduck

    TheCyberduck application can be used to provide a drag and drop interface between a Mac desktop and Hamilton.

  • Linux - the GNOME Files application

    The GNOME desktop environment provides a file browser which is able to access remote storage and provide a drag and drop interface between a Linux desktop and Hamilton.

    To use, start the Files application and click on + Other Locations on the bottom left. A small Connect to Server section should appear on the bottom right. In the Enter server address... section, type sftp://<username>@hamilton.dur.ac.uk (replacing <username> with your University user name). Enter your password when prompted.

    Once connected, you are able to bookmark the connection so you can more easily use it later. Right click on the newly created <username> on @hamilton.dur.ac.uk section on the left and select Add Bookmark

Figure 1: Other Locations

Figure 2: Bookmark

The University provides storage for researchers - please see the CIS pages for more details:

  • Personal Research Storage (PRS)

    Storage for research-active staff

  • Shared Research Storage (SRS)

    Storage for research projects

Access has not been integrated into Hamilton yet, but is accessible via mira.dur.ac.uk:

  1. Login to the Storage Manager website, taking note of the "Linux Desktop Service" path to the storage you wish to access (referred to below as <prs_path>)
  2. Login to Hamilton and take a note of the path (referred to below as <hamilton_path>) to the Hamilton data you wish to transfer to PRS
  3. Login to Mira
  4. On Mira, transfer files from Hamilton to PRS with: scp hamilton:<hamilton_path> <prs_path>

Each member of staff has access to some Microsoft OneDrive storage, normally available through a web interface, or the OneDrive folder on a CIS management PC. Files can be transfered to/from this using the rclone utility on Hamilton. To use:

  • Execute module load rclone to make the rclone command available. man rclone for detailed documentation.
  • Execute rclone config and:
      as you are about to store credentials in rclone, secure it's configuration by selecting:
      1. s) Set configuration password
      2. a) Add Password
      3. Enter a password to secure your credentials with
      4. q) Quit to main menu
      authorise rclone to use your Microsoft account with:
      1. n) New remote
      2. name> (something memorable, like "durham")
      3. Storage> onedrive
      4. client_id> (leave blank)
      5. client_secret> (leave blank)
      6. Edit advanced config? y/n> n
      7. Use auto config? y/n> y
      8. A web browser will start, showing the Microsoft login page: login using your University credentials
      9. Your choice> onedrive
      10. Chose drive to use:> 0
      11. Is that okay? y/n> y
      12. y) Yes this is OK
      13. q) Quit config

You can now use rclone to access your University Onedrive:

    • List files: rclone ls durham:
    • Copy files to onedrive: rclone <file> durham:
    • Copy files from onedrive to a local directory: rclone durham:<file> <local_dir>

    rclone is a versatile tool, that can be used to transfer files between many different services.