GRUS SFTP download instructions

This is a condensed verison of the GRUS user guide provided by UPPMAX. If you find the instructions below too brief, please refer to that guide.

Typical deliveries from NGI consist of large amounts of data and we generally don't recommend using a graphical software for downloading. We recommend using a command line client for SFTP transfers. Many different clients are freely available, such as tools from OpenSSH for e.g. Linux and MacOSX or Putty for Microsoft systems. In the examples below, we use the sftp client, which comes bundled with most Linux distributions.

Working on Bianca or Rackham

Note that if you are going to transfer data from GRUS to a compute project on the Bianca or Rackham HPC resources, you should use the data transfer service provided by UPPMAX, rather than transferring files over SFTP.

Prerequisites

In order to be able to retrieve data from GRUS via SFTP, you must:

Download data using a SFTP client

For accessing GRUS, the username is constructed from your regular UPPMAX username and the name of the delivery project to access, separated by a hyphen ('-'). That is, if your username is myuser and the delivery project is delivery00001, the username for GRUS will be myuser-delivery00001. Similarly, the password is your UPPMAX password followed by the current six-digit code generated by your two-factor application. That is, if your regular password is mypassword and the current two-factor code is 123456, the GRUS password is mypassword123456.

  • log in to the system you will download data to, e.g. your local HPC cluster

  • connect to grus using your SFTP client

    sftp <username>-<projid>@grus.uppmax.uu.se

  • download the data recursively

    get -r . /path/to/local/destination/folder/

  • verify the integrity of the downloaded files

    • locate files with checksums

      find /path/to/local/destination/folder/ -name "*.md5"

    • for each checksum file, go to the folder where it is located and verify the checksums within

      cd folder/containing/file_with_checksums && md5sum -c file_with_checksums.md5

Tips

  • Do not skip the verification of the downloads using the supplied checksums. The SFTP client may not be as reliable as e.g. transferring files using rsync and it does happen that files are corrupted or lost in transfer. If the checksum for a file doesn't match (or if a file is missing), simply re-download that file.
  • ​The download is likely to consist of large amounts of data and thus take a long time to complete. To avoid interrupted transfers, it's usually a good idea to launch the SFTP client in a screen or tmux session.
  • More versatile and robust SFTP clients, e.g. lftp, may be more suitable for downloading. However, because the code for two-factor authentication is used as part of the password, you cannot use features of the client that involves making additional connections to the server. Be sure to disable such behaviour.
  • UPPMAX provides a brief tutorial with basic SFTP commands, available here.

Cookies are not used for anonymous users.