In this part you will find documentation about rclone. Rclone is the rsync for cloud storage. Information on how to install rclone and other things may be found at: https://rclone.org.
Installing rclone from package managers is not recommended as these versions are often quite old. Please follow the instructions on https://rclone.org
Contents
What can Rclone do?
Apart from being an rsync-type tool for cloud storage, it has the following features:
- MD5/SHA1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Check mode to check for file hash equality
- Can sync to and from network, eg two different cloud accounts
- Optional encryption ( Crypt )
- Optional FUSE mount ( rclone mount )
Rclone configuration
In order to use Rclone, an Rclone configuration must be created first. Start the rclone config utility to generatie a Rclone configuration;
rclone config
Then you get the following options:
- Select n for new remote and choose the name, RD in this example
- Select the storage system. Select WebDAV
- Fill in the URL, that is displayed when Getting your WebDAV credentials. In this example we use: https://researchdrive.surfsara.nl/remote.php/webdav
- Select the type of WebDAV storage system. Select Owncloud
- Type in your user name, displayed when creating the WebDAV credentials
- Select: y) Yes type in my own password
- Type in your password and type it in again for confirmation. This is the WebDAV password generated in step 3 on your personal settings page.
- Confirm your settings
Environment URL can be different
Here you enter the web address of the environment that you are using.
- For example; https://researchdrive.surfsara.nl or look up here: Institutional instances
The rclone.conf file resides in .config/rclone/ by default and will contains the following contents:
[RD] type = webdav url = https://researchdrive.surfsara.nl/remote.php/webdav/ vendor = owncloud user = <user name> pass = <encrypted password>
If you want to put the rclone.conf file in a non-standard place, then that is possible too, but then you need to run your rclone commands in the following manner:
rclone --config /path/to/rclone.conf <command> ......
It is important to set the '--timeout
' option high enough. As a rule of thumb, set it to 10 minutes for every GB of the biggest file in a collection. So if the biggest file you want to upload in a collection is 10GB, set --timeout 100m
This may look ridiculously large, but it provides a safe margin to avoid problems with timeout issues.
Use also the flag
`--use-cookies`
to return always on the same Research Drive back end, to prevent file lock between Research Drive back-ends.
Command examples
Listing destination directory
To get an overview of your current files on Research Drive enter the following:
rclone ls RD:
Spaces in path
When your path contains a space, place it then between quotation marks. For example;
./rclone ls "RD:My project with spaces (Projectfolder)"
Copy source directory to destination directory
Copy the source to the destination. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Doesn’t delete files from the destination.
If my/destination/folder doesn’t exist, it is created and the contents of /my/folder goes there:
rclone copy /my/folder RD:my/destination/folder
Working with large objects
When you want to upload large files to Research Drive, we recommend using a timeout of 10 minutes per gigabyte of the largest source file.
As an example, the largest file in the source directory is 5GB. Calculating the argument for --timeout
gives: 10 minutes x 5GB = 50 minutes:
rclone copy --use-cookies --timeout 50m ~/my_5gb_file.bin RD:my/destination/folder
Sync source directory with destination directory
Sync the source to the destination, changing the destination only. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary:
rclone sync /my/folder RD:my/destination/folder
Important: Since this can cause data loss, test first with the --dry-run
flag to see exactly what would be copied and deleted.
Note that files in the destination won’t be deleted if there were any errors at any point.
If my/destination/folder doesn’t exist, it is created and the contents of /my/folder goes there.
Check if files in the source and destination match
Checks the files in the source and destination match. It compares sizes and hashes (MD5 or SHA1) and logs a report of files which don’t match. It doesn’t alter the source or destination.
rclone check /my/folder RD:my/destination/folder
Use Rclone to mount file systems in user space
Using Rclone to mount a file system in user space is done as follows:
rclone mount --use-cookies --timeout 15m RD:[path/to/dir] /path/to/local/mount
The flag
`--use-cookies`
is needed, to get you always on the same Research Drive back-end, to prevent file lock between back-ends. The timeout flag is useful for uploading large files, we recommend using a timeout of 10 minutes per gigabyte of the largest source file.
Enable verbos mode
If you are not receving any response from the mount command, enable the verbos mode using "-v" or "-vv". Rclone will then tell you about each file that is transferred and a small number of significant events.
rclone -v ...
rclone -vv ...
You can unmount this file system by:
fusermount -u /path/to/local/mount
An example is shown below: