Rclone Sync Basics and Advanced Features Explained

Rclone sync is a powerful tool that allows you to synchronize files between different cloud storage services or local directories. It's a game-changer for anyone who needs to manage files across multiple platforms.

To start using rclone sync, you need to install it on your system, which is a straightforward process. The installation process is well-documented in the official rclone documentation.

Rclone sync uses a simple command-line interface that makes it easy to configure and run sync jobs. You can specify the source and destination directories, as well as the sync mode, using a variety of flags and options.

The rclone sync command is all about syncing the source to the destination, changing the destination only.

It doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM.

If you don't want to delete files from the destination, use the copy command instead.

Always test first with the --dry-run or the --interactive/-i flag, since this can cause data loss.

Files in the destination won't be deleted if there were any errors at any point.

Rclone syncs the contents of the directory, not the directory itself.

If the source is a directory, it's the contents of that directory that are copied, not the directory name and contents.

If dest:path doesn't exist, it is created and the source:path contents go there.

The modification times of files and directories are synced if the backend supports it.

Use the --metadata flag if metadata syncing is required.

Note that the modification time and metadata for the root directory will not be synced.

You can configure rclone remotes in several ways, including using command line flags, environment variables, or connection strings.

If you supply extra configuration to a backend using a command line flag or environment variable, rclone will add a suffix based on the hash of the config to the name of the remote, so it can be told apart from the unmodified remote when caching the backends.

To set defaults for values in the config file on an individual remote basis, you can use the RCLONE_CONFIG_ + name of remote + _ + name of config file option syntax, making sure to only use letters, digits, or the underscore character in the remote's name.

You can also use connection strings to modify existing remotes or create new ones, which only applies to the specific remote and not to all remotes of that type.

Remote paths can be tricky, but don't worry, I've got you covered. The syntax of the paths passed to the rclone command is straightforward: it follows the standard path structure.

To avoid confusion, it's essential to understand how rclone handles paths with colons (:). On non-Windows OSes, a colon is a valid filename component, but rclone uses it to mark a remote name.

If you need to sync a directory with a colon in its name, you can use the full path starting with a /. This is a lifesaver when dealing with directories like sync:me.

You can also use ./ as a current directory prefix to avoid the colon confusion. For example, to sync a directory called sync:me to a remote called remote:, you can use the full path starting with a /.

Connection strings are a powerful way to modify remotes in rclone, and they can be used to create new remotes on the fly.

You can use connection strings to apply extra configuration to a backend by command line flag, environment variable, or connection string. This allows rclone to tell the modified remote apart from the unmodified remote when caching the backends.

To use a connection string, you need to enclose it in quotes or single quotes. If the parameter has a colon or comma, it must be placed in quotes. For example, if you want to use a connection string with a colon, you would use:

"parameter: value"

This will make the parameter equal to the value. If you leave off the =parameter, rclone will substitute =true, which works very well with flags.

For example, to use s3 configured in the environment, you could use:

Which is equivalent to

Note that on the command line, you might need to surround the connection strings with quotes or single quotes to stop the shell interpreting any special characters within them.

If you are a shell master, you'll know which strings are OK and which aren't, but if you aren't sure, enclose them in quotes and use single quotes as the inside quote. This syntax works on all OSes, except for some characters on Linux/macOS that are still interpreted inside strings.

Remote names are case sensitive, and must adhere to specific rules to ensure they work properly.

Remote names can contain a variety of characters, including numbers, letters, _, -, ., +, @, and space. However, they must not start with a space or a hyphen.

Remote names can end with a space, but it's generally a good idea to avoid it to prevent potential issues. Unicode numbers and letters are allowed in remote names starting with rclone version 1.61, but older versions are limited to plain ASCII.

Be cautious when using non-ASCII characters in remote names, especially on Windows, as the console traditionally uses a non-Unicode character set. This can lead to problems if you use the same rclone configuration from different shells.

Here are the specific rules for valid remote names:

Avoid using single character names on Windows, as it can create ambiguity with Windows drives' names. For example, a remote called "C" is indistinguishable from the C drive.

Server Side Copy is a powerful feature in rclone that allows you to copy files and directories without downloading and re-uploading them. This can save a lot of time and bandwidth, especially when dealing with large files or directories.

Most remotes support server-side copy, but some don't, so it's good to check the remote's capabilities before relying on this feature. I've used rclone for backups and have seen the time and effort it saves when copying large directories.

Server-side copies are used with sync and copy commands, and can be identified in the log when using the -v flag. This feature is especially useful for scripting and making aged backups efficiently.

Server-side copies will only be attempted if the remote names are the same, which means you can't use this feature to copy files from one remote to another with different names. I've found this to be a useful limitation, as it prevents accidental overwrites.

The move command can also use server-side copies if the remote doesn't support server-side move directly. This is done by issuing a server-side copy followed by a delete, which is much quicker than downloading and re-uploading the file.

If you want to use server-side copies with the sync command, you can use the --copy-dest flag to specify a directory to check for identical files. This is useful for incremental backups and can save a lot of time and effort.

The remote in use must support server-side copy and you must use the same remote as the destination of the sync. The compare directory must not overlap the destination directory, which means you can't use the same directory for both comparison and destination.

You can set defaults for values in the config file on an individual remote basis. The names of the config items are documented in the page for each backend.

To find the name of the environment variable, you need to set, take RCLONE_CONFIG_ + name of remote + _ + name of config file option and make it all uppercase. This is a bit of a mouthful, but it's a straightforward process.

The remote's name must be convertible into a valid environment variable name, so it can only contain letters, digits, or the _ (underscore) character. This means you can't use special characters or spaces in the remote's name.

For example, to configure an S3 remote named mys3: without a config file, you must create the RCLONE_CONFIG_MYS3_TYPE variable as above. This will set the type of the remote to S3.

Note that the name of a remote created using environment variables is case insensitive, in contrast to regular remotes stored in config file. This means you can refer to the remote in uppercase or lowercase.

You must write the name in uppercase in the environment variable, but it will be listed and can be accessed in lowercase. This can be a bit confusing, but it's worth getting used to.

The backend configurations set by environment variables can be seen with the -vv flag, e.g. rclone about myRemote: -vv. This is a handy way to check what's been set.

To configure rclone for Google Drive, you need to run the rclone program from the command line and type rclone config.

Pay attention to the configuration program, as it will assume you can open a web browser. If you're connected to a cluster with a GUI-enabled session, you can breeze through the config process. Otherwise, be careful to answer the questions in the config script.

The next two values, client_id and client_secret, can be left blank.

Working remotely with your Google Drive involves using rclone sync to upload and sync files. This command has several arguments that you should consider to avoid getting rejected by the server.

You can limit the number of requests per second to the server using the --cache-rps and --tpslimit arguments. For example, you can set --cache-rps 50 and --tpslimit 10 to ensure you don't exceed the server's limits.

The --dry-run argument can be used to show what will be uploaded without actually executing the command. This is helpful for testing and debugging purposes.

Deleted files can be either permanently deleted or moved to the trash using the --drive-use-trash argument. By default, this argument is set to true, which means deleted files will be moved to the trash.

A typical rclone sync command may look like this: rclone sync $source_dir $dest_dir. The source and destination directory paths should be formatted in a specific way, with the remote path preceded by "gdrive:".