Understanding Rclone Exclude Flags and Rules

Rclone Exclude Flags and Rules can be overwhelming, especially for beginners. The `--exclude` flag is used to specify a pattern to exclude files or directories.

You can use the `--exclude` flag multiple times to specify multiple patterns. The patterns are combined using a logical AND operation.

For example, `rclone sync --exclude "*.txt" --exclude "*.mp3" src dst` will exclude both .txt and .mp3 files.

Rclone also supports using regular expressions with the `--exclude` flag.

Explore further: Aws S3 Lots of Files Rclone

Pattern matching is a powerful tool in rclone, allowing you to exclude files based on specific patterns. You can use the `--exclude` flag to specify a pattern, and rclone will exclude any files that match it.

Simple patterns are case sensitive by default, but you can use the `--ignore-case` flag to make them case insensitive. For example, if you use `--exclude *.bak` without `--ignore-case`, it will only exclude files with a `.bak` extension in all lowercase.

You can also use wildcard characters in your patterns, such as `*` to match any characters. For example, `*.jpg` will match any file with a `.jpg` extension, regardless of its location in the directory tree.

You might enjoy: How to Use Rclone

Here are some examples of patterns you can use with `--exclude`:

You can also use rooted patterns, which start with a `/` and match files at the top level of the directory tree. For example, `/*.jpg` will only match files with a `.jpg` extension at the top level of the directory tree.

Rooted patterns can be especially useful when working with remote storage, as they allow you to match files at the top level of the remote directory tree, even if the remote directory tree is not the same as the local directory tree.

For example, if you use `rclone ls remote: --exclude "/dir/**"`, it will return the total size of all files on the remote, excluding those in the root directory `dir` and its subdirectories.

See what others are reading: Rclone Remote Machine

You can use the --exclude-from flag to add exclude rules from a file. This is useful if you have a lot of rules to exclude files.

The file should be prepared with exclude rules, like in the example where exclude-file.txt excludes files ending in bak and file2.jpg.

You can also use the --filter flag to add a single include or exclude rule. Include rules start with + and exclude rules start with -. A special rule called ! can be used to clear the existing rules.

The --delete-excluded flag is used in conjunction with rclone sync to delete files on the destination that are excluded from the sync. This can be useful if you want to remove files that are no longer needed.

Here's a summary of the exclude options:

Read from a file to exclude or include files in your Rclone sync. You can use the --exclude-from flag to read exclude patterns from a file.

This flag is useful for applying multiple exclude filter rules to an Rclone command. For example, you can use it to exclude files with a suffix .bak and file2.jpg.

The file containing the exclude rules should have a list of remarks and pattern rules. Rclone will ignore lines starting with # or ;, which can be used for comments.

The rules are processed in the order they are defined, so make sure to arrange the most restrictive rules first. You can use the -vv --dump filters flag to see how the rules appear in the final regexp.

You can also use the --filter-from flag to read filtering patterns from a file. This flag adds include/exclude rules from a file, which can be useful for complex sync operations.

The file containing the filter rules should have a list of remarks and pattern rules, with include rules starting with + and exclude rules starting with -. The ! symbol clears existing rules.

To use the --filter-from flag, prepare a file like filter-file.txt, and then use it as --filter-from filter-file.txt. The rules will be applied in the order they are defined.

The --delete-excluded flag is a powerful tool for cleaning up your destination folder. It deletes any files on the destination that are excluded from the sync command.

You can use it in conjunction with rclone sync to restrict the scope of the command. For example, if you run rclone sync with the --min-size 50k flag, it will exclude files less than 50 KiB from the sync.

If you repeat the sync command without the --min-size 50k flag but with the --delete-excluded flag, it will delete all files on the destination that are less than 50 KiB, as they are now excluded from the sync.

This can be useful if you want to remove files from the destination that are no longer relevant or have been superseded by newer versions.

For your interest: Aws S3 Sync Exclude

Rclone's rules application is a powerful feature that allows you to fine-tune your sync operations. Rclone maintains a list of include rules and exclude rules, which are processed in order until a match is found.

Each file is matched against the list, and if a match is found, the file is included or excluded accordingly. If the matcher falls off the bottom of the list without finding a match, the path is included.

Rclone implies an --exclude * rule when using --include or --include-from flags, which adds a rule to the bottom of the internal rule list. However, using --filter...+ does not imply this rule. To clear existing rules, use the special rule !.

The order of rules matters, as Rclone processes them in the order they are defined. Arrange the most restrictive rules first and work down. Lines starting with # or ; are ignored, and can be used to write comments.

Here's a summary of how Rclone's rules application works:

By understanding how Rclone's rules application works, you can create complex filter rules to suit your needs. Experiment with different combinations of include and exclude rules to find the perfect balance for your sync operations.

Quoting shell metacharacters is crucial when using filter patterns in rclone commands, especially in Linux and OSX. This is because these metacharacters can be interpreted by the shell, leading to unexpected behavior.

To avoid this issue, you can use one of the following quoting methods: --include \*.jpg, --include '*.jpg', or --include='*.jpg'. These methods will ensure that the metacharacters are treated as literal characters, rather than shell commands.

In contrast, Microsoft Windows expansion is done by the command, not the shell, so quoting is not required for filter patterns. For example, --include *.jpg will work as expected.

Quoting shell metacharacters can be a bit tricky, but don't worry, I've got you covered. In Linux and OSX, shell metacharacters like * may not work as expected in your shell, and may require quoting.

If you're using a filter pattern with shell metacharacters, you may need to use one of the following formats: --include \*.jpg, --include '*.jpg', or --include='*.jpg'. I've found that using the backslash method (e.g. --include \*.jpg) can be a safe bet, but the single quote method (e.g. --include '*.jpg') can also work.

In Windows, expansion is done by the command, not the shell, so --include *.jpg should work fine without any quoting. This is because the command itself handles the expansion, rather than the shell.

You can read exclusion patterns from a file using the --exclude-from flag in rclone commands. This can be a big help when you have a long list of files to exclude.

The file should contain a list of rules, including comments. For example, an exclud-file.txt might look like this: rclone ls remote: --exclud-from exclud-file.txt. This lists all the files on remote: except for those named file2.jpg or with the .bak suffix, which is equivalent to rclone ls remote: --exclude file2.jpg --exclude "*.bak".

You can repeat the --exclude-from flag if you need to. It's also useful when you have multiple exclusion rules to apply to a single rclone command. Just be aware that --exclude-from shouldn't be used with --include, --include-from, --filter, or --filter-from flags.

如果你尝试匹配文件名 Zaphod.txt 而不使用 --ignore-case，rclone会忽略它，因为它默认是区分大小写的。

使用 --ignore-case 后，rclone 就会匹配 Zaphod.txt 文件了。

Rclone's filter flags decide which files are affected by commands like sync, move, and ls. These flags are specified as path/file name patterns, path/file lists, file ages and sizes, or file existence in a directory.

Take a look at this: Rclone Upload File

You can test filters without risking data damage by applying them to rclone ls, or using the --dry-run and -vv flags. Rclone's filter patterns can only be used in filter command line options, not in remote specifiers.

For example, rclone copy "remote:dir*.jpg" /path/to/dir has no filtering effect, but rclone copy remote:dir /path/to/dir --include "*.jpg" does. Be careful not to mix any two of the "--include...", "--exclude...", or "--filter..." flags in a single rclone command, as the result may not be what you expect.

The solution is to use the "--filter..." flag instead. This allows you to specify a single include or exclude rule, and gives you more flexibility when combining multiple rules.

Here is an example of how you can use the "--filter..." flag to exclude all .bak files from a list of remote files:

rclone ls remote: --filter "- *.bak"

This command will exclude all .bak files from the list of remote files.

If this caught your attention, see: Rclone Copy

You can exclude files matching a pattern using the --exclude flag, which can be repeated. This flag is processed in a specific order, so be aware of that when combining it with other flags.

The --exclude flag should not be used with --include, --include-from, --filter, or --filter-from flags. It also has no effect when combined with --files-from or --files-from-raw flags.

To exclude all .bak files from a listing, use the command rclone ls remote: --exclude *.bak. To exclude all files in a directory and its subdirectories, use rclone size remote: --exclude "/dir/**".

On Microsoft Windows, you can use the command rclone ls remote: --exclude "*\[{JP,KR,HK}\]*" to list files without [JP], [KR], or [HK] in their name. Quotes prevent the shell from interpreting the \ characters, and the { and } define an rclone pattern list.

Directory filter rules are defined with a closing / separator. For example, /directory/subdirectory/ is an rclone directory filter rule. Directory recursion optimisation may affect performance, but normally not the result.

Rclone maintains a list of include rules and exclude rules, and each file is matched in order against the list until it finds a match. The file is then included or excluded according to the rule type. If the matcher falls off the bottom of the list, the path is included.

Here are some examples of how filter rules are applied to files:

The order in which rules are applied is important, as it can affect the outcome of the filter. The --dump filters flag can be used to view the directory filter rules rclone has implied for a command.

Rclone's filtering system is based on a combination of rules and patterns. These rules and patterns can be specified using various flags, such as --include, --exclude, and --filter.

The patterns used in Rclone are similar to those used in glob syntax, with character-range, character classes, and regular expressions being supported. If you're familiar with Go regular expressions, you can use them in Rclone as well.

Filter patterns can start with a /, which means they'll only match at the top level of the directory tree, relative to the root of the remote. If a pattern starts with a /, it's treated as an absolute path. If it doesn't start with a /, it's matched starting at the end of the path/file name, but it must match from a / separator or the beginning of the path/file.

Simple patterns are case sensitive unless the --ignore-case flag is used. If you're running on Microsoft Windows, be sure to use / in path/file name patterns, not \.

Here's a summary of the flags that can be used to specify filter rules:

Note that each flag can be used multiple times, and the order in which they're processed is important. If you're using a combination of include and exclude rules, be sure to use the --filter... flag to specify the order in which they're applied.

You can use regular expressions in filter patterns to have more control over matching. This is especially useful for advanced users.

Regular expressions in rclone are defined in the Go regular expression reference and should be enclosed in {{}}. They will match only the last path segment if the glob doesn't start with / or the whole path name if it does.

Rclone will not attempt to parse the supplied regular expression, meaning that using any regular expression filter will prevent rclone from using directory filter rules, as it will instead check every path against the supplied regular expression(s).

Regexp syntax can be mixed with glob syntax, for example, you can use {{regexp}} and glob patterns together.

You can also use regexp flags, such as --regexp-case-insensitive to set case insensitive matching.

Be careful with wildcards in regular expressions - you don't want them to match path separators normally.

Here's an example of how to match any file name starting with "start" and ending with "end" using regular expressions:

{{^start.*end$}}

This will match a directory called "start" with a file called "end.jpg" in it as the .* will match / characters.

You can use -vv --dump filters to show the filter patterns in regexp format.

File and directory management with rclone is a powerful tool for fine-tuning your backups and data transfers. You can use directory filter rules to determine whether rclone recurses into subdirectories, potentially optimizing access to a remote by avoiding listing unnecessary directories.

To define directory filter rules, use a closing / separator. For example, /directory/subdirectory/ is a valid rclone directory filter rule. Rclone commands can use these rules to decide whether to recurse into subdirectories, which can be a big performance boost.

If you're using regular expression filters, however, no directory recursion optimization is possible, as rclone must check every path against the supplied regular expressions. This can make things slower.

Rclone directory recursion optimization occurs under three conditions: if the source remote doesn't support the ListR primitive, if the rclone command isn't naturally recursive, or if you've applied the --disable ListR flag.

Here are some examples of rclone directory filter rules:

You can also imply directory filter rules from path/file filter rules. To view the implied directory filter rules for a command, use the --dump filters flag.

For example, if you run `rclone ls remote: --include /directory/`, rclone will imply the directory include rule and ignore everything outside of that directory. However, this will not match any files, because the implied --exclude ** rule will prevent listing of any path/files not previously matched by the rules.

Rclone will match files below directories dir1 or dir2 only, but will not be able to use this filter to exclude a directory dir3 from being traversed. If you want to exclude a directory based on the presence of a file, use the --exclude-if-present flag.

The --exclude-if-present flag has priority over other filter flags and can be repeated to check for multiple file names. For example, `rclone ls --exclude-if-present .ignore dir1` will not list dir3, file3, or .ignore.

Here's a summary of the --exclude-if-present flag:

This flag is powerful but also potentially dangerous, so be sure to use it with caution and test it thoroughly with the --dry-run and -v flags.

Rclone has some key differences from rsync when it comes to patterns. Rclone implements bash style {a,b,c} glob matching which rsync doesn’t.

Rsync and rclone handle patterns differently, and it's essential to understand these differences when excluding files. Rclone ignores / at the end of a pattern.

Rclone always does a wildcard match so \ must always escape a \. This means you need to be mindful of your pattern syntax to avoid unintended matches.