Rclone Backblaze Automated Backup Solution

Rclone can connect to Backblaze B2, a cloud storage service, to create a robust automated backup solution.

Rclone's open-source nature makes it a cost-effective option, and its compatibility with various cloud services, including Backblaze B2, is a significant advantage.

With Rclone, you can schedule backups to run automatically, ensuring your data is safely stored in the cloud.

Using Rclone with Backblaze B2 offers a scalable and secure solution for your backup needs.

To set up Backblaze, you'll need to create a B2 bucket. Log in at backblaze.com and navigate to the 'Buckets' page to create your bucket.

First, you'll need to create a new set of credentials for rclone to use. Go to the Application Keys page and click on 'Add a New Application Key'. Give it a name and click 'Create New Key'.

You'll then see both a keyID and an applicationKey on the next page. Note them both down, as you'll need these later.

To get started, use Backblaze's interactive configuration wizard to set up the Backblaze config.

To back up your data, you can use a combination of tools like Nextcloud and rclone. Nextcloud supports WebDAV, and rclone supports both WebDAV and B2.

Using a separate VPS to host rclone can be a good idea, as it allows for more flexibility and control over your backup process. This is what I've done for my Mastodon instance's Elasticsearch.

Unfortunately, some data like user accounts and tags may be lost if you have to move providers, as it's not possible to back up your Nextcloud database in some cases.

When using Nextcloud, it's essential to have a backup strategy in place to protect your data. Thankfully, Nextcloud supports WebDAV.

You'll need a separate service like B2 on Backblaze to store your backups. Rclone is a great tool that supports both WebDAV and B2, making it a good choice for this task.

To use rclone, you'll need a separate VPS to host it, as well as your Mastodon instance's Elasticsearch. Unfortunately, it's not possible to back up your Nextcloud database while using Hetzner.

Creating a backup script is a crucial step in your backup strategy. It's essential to understand the options available to you.

The `rclone sync` command makes the destination identical to the source. You can specify the source remote as `nextcloud:` and the destination as `b2:your-bucket/your-directory`.

The `b2:your-bucket/your-directory` syntax is used to specify the backup destination. Replace `your-bucket` with the name of your B2 bucket and `your-directory` with the name of the directory where you want to store your backup.

To ensure you don't lose data, you can use the `--backup-dir` option. This will move updated or deleted files into a directory with a timestamp, allowing you to restore historical versions.

Here's a list of additional options you can use with `rclone sync`:

Remember to remove the `--dry-run` flag and run the script again until finished. Be prepared for a long process, as it may take several days to complete.

To configure rclone, you'll need to create two remotes: one for Backblaze and one for Nextcloud. You can do this by running the command `rclone config` and following the prompts.

You'll start by creating a "backblaze" container pointing to your B2 bucket. Select 'n' for a new remote and name it "backblaze". Use the bucket applicationKeyId as the account and the Application Key itself as the key.

To encrypt your files, you'll need to create an encrypted remote named "encrypted_b2" inside your backblaze remote. You can do this by running `rclone config` again and selecting the Encrypt/Decrypt a remote option from the list of remotes.

Here are the basic steps for configuring rclone:

Once you've configured your remotes, you can test them by running `rclone ls b2:` and `rclone ls nextcloud:`. This will give you an empty directory listing for Backblaze and a listing of all your files within Nextcloud.

To configure Rclone, you'll need to create a remote for your cloud storage service. This is done by opening a command prompt and typing `rclone config`. You'll then be prompted to create a new remote, naming it something like "backblaze".

Select the option to create a new remote and name it accordingly. For Backblaze, you'll need to use the bucket applicationKeyId as the account and the Application Key itself as the key. Note that you shouldn't use your actual Account ID listed at the top of the Buckets page.

You'll also need to decide if you want to have files permanently deleted when you delete them from your local machine. If so, select the option to delete files permanently.

To verify that your remote is set up correctly, run `rclone lsd backblaze` and check that the output is the name of your B2 bucket.

If you want to create an encrypted remote, you can do so by running `rclone config` again and selecting the Encrypt/Decrypt a remote option. This will ask for the name of the remote to encrypt, which should be in the format "backblaze:bucket_name".

Here's a summary of the remote configuration options:

Remember to review and finalize your settings, and then test your remote by running `rclone ls` to verify that it's working correctly.

Rclone's versioning system is a powerful feature that allows you to manage multiple versions of your files.

By default, Rclone keeps old versions of files, which means that when you upload a new version of a file, it creates a new version of it. This is useful for tracking changes to your files over time.

You can adjust the versioning settings on a per-bucket basis using the "Lifecycle settings" on the B2 control panel or by using the `--b2-lifecycle` flag.

Old versions of files are visible using the `--b2-versions` flag, which shows all the versions of a file in a directory listing.

To view a bucket as it was at a certain point in time, you can use the `--b2-version-at` flag, which shows the file versions as they were at that time.

If you want to remove all the old versions of files, you can use the `rclone cleanup` command, which deletes all the old versions of files, leaving the current ones intact.

Here are the different options for managing old versions of files:

Remember that deleting old versions of files will cause the current versions to become hidden old versions.

The Download URL configuration in Rclone is a crucial setting for optimizing file downloads. You can specify a custom endpoint for downloads using the `--b2-download-url` option.

This custom endpoint is usually a Cloudflare CDN URL, which offers free egress for data downloaded through the Cloudflare network. You can leave this setting blank if you want to use the default endpoint provided by Backblaze.

The URL you provide should have the protocol and should not have a trailing slash or specify the /file/bucket subpath, as Rclone will request files with "{download_url}/file/{bucket_name}/{path}". For example, you can set it to https://mysubdomain.mydomain.tld.

Here are the details of the `--b2-download-url` option:

SHA1 checksums are checked on upload and download, and are used in the syncing process.

You can upload large files in chunks, and the SHA1 checksum will be stored as X-Bz-Info-large_file_sha1, following Backblaze's recommendation.

The local disk supports SHA1 checksums, so large file transfers from the local disk will always have an SHA1 checksum.

However, sources that don't support SHA1, such as crypt, may upload large files without an SHA1 checksum.

Files sizes below the cutoff limit will always have an SHA1 checksum, regardless of the source.

SHA1 checksums are a crucial part of the syncing process, and it's essential to ensure they are enabled and working correctly.

The Description option in Rclone configuration allows you to provide a custom description for your remote.

You can configure the description by setting the RCLONE_B2_DESCRIPTION environment variable.

The description type is a string and it's not required to set a value for it.

Here are the details of the Description option:

Transfers and Uploads are key to maximizing speed when using rclone with Backblaze B2. The default setting of --transfers 4 is too low for optimal performance.

To achieve maximum speed, it's recommended to do lots of transfers simultaneously, with an optimum setting of about --transfers 32, although higher numbers may provide a slight speed improvement. However, this may vary depending on your hardware, file sizes, and system load.

You can also opt to use a 96 MiB RAM buffer for uploading big files, which sets the upper limit on memory used. This buffer is used for files bigger than 200 MiB by default.

Syncing your files to the cloud can be a game-changer for your workflow. It allows you to access your files from anywhere, on any device, as long as you have an internet connection.

You can sync files to the cloud using services like Google Drive, Dropbox, or Microsoft OneDrive. These services offer a certain amount of free storage space, which can range from 2GB to 15GB.

Syncing your files to the cloud can also help you collaborate with others in real-time. For example, if you're working on a project with a team, you can all access the same file and make changes simultaneously.

Some cloud services, like Google Drive, also offer automatic backup features that can save your files in case your device crashes or is lost. This can be a huge relief for people who work on important projects.

You can also set up automatic syncing to ensure that your files are always up to date. This can be done through the cloud service's settings, and it's usually a simple process.

To get the most out of Backblaze B2, you want to do lots of transfers simultaneously for maximum speed. This is especially true for big files, where you'll see a significant improvement.

Transfers are also limited by the amount of RAM you have available. By default, uploading big files (bigger than 200 MiB) will use a 96 MiB RAM buffer, and you can have at most --transfers of these in use at any moment.

You can adjust the number of simultaneous transfers to suit your needs. In tests, the optimum setting is around --transfers 32, but you may need to experiment to find the right balance for your hardware.

Here's a quick rundown of the default settings:

These numbers can be adjusted using the --transfers and --b2-upload-concurrency flags, respectively. Keep in mind that increasing these numbers can improve speed, but may also increase memory usage.

The upload cutoff is a crucial setting that determines when to switch to chunked upload. Files above this size will be uploaded in chunks of "--b2-chunk-size".

This value should be set no larger than 4.657 GiB (== 5 GB). If you set it higher, it could lead to issues with your uploads.

You can configure the upload cutoff through the "upload_cutoff" setting or by using the environment variable "RCLONE_B2_UPLOAD_CUTOFF".

The copy cutoff is a crucial setting when transferring large files with rclone. Any files larger than the specified cutoff will be copied in chunks of that size.

The minimum copy cutoff is 0, but it's generally not recommended to use this value. The maximum allowed cutoff is 4.6 GiB.

You can configure the copy cutoff through the rclone config or by setting the RCLONE_B2_COPY_CUTOFF environment variable. The default value is 4 GiB.

Here's a summary of the copy cutoff options: