Restic backups to B2 without delete

Update 2023-03-28: Restic now supports limited keys by default, and also recommends the B2 S3 api instead. See here

Original Post:

Since setting up restic to back up to b2 I had embarassingly missunderstood the following UI wording around lifecycle settings:

I had assumed that this meant that all versions of the file would be maintained, no matter what clients did. The B2 docs are much clearer, explaining that these rules are really only automation on your existing files. They're not blocks/holds/locks which can take any action before or affect api calls.

In short: Lifecycle Rules schedule cleaning up version history, they don't prevent you deleting that version history.

One of the capabilities I want from my backup system is that the system backing data up should not also be able to destroy existing backups. (I'm not bothered about it reading previous backups) I had assumed lifecycle rules gave me this, I was wrong, time for a quick fix...

The obvious solution is to restrict the capabilities of the b2 api key during creation, specifically to omit deleteFiles. Unfortunately, restic doesn't currently handle this well, for example during basic backups it requires the ability to delete files in the locks/ directory and will complain loudly. There are a few open issues and existing [PR]https://github.com/restic/restic/pull/2398(https://github.com/restic/restic/pull/2398)'s but nothing yet merged.

Enter rclone

A few years ago rclone gained support for serving storage to restic via it's api and since rclone supports using soft delete by default swapping to this backend is pretty trivial.

One of the best bits is that you can continue to use your existing config, the repository format is unchanged.

The first thing to do is to generate a new key without deleteFiles:

$ b2 authorize-account
Using https://api.backblazeb2.com
Backblaze application key ID: <account key id>
Backblaze application key: 

$ b2 create-key --bucket bucketname keyname listBuckets,listFiles,readFiles,writeFiles
<app key id> <app key secret>

...and then take that output to set up an rclone backend. You can either go through rclone config (See their docs for an example) or just drop a file into place:

$ cat ~/.config/rclone/rclone.conf
[b2]
type = b2
account = <app key id>
key = <app key secret>

Then all that's left is to update your backup script:

  1. e.g. if previously it was
    b2:bucketname:foldername
    then now it'll be
    rclone:b2:bucketname/foldername
    (Note that the second part (b2) now refers to the name of the rclone backend, not the type. e.g. the bit within the square brackets of rclone.conf config)
  2. Alter the options rclone is called with. Annoyingly, even though rclone uses soft delete in b2 by default, restic sets --b2-hard-delete by default... so add the following option whenever you run restic:
    -o rclone.args="serve restic --stdio"

Back to the lifecycle...

Beyond restic forget/prune to manage old snapshots, we've now got to consider cleaning up soft deleted files. Given my backups are largely additive, forget rarely frees much space so it's not my biggest priority.

rclone provides a handy cleanup command, but it's very tempting to go back to where I began and let Backblaze b2 lifecycles clean them up without any extra scheduling on your own infrastructure.

Obligatory https://restic.readthedocs.io/en/stable/045_working_with_repos.html#checking-integrity-and-consistency reminder.