Switching backups to Restic - update

In a previous post I mentioned swapping my backups to restic. Since then, I've found my B2 storage costs increased substantially and wanted to figure out why.

I had previously used duplicati which by default it checks some of the backend files for integrity:

At the end of each backup job, Duplicati checks the integrity by downloading a few files from the backend. The contents of these file is checked against what Duplicati expects it to be. This procedure increases the reliability of the backup files, but backups take a bit longer to complete and use some download bandwidth.

I wanted to continue this good practice when taking restic into use and you don't get it for free from restic so I set up a script on a daily systemd timer:

# Check repository for errors.
cd /home/restic
DOM=$(date +%d)
DIM=$(date -d "$(date +%m)/1 + 1 month - 1 day" +%d)
restic check \
	--read-data-subset=$DOM/$DIM \
  --option b2.connections=$B2_CONNECTIONS

The idea being that most (I accept not all) of the data in b2 would likely be checked each month. I'm willing to accept that not everything is checked - checking more each day means increased download costs.

I was surprised though when my Backblaze bills increase by a couple of orders of magnitude! The increases were primarily in download & class B transactions, but higher than anticipated. Given I aimed to download & check a full copy of my archive over a month, seeing costs daily in excess of an estimate of monthly cost was surprising. Disabling the check timer gave reassurance it was definitely the script above causing the issue.

After upgrading to 0.9.5 I found the increased costs went away. A quick skim through the project's changelog explained why:

Enhancement #1665: Improve cache handling for restic check

For safety reasons, restic does not use a local metadata cache for the restic check command, so that data is loaded from the repository and restic can check it's in good condition. When the cache is disabled, restic will fetch each tiny blob needed for checking the integrity using a separate backend request. For non-local backends, that will take a long time, and depending on the backend (e.g. B2) may also be much more expensive.

This PR adds a few commits which will change the behavior as follows:

  • When restic check is called without any additional parameters, it will build a new cache in a temporary directory, which is removed at the end of the check. This way, we'll get readahead for metadata files (so restic will fetch the whole file when the first blob from the file is requested), but all data is freshly fetched from the storage backend. This is the default behavior and will work for almost all users.

  • When restic check is called with --with-cache, the default on-disc cache is used. This behavior hasn't changed since the cache was introduced.

  • When --no-cache is specified, restic falls back to the old behavior, and read all tiny blobs in separate requests.

https://github.com/restic/restic/issues/1665 https://github.com/restic/restic/issues/1694 https://github.com/restic/restic/pull/1696

In short, restic <0.9.0 checks each file by refetching the blob that contains it from the backend, without any cache. restic > 0.9.0 uses a temporary cache used only for that restic check invocation.

I had intentionally stuck to restic 0.8.3 on my Nextcloud server (it's the default version packaged in Ubuntu's Universe) but this is a clear reason to upgrade. It reminds me of Chris's thoughts on Ubuntu's Universe - this isn't restic broken, just inefficiency causing unnecessary cost.

That's another subject...