Adam's blog: More off-site backup storage is needed

Backing data up is vital if we do not want to lose it. My NextCloud server saves all data to a BTRFS partition with daily snapshots, which are then weekly synced with a local secondary disc in case of a primary disc failure. But what to do in case of a lightning strike that would overload surge protection? For that reason, I also keep an off-site Raspberry PI, which works as a tertiary form of backup. The only problem is that it had just run out of storage.

How is the RPi storage connected?

Because I want to run the things as cost-effective as possible, the RPi runs with a single USB connected 4.5TiB external harddrive. It is also attached to an UniPi extension, which allows me to control eight relays with REST API. This gives me an opportunity to power on and off the connected external hard drive whenever needed.

What is the off-site backup process?

Every week, the RPi powers on the connected hard drive with a single LUKS-encrypted BTRFS partition. Because the data inside the backup are write-mostly, I am using BTRFS’s transparent compression to maximalize available virtual space. After a successful mount, the RPi syncs data from the primary NextCloud drive using rsync over SSH (with options --inplace and --no-whole-file to keep the advantage of BTRFS’s CoW feature). The privileges separation is designed so that at no point in time does any of the servers have write access to the other’s data to minimize the possibility of human error deleting all of it. After a successful sync, a new BTRFS snapshot is created.

How to extend the storage?

Thanks to the fact that the disc is formatted to use BTRFS, all I need to do is to call a btrfs device add command on a second device. This seems like a job for my old 640GB Fujitsu-SIEMENS Storagebird 35EV821 (sale ended before 2012). Surely, before using it, I have to make sure that it still works properly, and because this old drive does not support SMART, I have to stick with a good-old baddblocks command (BEWARE: this command will rewrite the whole drive, do not copy it blindly):

> badblocks -o badblocks.log -svw -b 512 -c 65536 /dev/sde
Checking for bad blocks in read-write mode
From block 0 to 1250263727
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff:done
Reading and comparing: done
Testing with pattern 0x00: 60.57% done, 42:04:35 elapsed. (0/0/0 errors)
Interrupted at block 758382592

I caused the interruption because the badblocks was unlikely to find new damaged blocks after multiple successful passes, so it was expected. Furthermore, the backup will be tested monthly with BTRFS’s scrub to ensure data consistency so that any potential hidden problems will be soon flagged.

The disc seems OK, so it is time to connect it to the RPi. First - I need to cut the power wire and insert it into an UniPi relay so that I can power it on and off as needed, like the first disc.

Finishing the software connection

First, the disc has to be encrypted. Because it is now empty, it has no ID except for the standard /dev/sda path:

> cryptsetup luksFormat /dev/sda
WARNING!
=====
This will overwrite data on /dev/sda irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sda: ********
Verify passphrase: ********

Next, because the /dev/sda is not permanent and can change based on the time the kernel detects a new drive, I want to use the drive UUID:

> ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 9 Jun 4 18:06  1234UUID -> ../../sda

Now that we have the UUID of the disc, we can use it to open the LUKS encrypted content (which is currently empty):

> cryptsetup luksOpen /dev/disk/by-uuid/12134UUID cryptBackup2

And now comes the magic moment I have been waiting on – adding the storage:

> df -h /media/backup/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cryptBackup1 4.6T 4.6T 33G 100% /media/backup
> btrfs device add -f /dev/mapper/cryptBackup2 /media/backup/
> df -h /media/backup/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cryptBackup1 5.2T 4.6T 629G 89% /media/backup

Done! My off-site backup server has almost 600GB of new storage (effectively more because of the transparent BTRFS compression).

Conclusion

My off-site backup server has run out of storage, but thanks to the magic of BTRFS, extending it was a piece of cake! The process, of course, involved more steps like connecting the disc power to a controlled relay and using transparent encryption.