The main issue is how many snapshots do you want to store? On the one hand it’s nice to be able to go back in time as far as possible, but on the other hand there’s the issue of how large your archives get (and consequently the cost).

There are three different charges for tarsnap; data sent, data received and data stored. Each is charged on a daily basis and subtracted from a total in your account (you keep an account in credit rather than being billed). If you’re doing backups on a daily basis the data sent and received will be approximately the same regardless of how long you retain the archives for. So the figure to consider is the cost for storing the data.

I decided to go for a model where I had X daily backups, Y weekly backups and Z monthly backups. I also decided I wanted to back up only certain directories, and that I wanted to keep them as separate archives (because I’m dealing with large numbers of files, and this breaks it down a bit – I don’t think it affects costs).

So I went about scripting this. First step was to write a “fake” tarsnap. The reasoning behind this was that it’d allow me to do quick backup runs without any time used for archiving or any costs. It’s basically just a perl script that adds and removes archives from a database file.

Next I wrote a backup script. It’s pretty basic at the moment, but fully automates the creation of archives and deletion of expired ones. You provide it with a list of directories to back up, and how many daily, weekly and monthly archives you want to keep. Then stick it in cron and off it goes.

It’s a bit tailored to my setup, and may only work on FreeBSD (are the date flags the same on other operating systems?). Also, its cleaning of old archives is primitive; it’s based on the number of archives, rather than the age.

• Tarsnap backup script
• “Fake” tarsnap for testing backups

I welcome feedback on these scripts and improvements, but bear in mind they’re very much a work in progress.

Related posts:

1. Machine backups using tarsnap
2. Maildirarc – a Maildir archiving tool
3. FreeBSD filesystem snapshots

I started off by looking at replacements for rsyncing to my remote server. rdiff-backup seemed an obvious choice, but some friends had previously had problems with it, so I decided to give it a miss. There were various other similar tools that involved copying my data to another server (of mine), possibly with incremental features, but I decided to expand my search criteria and came across online backup services.

These services involve paying someone else to store your data remotely. In principal there didn’t seem to be much point; I had somewhere else to put my data, so why pay for it? One solution that seemed quite nice was brackup – a set of perl scripts that backed up to Amazon S3. Still not quite what I wanted though.

Then, I remembered a project Colin Percival was working on – tarsnap. It’d be a while since I last checked up on progress so I was quite pleased to see that he’d just launched the public beta. I signed up and handed over some money. What made me decide on this solution over others? Well,

1. It works pretty much like tar, except the archives are remotely stored. So that made it simple to use and fit it to my way of working.

2. It has all the benefits of snapshots with all the benefits of incremental backups. By this I mean that you can access each archive as if it was a full copy, but behind the scenes it only transmits and stores the differences (and charges you accordingly). You can delete any archive in the set without worrying about things like needing all the incrementals to the last full backup, and it sorts it all out for you.

3. Colin is a FreeBSD guy, so that pretty much guarantees it’s going to work well on FreeBSD and be supported in the future.

4. After doing the sums I realised the charges are pretty insignificant compared with the cost of running my own dedicated server.

5. There’s a whole load of stuff about security. It doesn’t particularly interest me though, but it’s nice to know my data is safe (nobody can read it – Colin included – without my key).

So I’ve been using tarsnap now for a few months and on the whole I’ve been very happy. There was a minor issue with the large number of files (Maildir email storage can get insane ) I had to backup, but once I discovered the --lowmem flag all was fine. And Colin’s support has been great – he’s always been quick to answer if I’ve had a question or problem.

If you’re looking for an online backup service that’s geared towards Unix systems I highly recommend giving tarsnap a try.

(How about reading my post on Automating Tarsnap Backups?)

Related posts:

1. Automating tarsnap backups
2. A new router (Soekris, Draytek and NanoBSD)
3. FreeBSD filesystem snapshots