Automating tarsnap backups

In my last post I wrote about backing up my dedicated server and why I decided to use tarsnap. After a couple of months of running tarsnap manually I decided it was way past the time to properly automate it.

The main issue is how many snapshots do you want to store? On the one hand it’s nice to be able to go back in time as far as possible, but on the other hand there’s the issue of how large your archives get (and consequently the cost).

There are three different charges for tarsnap; data sent, data received and data stored. Each is charged on a daily basis and subtracted from a total in your account (you keep an account in credit rather than being billed). If you’re doing backups on a daily basis the data sent and received will be approximately the same regardless of how long you retain the archives for. So the figure to consider is the cost for storing the data.

I decided to go for a model where I had X daily backups, Y weekly backups and Z monthly backups. I also decided I wanted to back up only certain directories, and that I wanted to keep them as separate archives (because I’m dealing with large numbers of files, and this breaks it down a bit – I don’t think it affects costs).

So I went about scripting this. First step was to write a “fake” tarsnap. The reasoning behind this was that it’d allow me to do quick backup runs without any time used for archiving or any costs. It’s basically just a perl script that adds and removes archives from a database file.

Next I wrote a backup script. It’s pretty basic at the moment, but fully automates the creation of archives and deletion of expired ones. You provide it with a list of directories to back up, and how many daily, weekly and monthly archives you want to keep. Then stick it in cron and off it goes.

It’s a bit tailored to my setup, and may only work on FreeBSD (are the date flags the same on other operating systems?). Also, its cleaning of old archives is primitive; it’s based on the number of archives, rather than the age.

I welcome feedback on these scripts and improvements, but bear in mind they’re very much a work in progress.

  • Share/Bookmark

Related posts:

  1. Machine backups using tarsnap A look at backup options for my server and why I decided to go with tarsnap....
  2. Maildirarc – a Maildir archiving tool I keep my email in Maildir folders. It works well on the whole for every-day email, but it doesn’t work so well for large email archives (mainly because Unix systems don’t tend to cope well with folders containing a very large number of files). My system of archiving had been to simply copy messages older […]...
  3. FreeBSD filesystem snapshots Being able to take snapshots of a filesystem is very handy for backups, but does it actually work on FreeBSD?...
  4. A new router (Soekris, Draytek and NanoBSD) A look at how I replaced my ageing router and ADSL modem with a Soekris net5501 running FreeBSD....

Tags: , , , ,

8 Responses to “Automating tarsnap backups”

  1. Gabriel says:

    Works well on Linux, except tail expect a -n flag before +$DAILY, +$WEEKLY and +$MONTHLY.

    Thank you for your script !

  2. Pig Monkey says:

    Nice shell script!

    I found your post this morning when searching for ways to automate Tarsnap. It inspired me to write my own script, which does something sorta-kinda similar, but deletes old archives based on age. And it’s written in Python, because my shell-fu is weak.

    https://github.com/pigmonkey/backups/blob/master/tarsnapper.py

  3. Larry Hynes says:

    Thanks for this Tim, it’s working just fine on OSX Mountain Lion.

  4. Pronoiac says:

    I liked this, but I wanted to split up the archiving and the pruning, and I let cron handle the daily/weekly/monthly labels. I posted it here:

    https://github.com/pronoiac/tarsnap-cron

  5. Chris says:

    I’m using the script but ran into a problem when running it on a hourly basis. Basically on mondays the script would keep creating a weekly backup each hour and delete older backups from pervious weeks (I’ve set WEEKLY=4). Is there any way to prevent this?

  6. Tim Bishop says:

    Chris – yeah, it basically needs hacking to add hourly support. It’d need to know the hour of day to do daily/weekly operations on, and otherwise do hourly. I’ve got a copy somewhere that does that, but I never got around to uploading it.

  7. Pronoiac says:

    I think my fork can handle that with a bit of work. I’ll look into it.

  8. Pronoiac says:

    Ok, I think I’ve got hourly backups and pruning going in my fork, mentioned above. I changed the archive filenames to taste, so unfortunately, it’s not a drop-in solution; switching back and forth isn’t recommended.

Leave a Reply

*