A Blog by Tim Bishop » FreeBSD

Automating tarsnap backups

Tim Bishop — Wed, 28 Jan 2009 19:18:07 +0000

In my last post I wrote about backing up my dedicated server and why I decided to use tarsnap. After a couple of months of running tarsnap manually I decided it was way past the time to properly automate it.

The main issue is how many snapshots do you want to store? On the one hand it’s nice to be able to go back in time as far as possible, but on the other hand there’s the issue of how large your archives get (and consequently the cost).

There are three different charges for tarsnap; data sent, data received and data stored. Each is charged on a daily basis and subtracted from a total in your account (you keep an account in credit rather than being billed). If you’re doing backups on a daily basis the data sent and received will be approximately the same regardless of how long you retain the archives for. So the figure to consider is the cost for storing the data.

I decided to go for a model where I had X daily backups, Y weekly backups and Z monthly backups. I also decided I wanted to back up only certain directories, and that I wanted to keep them as separate archives (because I’m dealing with large numbers of files, and this breaks it down a bit – I don’t think it affects costs).

So I went about scripting this. First step was to write a “fake” tarsnap. The reasoning behind this was that it’d allow me to do quick backup runs without any time used for archiving or any costs. It’s basically just a perl script that adds and removes archives from a database file.

Next I wrote a backup script. It’s pretty basic at the moment, but fully automates the creation of archives and deletion of expired ones. You provide it with a list of directories to back up, and how many daily, weekly and monthly archives you want to keep. Then stick it in cron and off it goes.

It’s a bit tailored to my setup, and may only work on FreeBSD (are the date flags the same on other operating systems?). Also, its cleaning of old archives is primitive; it’s based on the number of archives, rather than the age.

I welcome feedback on these scripts and improvements, but bear in mind they’re very much a work in progress.

Machine backups using tarsnap

Tim Bishop — Wed, 28 Jan 2009 16:34:54 +0000

I’ve got a dedicated server that I’ve been backing up for the past few years. My crude backup system involved taring everything to local disk and then rsyncing it to a remote server. It worked well at first, but as the amount of data grew it was taking half a day to run. Add to that the amount of disk space being used by the local copy and I had to find a better solution.

I started off by looking at replacements for rsyncing to my remote server. rdiff-backup seemed an obvious choice, but some friends had previously had problems with it, so I decided to give it a miss. There were various other similar tools that involved copying my data to another server (of mine), possibly with incremental features, but I decided to expand my search criteria and came across online backup services.

These services involve paying someone else to store your data remotely. In principal there didn’t seem to be much point; I had somewhere else to put my data, so why pay for it? One solution that seemed quite nice was brackup – a set of perl scripts that backed up to Amazon S3. Still not quite what I wanted though.

Then, I remembered a project Colin Percival was working on – tarsnap. It’d be a while since I last checked up on progress so I was quite pleased to see that he’d just launched the public beta. I signed up and handed over some money. What made me decide on this solution over others? Well,

1. It works pretty much like tar, except the archives are remotely stored. So that made it simple to use and fit it to my way of working.

2. It has all the benefits of snapshots with all the benefits of incremental backups. By this I mean that you can access each archive as if it was a full copy, but behind the scenes it only transmits and stores the differences (and charges you accordingly). You can delete any archive in the set without worrying about things like needing all the incrementals to the last full backup, and it sorts it all out for you.

3. Colin is a FreeBSD guy, so that pretty much guarantees it’s going to work well on FreeBSD and be supported in the future.

4. After doing the sums I realised the charges are pretty insignificant compared with the cost of running my own dedicated server.

5. There’s a whole load of stuff about security. It doesn’t particularly interest me though, but it’s nice to know my data is safe (nobody can read it – Colin included – without my key).

So I’ve been using tarsnap now for a few months and on the whole I’ve been very happy. There was a minor issue with the large number of files (Maildir email storage can get insane ) I had to backup, but once I discovered the --lowmem flag all was fine. And Colin’s support has been great – he’s always been quick to answer if I’ve had a question or problem.

If you’re looking for an online backup service that’s geared towards Unix systems I highly recommend giving tarsnap a try.

(How about reading my post on Automating Tarsnap Backups?)

FreeBSD filesystem snapshots

Tim Bishop — Mon, 10 Nov 2008 13:13:26 +0000

I’ve been looking at filesystem snapshots on FreeBSD lately and I have to say I wasn’t too impressed. Yes, the functionality is there, but the interface sucks. For UFS you have to create a snapshot, which takes a while, and which appears as a file within the filesystem you’re snapshotting. Then you have to use an md device to mount it. ZFS is easier, but different. What we need is a simple and standard interface to both.

Fortunately I discovered Ralf Engelschall’s snapshot management utilities. He’s written a couple of tools that make creating and managing snapshots really easy.

Using cron one can schedule creation of hourly, daily and weekly snapshots for filesystems. You can specify that you want 3 hourly snapshots, 2 daily and 1 weekly for a given filesystem. The system takes care of everything else. And by using amd (the FreeBSD automounter) these snapshots can automatically be made available through a given mount point. It couldn’t get any easier!

Well, actually, it could get slightly easier. Currently these tools are available from Ralf’s site or from the FreeBSD ports collection. It strikes me that something as useful and fundamental as this should really make its way in to the base system.

Update: Whilst this worked well in testing, once I applied it to my server with a large (approx, but not over, 2TB) filesystem it hung the machine completely, even on the console. I haven’t investigated further yet.

Update 2: Well, I have to say I’m disappointed. Ralf’s scripts worked great, but the snapshotting of large UFS filesystems in FreeBSD is as good as broken. When it takes hours to create a snapshot, locking out the filesystem (and maybe even the machine) for hours, it might as well not be there. There seems to be an attitude of “just accept it”, which I’m not impressed about either. Roll on ZFS… (yes – I know it’s there now, but I’d like it to mature just a little more ).

FreeBSD with Netgear WG311T

Tim Bishop — Sat, 08 Nov 2008 13:15:58 +0000

A few days I wrote about my new Soekris net5501 router. In that post I mentioned that the only thing left to sort out was the wireless card. It turned out to be simpler to do than I thought.

I decided to go for a Netgear WG311T. It’s a 802.11b/g PCI card that’s compatible with FreeBSD through the Atheros chipset and ath driver, and it fits in the net5501 just fine. As expected I had to remove the net5501 board from the case to attach the card, but that only involved undoing a handful of screws.

Getting it working on FreeBSD was trivial. I added the following lines to my kernel configuration (they’re already there in GENERIC, I believe, but I built my own kernel because of the net5501):

device wlan device wlan_ccmp device wlan_scan_ap device wlan_scan_sta device wlan_xauth device ath device ath_hal device ath_rate_sample

Then it was a simple case of initialising the card in /etc/rc.conf:

ifconfig_ath0="inet 1.2.3.4 netmask 255.255.255.0 ssid myssid mode 11g mediaopt hostap" ipv6_ifconfig_ath0="1:2:3::4 prefixlen 64"

And I also added ath0 to rtadvd_interfaces and dhcpd_interfaces.

With that done the final step was to configure hostapd through /etc/hostapd.conf:

interface=ath0 debug=1 ctrl_interface=/var/run/hostapd ctrl_interface_group=wheel ssid=myssid country_code=GB wpa=2 wpa_passphrase=my passphrase wpa_key_mgmt=WPA-PSK wpa_pairwise=CCMP

This enables WPA2 using AES (rather than TKIP).

Connecting clients was no problem. My laptop and my squeezeboxes all connected fine.

One concern I had doing this was whether a PCI wireless card would have the same signal strength as a purpose access point. I seem to be getting the same signal power from this new setup as I did from my old 3com 802.11b access point. What does that tell me? I’m not sure. I would have expected technology to have come on a bit over the years. Maybe it has, but the fact that the card is lower powered balances it out? Regardless, it covers my whole house, so it’s not an issue.

So now I’m done. I’ve switched off my last piece of old equipment. My power draw has dropped significantly, the noise levels have decreased, and I’m a happy geek with a cupboard full of shiny new stuff.

Eduroam on FreeBSD

Tim Bishop — Fri, 07 Nov 2008 12:06:20 +0000

We use the Eduroam wireless network at the University of Kent. There’s various guides for getting it working on Linux, but I thought I’d try on FreeBSD. It’s pretty simple.

First create /etc/wpa_supplicant.conf with the following content:

network={ ssid="eduroam" proto=WPA WPA2 key_mgmt=WPA-EAP eap=PEAP group=TKIP identity="tdb@kent.ac.uk" password="YourPassword" }

Then add the following to /etc/rc.conf:

ifconfig_ath0="WPA DHCP"

Replacing ath0 with your wireless adapter if it’s different.

Then start it up as follows (or reboot):

# /etc/rc.d/netif start

You can use ifconfig to confirm the link is up, and ping to test.

For more general information about wireless networking in FreeBSD please see this section of the handbook.

A new router (Soekris, Draytek and NanoBSD)

Tim Bishop — Wed, 05 Nov 2008 12:35:31 +0000

A few months back I wrote about building a new server. It turned out to be more complicated than I thought, but 5 months on it’s still working well. Over the last few weeks I’ve been working on my next project – replacing my router.

The old router was an old dual CPU Pentium 3 machine with a couple of small SCSI hard disks in it. It was a full tower case and took up a lot of room and made a lot of noise. And, surprisingly, the power consumption was pretty similar to the server I recently built (which has way more in it). It even still had the original Speedtouch USB modem that BT once gave to me. So it had to go.

I spent quite some time deliberating the way forward. I could have gone for a domestic router like most other people do, but I’m a geek, and I like the flexibility of doing it myself. But at the same time, a small sized unit, with low power requirements, and no noise, is what I wanted. The solution came in the form of a Soekris net5501.

I went for the net5501-70 which has a 500Mhz CPU and 512MB of RAM. Not a lot by today’s standards, but more than sufficient for what I needed. And incidently, it’s quicker than the old router. I got the model with a case, and got the mounting brackets for a hard drive (which although I don’t intend to use at this stage, it was cheaper to get it now than later). I also purchased a pair of 4GB SanDisk Extreme III Compact Flash cards to run the thing from. It’s worth noting that Soekris recommend SanDisk CF cards, and they’re peanuts at play.com.

The next point to consider was how to connect to the ADSL line. I could have stuck with the USB modem, but the drivers were aging, I wasn’t sure if it was the cause of the odd disconnections and failures to reconnect that I’d been getting. I looked at internal ADSL cards, but it seemed to be a bit of a gamble as to how they worked and if I’d be able to get the right drivers. In the end I settled on the Draytek Vigor 110.

The Vigor 110 is basically just a PPPoA to PPPoE bridge. PPPoE isn’t widely used in the UK, but is in other parts of the world, so the support in FreeBSD was good (via ppp and the ng_pppoe module). It worked perfectly. It really was just a case of plugging it in and pointing ppp at it – no configuration required! And, just like the USB modem, it gives the router IP directly to the PC, so there’s no messing around to get that working either.

Longer term I plan to fill the net5501’s PCI slot with a wireless card, but I haven’t decided which to go for yet. This would turn the unit in to my wifi access point as well, but for now I’ll just stick with the separate one. I’d welcome advice on cards that are supported by FreeBSD.

So, that’s hardware all sorted. Next came the software. If you’ve been following my other posts you’ll know I’m a big fan of FreeBSD, so it’s pretty clear what route I was going to take here. But given the use of CF cards I had to approach things differently. I also wanted to be able to power the system off without causing any filesystem problems, so this required the card to be mounted read-only.

NanoBSD to the rescue! NanoBSD is a script that builds an image containing FreeBSD that can be written directly to a CF card (or anything else, really). It’s customisable, and I wrote a few bits to pull down the packages I wanted, and to make some configuration tweaks. It has a read-only root filesystem on the card, and uses memory-backed filesystems for /var and /etc. Config is stored in a separate partition on the disk and is copied to the memory-backed /etc during the boot process. But the best bit is the way it handles upgrades.

Upgrades are neatly done by having two root filesystems on the card. When you’re running off one you’re free to upgrade the other. NanoBSD generates two images; one for the entire card, and one that can be written to a single root filesystem. It also provides a script to write the image to the card and update the boot loader to boot from the right partition. So upgrading is as simple as re-running the NanoBSD script, writing the new image to the “other” partition, and rebooting. It can all be done live, and the only downtime is the time taken for a reboot (which is under a minute).

Of course, to use NanoBSD you need another system to do the builds on. Fortunately I’ve got a nice beefy server that can handle the job (although it took a few hacks to build the i386 image on an amd64 system). I’ve also got a nice Tinderbox setup which I already use for testing ports and which provided a nice supply of up-to-date packages.

So I’m happy, at last. Apart from my wifi access point I’ve managed to replace all my aging, power hungry, noisy equipment with nice new stuff. I guess I’ll be doing it all again in a few years .

FreeBSD stuff

Tim Bishop — Wed, 16 Jul 2008 16:22:27 +0000

I’ve done a bit of work on my FreeBSD ports lately. Firstly, after building my new server, I got round to upgrading from SlimServer to SqueezeCenter. This also meant sorting out ports for all the plugins I use. This didn’t take too long, and you can find them all over here. So far I’m liking SqueezeCenter, and I’d highly recommend it (and a SqueezeBox, of course).

I also maintain a port for a suite of software called KRoC. KRoC is written and maintained where I work, so apart from making it available to FreeBSD users I also have an interest in supporting the work done by our department. I’ve been waiting some time for a 1.5.x release of KRoC, but I finally got impatient. I automated the production of snapshots from their stable branch, and updated the port to build from that. I also run a FreeBSD 7 machine in their buildbot system to further test KRoC on my favourite operating system

And in other FreeBSD news, I cast my vote in the FreeBSD Core elections. It’s hard to know who to vote for, but I gave their statements a good read and made a decision. Good luck to them all!

“I’ll build a new server; it’s got to be easier than patching up the old one…”

Tim Bishop — Thu, 19 Jun 2008 22:24:29 +0000

A few weeks back I started having problems with my file server at home. This machine is fairly important to us; it holds all our photos, music and other files. For years I’ve been bodging it together with various old parts scavenged from other machines and some new parts when needed. But, once again, it’d started to break. Disks were dropping out of the RAID unexpectedly, and the replacements were refusing to rebuild. Unsure of where the problem was I uttered the fateful words “I’ll build a new server; it’s got to be easier than patching up the old one…”. My colleagues were sceptical, but I ploughed on anyway. Maybe I should have listened to them?

It took the best part of a week to work out what I wanted. There were so many decisions to make: which RAID card, disks, motherboard, CPU, RAM, case, etc. I researched each one as much as I could, but there’s a bottomless pit of information on the Internet. Eventually I settled on a 3ware 9690SA RAID card with 4 Seagate ST31000340NS disks. The other bits were fairly decent to make sure the machine would have a good life, but not excessive.

The reason for choosing a hardware RAID solution over software RAID was simple – reliability. Now, I’m not knocking software RAID in principal (look at ZFS, for example), but the implementations for RAID 3-5 on FreeBSD aren’t great (yes, it has ZFS, but I’m not in the mood for trailblazing this time round). I wanted to stick with FreeBSD so I opted for the well known reliablity that 3ware cards provide. And the 5 year warranty on Seagate disks made them an attractive choice.

The purchasing process wasn’t as simple as it could have been. I ordered from dabs.com, span.com (they specialise in storage stuff) and overclockers.co.uk. I’ve used all three companies before, so I wasn’t too concerned about problems. The bulk of it was ordered from Dabs – it looks like they’re back to being competitive on prices. The problems started almost immediately; Dabs held my order over an issue with my address. It’s happened once before and that put me off Dabs for some time, but we use them all the time at work, so I had hopes they’d be better now. It took a working day to resolve that issue… and then next day I get an email to say my credit card company has declined the order. On the phone to them and through to their security department; seems buying lots of stuff online is unusual… not for me it isn’t. Anyway, that was resolved and then I had more waiting for Dabs to try the transaction again. Eventually I got impatient and tried their online chat thing and the matter was resolved in minutes. Meanwhile the parts ordered from the other two suppliers were sitting on my desk.

Eventually it all arrived and I took it home. Ruth wasn’t overly impressed when I cleared off the dining room table and covered it in computer parts, but I assured her it wasn’t for long. That was a couple of weeks ago – it’s all still there.

I spent a weekend putting things together and testing it all out. I routed every cable neatly and tied them carefully to the case to ensure nothing moved about. Airflow was good and the additional fans in the case were doing a great job of keeping things cool (not sure about their blue LEDs though…). All was looking good and I was enjoying the process.

Then I tried to use the RAID card. The first problems hit when I turned on the motherboard’s RAID, which I’d intended to use to mirror the system disks, whilst the 9690SA was plugged in. I’d gone for a Asus P5E3 and expected both RAID systems to work happily together, but sadly I was wrong. I experienced unusual problems such as the machine hanging on the Intel Matrix Storage (the onboard RAID) screen and disks randomly disappearing from both arrays. In the end I gave up and turned off the onboard RAID; I figured the FreeBSD RAID 1 (gmirror) is pretty solid, so I’d use that.

Thinking I’d got over the worst of the problems I moved on to setting up the 9690SA. Things looked good for a while; the interface was clear and everything was easy to set up. It wasn’t until I started trying to put data on that I noticed problems. Here’s a snippet from the error log (largely for the benefit of Google):

E=0200 T=08:26:00 : Cable CRC error
SATA Device. port = 0x0
task file written out : cd dh ch cl sn sc ft
                      : 00 70 00 00 00 1200 00
  task file read back : st dh ch cl sn sc er
                      : 00 00 00 00 00 8441 00
E=0200 T=08:26:00 P=0h: Soft reset drive
E=0200 T=08:26:00 P=0h: exitCode = 1013
Port retry not allowed
E=0200 T=08:26:00 P=0h: Prepare for command retry
exitCode = 1013

At first I wasn’t sure what to make of this. Maybe it was the cable or connection, but on all four drives? It was a special 4-in-1 (SFF8087) cable, but it still seemed odd. I logged the case with 3ware’s technical support and got back a response suggesting I try another cable. Well, duh, I could have figured that myself. I was hoping they might be able to point out any other less obvious potential causes.

So, I purchased another cable. It took a couple of days to arrive and did absolutely nothing to resolve the problem. Sigh. At the same time as this was going on I had another problem – it’s only with hindsight that I know to separate the two:

E=0204 T=18:34:36     : Port timeout (ext)
SATA Device. port = 0x2
task file written out : cd dh ch cl sn sc ft
                      : 00 04 00 00 00 00 00
Send AEN (code, time): 0x9, 06/10/2008 18:34:36
Drive timeout detected
(EC:0x09, SK=0x04, ASC=0x00, ASCQ=0x00, SEV=01, Type=0x71)
phy=6
  task file read back : st dh ch cl sn sc er
                      : 00 00 00 00 00 00 00
E=0204 T=18:34:36 P=2h: Soft reset drive
E=0204 T=18:34:36 P=2 : Inserting Set UDMA command
E=0204 T=18:34:36 P=2h: Check power cycles, initial=40, current=40
E=0204 T=18:34:36 P=2h: exitCode = 1013
Port retry not allowed
E=0204 T=18:34:36 P=2h: Prepare for command retry
exitCode = 1013
E=0204 T=18:34:36 U=0 : Retrying command

These errors happened less frequently, but eventually caused I/O to hang and the controller to reset. Again I logged this with 3ware’s technical support and got back a bunch of not so helpful responses. They suggested moving the card in the machine, testing the disks, checking the power supply, and so on. All valid points, but what annoyed me was they could only ask me to check one at a time… and they could only reply to me once a day. Plus I’d already done everything they suggested. It took a week to go through this nonsense.

In the mean time I spent a lot of time experimenting, fiddling, and web searching. Eventually I found the following two pages, although it took me a while to realise their significance:

http://www.3ware.com/KB/article.aspx?id=15385
http://www.3ware.com/kb/article.aspx?id=15171

The first of the articles explicitly mentions my controller card and drives, so it seemed to be the right thing to do. But I had the SN04 firmware on my drives and they wanted me to apply AN05. I asked both 3ware and Seagate to clarify the differences, but neither gave satisfactory answers. Seagate managed to give me the SN05 firmware to try, but it didn’t help. In slight desperation, and without anyone giving me much help, I decided to take a punt on the AN05 firmware.

IT WORKED!

There was a lot of tension for the next few hours whilst I continued testing, but eventually I was satisfied that the AN05 firmware solved the problem. Later attempts to clarify with Seagate why SN05, which they gave me, didn’t work and AN05, which 3ware pointed me at, did work, got nowhere. Seagate support actually admitted that they basically don’t know.

So on to the next issue. The second article suggested limiting the speed of the drives to work around the drive timeout issue. It’s definately a workaround, but it was worth a shot. I’d already removed the jumpers from the drives that limited them to 1.5 Gb/s, and they were a nightmare to do – I’ve never seen such small and fiddly jumpers on a disk… it was completely unnecessary given the available space. This time I decided to do the limiting in the 9690SA’s software.

ONCE AGAIN, IT WORKED!

So at this point I’m happy. Things are looking good. That last fix is definately a workaround, and I’ve told 3ware they need to fix it. It’s a bug, and bugs need fixing. I’m now using the array to store my data on, it’s nice and quick (a 512MB write cache helps!), and I have plenty of space. And Ruth might get the dining room table back soon… assuming I can work out how to lift this massive machine (did I mention the case was quite big?).

But I’d like to finish this post with a rant. It turned out that the solutions to my problems were both in the 3ware knowledge base. Now maybe I should have searched harder initially, but it took me some time to find these articles. But more to the point, 3ware support should definately have known about these issues and should have directed me straight to them. I wasted a week of my time messing around with them, and I’m not happy about it. The card is great (apart from the aforementioned bug), but the support sucks. It will seriously make me think twice about going with 3ware again.

I hope this post will fill in the whole story to those I’ve been ranting at recently, and maybe it’ll help someone else on the Internet out if/when they hit the same problem. That’s assuming they can read this lengthy post in less time that in takes to figure out the solution themselves .

Good night.