Upgrading Debian

If you’ve been following my blog you’ll know that I’ve been working on a new filestore project at work for a while now. After getting things working nicely on our Solaris machines, and finally moving my home directory over, I decided to tackle our Debian server. It quickly became apparent that I’d need to upgrade the machine, which was running Woody with a 2.4 kernel, to get to a decent IPsec and autofs setup.

Now, I’m not a Linux user, let alone a Debian one. So this was a new experience for me. After a quick nose around online, and with a few helpful pointers, I found some useful instructions on how to upgrade. It boils down to a fairly simple process;

  1. Make sure the system is running the latest Woody updates.
  2. Modify apt sources.list file to change woody to sarge.
  3. Run apt-get update.
  4. Install/update aptitude.
  5. Run aptitude -f --with-recommends dist-upgrade to do the full upgrade.

Then it’s just a case of fixing up any conflicting files and changes, and you’re done. I had to remove our backup software (lgtoclnt) and re-add it though, because it messed with the X packages.

I decided at this point to make sure Sarge worked before looking at the kernel. So I rebooted the system. I waited. And I waited some more. The console showed that it had gone through the BIOS and RAID POST, but nothing else. A brief trip back to the machine room showed a scary looking “LI” message, which I knew meant lilo wasn’t working.

At this point I consulted some friends who explained what I needed to do. A short while later, and with a freshly burnt boot CD, I had the system back up and running. To reinstall lilo I’d booted the CD up to the point where it loaded the aacraid drivers, switched to another terminal, mounted my root parition, chrooted, and run lilo.

By this point I’m starting to grumble about Linux/Debian being stupid. But, I move on. I discover that I’m also going to need to upgrade to 2.6 if I’m going to get IPsec support. After a short while of looking at rebuilding kernels, and boggling at the myriad of build options available, I decide to apt-get install kernel-image-2.6. That can’t be too hard, can it? A few moments later I’m left staring at an Oops message referring to a “kernel NULL point deference” which appears to have come from the install running dd.

Nasty. Anyway, to cut a long story short I tweaked the postinst script to stop it running dd, and that allowed me to get the kernel installed. Surprisingly it worked first time, but I did have to fix the modules list afterwards to silence some error messages.

Now a few hours later, and after discovering the difference between autofs4 and the Solaris automounter, I now have a working system. But I’m left wondering why I’d really want to be using Debian at all.


Now what? It’s too scary to use…

Its been months in the making, but it’s finally done. We have our new filestore ready to go. There’s still plenty to do, like rolling it out for the teaching machines and web filestore, but at least we’ve got the main part done.

So why has it taken so long? I spent a long time researching and testing the technologies involved. For example, choosing the file system was tricky. UFS doesn’t work well on large (>2TB) file systems, and VxFS doesn’t work with NFS and Quotas. I managed to solve that one by fixing the quota issue with VxFS. There was also the issue of how we backup this quantity of filestore, and working out how we’d make it available from the cluster to the user machines. In the end we opted for a single filesystem split in to chunks on the server side for backups and used the automounter to make these divisions transparent to the end users.

The other time consuming factor was the software development stage. We have automated systems for creating users on machines, so I needed to integrate this with the new filestore. This required writing code to facilitate the creation of directories, setting up of quotas, and automount map building.

Anyway, I’ve written about this before. So now it’s done what do we do next? The logical step is to test it on myself and/or the rest of the systems group. Personally I’m in of favour testing it on everyone else first, but that doesn’t seem fair 🙂

The question is, am I brave enough to actually use it?


BT Exact IPv6 Tunnel Broker is back

It looks like the BT Exact IPv6 tunnel broker is finally back up and running after being offline for a week. It seems they had a hardware failure of some kind which knocked out their whole TB operation. I appreciate this is a free service, but it’s still a pain not having it available. That said, I was reluctant to change to another broker since so far, ignoring this incident, their service seems to be pretty good:

  • Very simple to set up – no messing around with special applications.
  • UK based, so only 6 hops outside of my ADSL provider’s network, and 4 away from my hosted server’s network. In both cases it’s a single hop straight from the provider’s network on to the BT network.
  • Supports reverse DNS delegation.
  • Simple interface for setting up and modifying tunnels.

Finding an alternative to this would have been hard. Unless anyone has any recommendations?

Obviously this is not an ideal long-term solution; I still have to tunnel over the relevant IPv4 networks to get to the broker. What I really want is native IPv6 straight from my service providers. I guess I expect this sort of service more from my hosting company, and when I last asked they said it’s something they wanted to look at. For the average person, though, this is something that needs to come from the ISP, but that’s probably a long way off.


Why I absolutely hate spam

If there’s one thing that drives me completely insane in the modern world of computing it’s spam. It consumes my time, day after day, and devours the resources of our mail systems. In my own mailbox I get a few hundred spam messages a day, most of which I’ll never even see, let alone read. Thankfully most of these are filtered, but there’s still at least 20+ which I have to manually deal with every morning.

At work the mail systems for the Computer Science department are processing around 20,000 incoming email messages every day. A remakable 61% of these are spam, which is quite an increase from 49% a year ago. We run two mail hubs to process the incoming email which means we’ve effectively had to buy and run one server just for processing the spam email. I don’t even want to start on the amount of time spent dealing with spam messages that make it through to our helpdesk systems.

Ever noticed how spam email comes from rather an ecletic selection of email addresses? Has one of those addresses ever been yours? If there’s one type of email even more annoying that spam it’s bounces generated as a result of spam, sometimes thousands of them. You’ve suddenly become an unwilling victim of spam. Your address abused, and maybe even your name tarnished. What gives spammers the right to do this? At least SPF and similar technologies go some way to preventing this.

And as if spam email wasn’t enough we now see it creeping in to many other Internet based systems. How long until there’s a spam comment on this weblog? Or a stack of spam referrer entries in my apache logs (and consequently my statistics)? Or until I receive the next random message on one of my messenger services?

Whilst I’m ranting, another thing I can’t stand are those pages of junk links that appear when you try and google for something, particularly if it’s a fairly common term. Thankfully google is trying to deal with that, but it’ll be a neverending battle.

It seems in the non-Internet world we can easily regulate junk messages. We used to get a fair amount of sales telephone calls and general junk mail through the front door. Within weeks of registering with the Mail Preference Service and the Telephone Preference Service these have completely stopped. I’m not naive enough to believe this could be done with the Internet, but it helps put things in to perspective.

One of these days I’m going to get sick of the battle and just say “screw ’em all” and unplug my ADSL modem. After all, people keep telling me I should try reading more books.


A new libstatgrab release

We’ve finally done another libstatgrab release. It’s been the best part of 8 months since the last release. Given the length of time you might be mistaken for thinking we’ve made lots of changes, but we haven’t. All this release really includes is some mostly untested Windows support, and handful of bugfixes.

I guess the problem is that we’ve hit a bit of a brick wall. Adding more features is now quite tricky; we’ve done all the common ones that make sense across multiple platforms. Adding more platforms is hard since any new ones would be the slightly more obscure operating systems (otherwise we would have done them already). Add to that our lack of enthusiasm and interest for making any radical changes and you get very little progress.

I suppose it at least works in its current state, so as long as we fix any bugs we find we’re probably keeping people happy.


Router rebuild (or, an excuse to play with IPv6?)

So recently my router decided it didn’t want to whir its fans anymore and consequently gave up on life. It’s a dual CPU machine and both CPU fans had managed to wedge. After fixing them and getting things running again I heard klunking noises coming from the front of the case; one of the disks in the mirror had failed. I rapidly copied everything off the remaining disk, but didn’t have a spare to hand. Next morning the remaining disk went too. I wasn’t having much luck really, but on the positive side I did have a full backup.

After a day or so of fiddling with hardware I got something that resembled a working machine; I’d gone through a stack of various old disks by this point, most of which were dead. For a while I’d been pondering a fresh install for the machine, so this was the perfect opportunity. I decided to think about what I wanted it to do – this is what I came up with.

  1. Obviously needs ADSL connection (via rather old, but working, USB modem)
  2. I’d quite like a VPN connection to work for various (but not all) work servers
  3. IPv6 routing both internally and out to the world
  4. Internal NIC with my private and public address ranges
  5. A second internal NIC for my wireless network
  6. A better firewall setup (I decided on PF in the end)

Rather predictably I decided to do all this with FreeBSD. Nothing exciting about the install, other than I used gmirror this time. I’m still trying to find the best RAID solution on FreeBSD. So far I think gmirror has impressed me most compared to ataraid and gvinum.

So most of the things I wanted the router to do are things it did before. The new things were the VPN, IPv6 and PF. Those are what I’ll write about.

Setting up the VPN was straightforward. I installed the net/pptpclient port, bunged the sample config and my credentials in /etc/ppp/ppp.conf, and knocked up a quick RC script (let me know if you’d like a copy). I also added specific entries to ppp.conf for the hosts I wanted to route over the VPN, rather than letting it route whole subnets.

Something worth noting about ppp is the -unitN flag. Using this you can make sure ppp always uses the same numbered tun device. For example, my VPN connection has -unit1 ensuring it is always tun1. This makes firewall configuration a bit more manageable.

I’ve also knocked up a slightly better RC script for starting the ADSL connection (compared to the one provided with net/pppoa) that checks the line is up before returning. This allows subsequent startup scripts to be pretty much guaranteed access to the Internet. Again, let me know if you’d like a copy.

The next task was getting the IPv6 connection going. I decided to use the BT IPv6 Tunnel Broker service. In retrospect this might not have been the best choice; it’s been down for the last few days. I’ll let you know how I decide to proceed with that, but I’m reluctant to change because I’ll get a whole new address range. Getting this set up was pleasantly simple, particularly when compared with my past experiences trying to set up an IPv6 tunnel. Upon registering I was allocated an IP range and given a FreeBSD-compatible script to bring the link up. I decided to set things up more permanently using the excellent guide on the FreeBSD Diary website and the details from the broker’s script.

Surprisingly with the relevant tunneling, routing, and advertisments going setting up clients was a doddle. On my FreeBSD desktop machine I turned on ipv6_enable in rc.conf and it sprang to life (after a reboot). Even on our Windows systems it was as simple as running “ipv6 install”.

This finally left PF. Now that I’ve finished setting it up I can happily say it seems much nicer than IPFW, but I won’t pretend the journey was easy. It took a while to get my head around the differences, the main one being last-match versus first-match rules. I still need to figure out some of the ALTQ stuff though; my last attempt left me throttling internal traffic to 0.5Mb/s 🙂


The wonders of computer generated art

I’ve always been fascinated by computer generated art, in particular the landscape work. Over the past few years they seem to be getting so good that it’s often hard to tell the difference between them and real scenery photographs.

The main site I visit is digitalblasphemy.com, and I pay the small subscription fee to get access to the latest images. Today I found the user gallery – not sure how I missed it in the past! I was quite shocked to see such high quality images. I don’t know what I was expecting, but it wasn’t what I found!

One artist in the gallery stood out to me, and he’s Juergen Eilts over at shiftedreality.com. He also maintains a blog, but doesn’t seem to keep it updated on a regular basis. It’d be nice to have a feed of his work.

It’s great to have guys like these generating such good work; there’s no hope of me ever creating anything like it. Without them I’d have nothing to put on my desktop!


Neat tool: bwm-ng

I recently found a neat little tool whilst looking for applications that link against libstatgrab. It’s called bwm-ng and is written by a guy called Volker Gropp. The tool itself isn’t anything revolutionary (it’s influenced by the original bwm tool), it’s just a handy way of displaying current bandwidth usage across multiple interfaces.

bwm-ng screenshot

This screenshot shows bwm-ng in action on my FreeBSD router.

It has a bunch of input methods to make it more portable, including libstatgrab which in theory might make it work on Windows. The default output method is the curses interface as shown in the above screenshot, but it’ll also do various textual formats including HTML.

The bwm-ng website gives links to a whole bunch of pre-packaged builds for various Linux distributions, and I’ve recently added it to the FreeBSD ports collection. Building from source is trivial too. 

I’m always on the lookout for handy little tools like this that just give you the raw facts in a simple and easily digested format. And it’s even better when they make use of libstatgrab 🙂


Impending doom (for our filesystems, anyway)

Over the past year or so the space usage on our research and web filesystems has pretty much doubled to the point where we’re dangerously close to running out of space. There’s currently about 1TiB of filestore available of which less than 10% remains unused.

Teaching filestore, however, has barely grown at all during the last year. I attribute this primarily to quota control, but also to the regular turnover of undergraduate students.

Fortunately we saw this problem arising quite a while ago, so we’ve had time to purchase new storage and infrastructure that should alleviate this problem and make it easier for us to expand the storage availability in the future.

Our new system consists of a pair of Sun StorEDGE 3511 arrays attached by fibre channel to our existing Veritas cluster. We’ll use VxFS for the filesystems, which could lead to some interesting new technologies like filesystem checkpointing; we could have a mount point of /yesterday to allow users to retrieve their files as they were at some point during the previous day, thereby reducing the need for us to do tape restores. VxFS also works quite happily with large filesystems, unlike Solaris UFS. The only problem we’ve found is that VxFS doesn’t support hard linking directories, but that’s not something we commonly, if ever, want to do. We also initially had problems integrating VxFS with the Solaris quota system over NFS, but we soon fixed that the “fun” way 🙂

Currently the research and teaching servers have locally attached filestore, which means if we have a hardware failure in one of the main servers we’re unable to get at user filestore from any other systems (without moving cables). The new solution provides NFS mounts of the filestore directly to each of the servers, which will allow files to be accessed via secondary machines should one of the main servers die. This is all part of our long term plan to increase the resilience of our systems.

One other interesting point to note is the use of the Solaris automounter to individually mount user home directories. Soon there’ll be mounts a bit like this all over the place:

resfs.cs:/home/cur/tdb 1.5T 54G 1.4T 4% /home/cur/tdb

Which will make things much more interesting!


How not to set up a blog

At some point towards the end of last week I had the idea of writing a blog. I spent a while looking around for a decent online blogging tool, and, as I suspect most people do, I landed on blogger.com. It looked fairly swish, and had plenty of useful features. Except one. Where is the support for categories? So, moving on, and with the advice of some friends, I ended up at wordpress.org. It ticked all the boxes and is available in the FreeBSD ports collection.

This is where things went downhill. Step one, I figured, would be to get the existing packages on my server updated, since I’d quite like to use PHP5 for running WordPress, and I noticed it also needed MySQL. This didn’t take too long, despite there being a new version of perl to install. The problems came when I tried to build MySQL; it wouldn’t even build. After a while I got it to build and install, only to discover PHP behaving oddly as well; it would hang when WordPress used it’s mail() function.

I asked a few fellow FreeBSD developers, but didn’t get much sympathy since I was at that point running FreeBSD 5.2.1, a long since outdated version.

So on Sunday I started the task of upgrading to RELENG_5 (see my other post for more). The build process was fairly straightforward, thankfully. Then I noticed the serial line to my server was down, so I figured it would best to wait until Monday to do the actual install.

On Monday the install went fine, gvinum started without any problems, and the machine booted up with no unusual errors. There were a few tense moments during the actual reboots, but a lot of relief afterwards. I was surprised to find my PHP problems, and a long standing issue with mutt, had gone away before I’d even started the rebuilds.

The rest of Monday, and half of Tuesday was spent rebuilding all the packages on the system. By the end of Tuesday I was ready to start setting up the blog.

So, I make that the best part of 5 days to set up a blog. Surely that’s got to be a record?

Now to think about which theme to use…