A Blog by Tim Bishop

I’ve known about Google Bookmarks for a while, but until recently couldn’t really see how they’d be useful to me. A single set of bookmarks on the web is great, but if you have to go to a webpage to find them it rapidly becomes too much effort. Compare this to a single click within your browser’s menu.

My Internet Explorer using friends have pointed out that it’s now integrated in to the Google Toolbar, but at the moment it’s only the IE version that has it. The new version for Firefox has many of the newer features in the IE version, but unfortunately not the bookmark functionality.

After giving up on waiting for Google to integrate this functionality in to their toolbar I decided to have a look for a Firefox extension to do it instead. Quite why I hadn’t done this before I don’t know. On the first page of search results I found this one.

It’s working pretty well so far. Sometimes it seems to be a bit fussy about the title of pages, and it’s not the best at automatic updating, but it does do the job pretty well. I’ve dropped the extension on to my menubar too, so it fits in quite nicely. It also works quite happily under all my different Firefox installations (Windows, FreeBSD, and Solaris).

This handily means I don’t need the Google Toolbar. Firefox already has a perfectly good search box, and this extension gives me the bookmark features. Sorted!

I’ve been playing a game called Entropia Universe (previously Project Entropia) recently, and I’m finding it quite addictive. It’s much like other MMORPGs at a first glance, but it boasts a “Real Cash Economy”. This means that there’s a direct exchange rate (fixed to the US dollar) and you can exchange money to and from the game currency just like you can with any other currency in the real world.

Within the game itself you learn skills, take on professions, and if you’re lucky you can even make a real profit. My experience so far, though, suggests that making real money is highly unlikely; but if you get good enough you might just about be able to fund yourself within the game - so no loss of real money either.

It’s worth giving the game a go. It’s a free (but large - approx 1Gb) download and it’s free to get an account. You can also get started in the game at no expense by “sweating” animals, although it gets tedious after a while.

I’m still fairly new, so I’ve only deposited a bit of real money to help progress me along a bit. I’ve just bought my first gun, and I’m now running around killing the animals instead of just sweating them. I have a reasonable amount of confidence in my ability not to get carried away and soak too much money in to this game

There’s a wealth of information on the game website, on wikipedia, and on the forums. There’s not much point in me repeating it all, so go take a look for yourselves. Maybe I’ll see you in the game?

We’ve recently moved to having our filestore NFS exported from a cluster. This provides almost complete resilience from hardware failures, and moves us away from depending on individual end-user systems with locally attached filestore.

Given the inherent insecurities with NFS we opted to use IPsec authentication (but not encryption) between the hosts involved. The NFS server only accepts connections from a list of hosts, and we know those hosts are who they say they are by relying on the IPsec authentication. We’ve also made it use privileged ports to ensure local users don’t try any spoofing

The trade-off here appears to be latency. I’ve done some completely unscientific tests that involved shovelling UDP data at a fixed rate between two machines. These are the ”jitter” figures they produced:

• 0.10ms - direct
• 0.30ms - via router
• 0.70ms - via router with IPsec

Bear in mind that those figures might not bear any relation to the latencies involved with NFS packets, but it should give an idea of the relative delays added by routing and IPsec.

We could, to some extent, reduce those figures by replacing hardware. Quicker routers would undoubtedly remove some of the routing latency, and quicker machines could perform the IPsec calculations faster. But this probably isn’t the cheapest solution.

The first test I want to try is adding a private network between the NFS server and NFS client, with no routing involved. Seeing as it’s private we can reasonably trust that people won’t be able to spoof packets on that network and remove the IPsec authentication. In theory, these differences could signficantly reduce the latencies involved.

We’ll continue to monitor this for a while first, though. We need to keep an eye on loading on the NFS server, network usage, and so on. But, at the moment, it seems likely the problems are in the network part of NFS communication process.

It seems I’ve never got enough time these days.

I’m a FreeBSD ports committer, but recently I’ve hardly done anything. All I’ve managed to do is keep my own ports updated. It’s quite frustrating because I want to be more involved. Then there’s the other projects like libstatgrab, they don’t even get a look in.

I blame the “day job”; if it’s not using my actual time it’s occuping my mind…

If you’ve been following my blog you’ll know that I’ve been working on a new filestore project at work for a while now. After getting things working nicely on our Solaris machines, and finally moving my home directory over, I decided to tackle our Debian server. It quickly became apparent that I’d need to upgrade the machine, which was running Woody with a 2.4 kernel, to get to a decent IPsec and autofs setup.

Now, I’m not a Linux user, let alone a Debian one. So this was a new experience for me. After a quick nose around online, and with a few helpful pointers, I found some useful instructions on how to upgrade. It boils down to a fairly simple process;

1. Make sure the system is running the latest Woody updates.
2. Modify apt sources.list file to change woody to sarge.
3. Run apt-get update.
4. Install/update aptitude.
5. Run aptitude -f --with-recommends dist-upgrade to do the full upgrade.

Then it’s just a case of fixing up any conflicting files and changes, and you’re done. I had to remove our backup software (lgtoclnt) and re-add it though, because it messed with the X packages.

I decided at this point to make sure Sarge worked before looking at the kernel. So I rebooted the system. I waited. And I waited some more. The console showed that it had gone through the BIOS and RAID POST, but nothing else. A brief trip back to the machine room showed a scary looking “LI” message, which I knew meant lilo wasn’t working.

At this point I consulted some friends who explained what I needed to do. A short while later, and with a freshly burnt boot CD, I had the system back up and running. To reinstall lilo I’d booted the CD up to the point where it loaded the aacraid drivers, switched to another terminal, mounted my root parition, chrooted, and run lilo.

By this point I’m starting to grumble about Linux/Debian being stupid. But, I move on. I discover that I’m also going to need to upgrade to 2.6 if I’m going to get IPsec support. After a short while of looking at rebuilding kernels, and boggling at the myriad of build options available, I decide to apt-get install kernel-image-2.6. That can’t be too hard, can it? A few moments later I’m left staring at an Oops message referring to a “kernel NULL point deference” which appears to have come from the install running dd.

Nasty. Anyway, to cut a long story short I tweaked the postinst script to stop it running dd, and that allowed me to get the kernel installed. Surprisingly it worked first time, but I did have to fix the modules list afterwards to silence some error messages.

Now a few hours later, and after discovering the difference between autofs4 and the Solaris automounter, I now have a working system. But I’m left wondering why I’d really want to be using Debian at all.

It looks like the BT Exact IPv6 tunnel broker is finally back up and running after being offline for a week. It seems they had a hardware failure of some kind which knocked out their whole TB operation. I appreciate this is a free service, but it’s still a pain not having it available. That said, I was reluctant to change to another broker since so far, ignoring this incident, their service seems to be pretty good:

• Very simple to set up - no messing around with special applications.
• UK based, so only 6 hops outside of my ADSL provider’s network, and 4 away from my hosted server’s network. In both cases it’s a single hop straight from the provider’s network on to the BT network.
• Supports reverse DNS delegation.
• Simple interface for setting up and modifying tunnels.

Finding an alternative to this would have been hard. Unless anyone has any recommendations?

Obviously this is not an ideal long-term solution; I still have to tunnel over the relevant IPv4 networks to get to the broker. What I really want is native IPv6 straight from my service providers. I guess I expect this sort of service more from my hosting company, and when I last asked they said it’s something they wanted to look at. For the average person, though, this is something that needs to come from the ISP, but that’s probably a long way off.

If there’s one thing that drives my completely insane in the modern world of computing it’s spam. It consumes my time, day after day, and devours the resources of our mail systems. In my own mailbox I get a few hundred spam messages a day, most of which I’ll never even see, let alone read. Thankfully most of these are filtered, but there’s still at least 20+ which I have to manually deal with every morning.

At work the mail systems for the Computer Science department are processing around 20,000 incoming email messages every day. A remakable 61% of these are spam, which is quite an increase from 49% a year ago. We run two mail hubs to process the incoming email which means we’ve effectively had to buy and run one server just for processing the spam email. I don’t even want to start on the amount of time spent dealing with spam messages that make it through to our helpdesk systems.

Ever noticed how spam email comes from rather an ecletic selection of email addresses? Has one of those addresses ever been yours? If there’s one type of email even more annoying that spam it’s bounces generated as a result of spam, sometimes thousands of them. You’ve suddenly become an unwilling victim of spam. Your address abused, and maybe even your name tarnished. What gives spammers the right to do this? At least SPF and similar technologies go some way to preventing this.

And as if spam email wasn’t enough we now see it creeping in to many other Internet based systems. How long until there’s a spam comment on this weblog? Or a stack of spam referrer entries in my apache logs (and consequently my statistics)? Or until I receive the next random message on one of my messenger services?

Whilst I’m ranting, another thing I can’t stand are those pages of junk links that appear when you try and google for something, particularly if it’s a fairly common term. Thankfully google is trying to deal with that, but it’ll be a neverending battle.

It seems in the non-Internet world we can easily regulate junk messages. We used to get a fair amount of sales telephone calls and general junk mail through the front door. Within weeks of registering with the Mail Preference Service and the Telephone Preference Service these have completely stopped. I’m not naive enough to believe this could be done with the Internet, but it helps put things in to perspective.

One of these days I’m going to get sick of the battle and just say “screw ‘em all” and unplug my ADSL modem. After all, people keep telling me I should try reading more books.

We’ve finally done another libstatgrab release. It’s been the best part of 8 months since the last release. Given the length of time you might be mistaken for thinking we’ve made lots of changes, but we haven’t. All this release really includes is some mostly untested Windows support, and handful of bugfixes.

I guess the problem is that we’ve hit a bit of a brick wall. Adding more features is now quite tricky; we’ve done all the common ones that make sense across multiple platforms. Adding more platforms is hard since any new ones would be the slightly more obscure operating systems (otherwise we would have done them already). Add to that our lack of enthusiasm and interest for making any radical changes and you get very little progress.

I suppose it at least works in its current state, so as long as we fix any bugs we find we’re probably keeping people happy.

So recently my router decided it didn’t want to whir its fans anymore and consequently gave up on life. It’s a dual CPU machine and both CPU fans had managed to wedge. After fixing them and getting things running again I heard klunking noises coming from the front of the case; one of the disks in the mirror had failed. I rapidly copied everything off the remaining disk, but didn’t have a spare to hand. Next morning the remaining disk went too. I wasn’t having much luck really, but on the positive side I did have a full backup.

After a day or so of fiddling with hardware I got something that resembled a working machine; I’d gone through a stack of various old disks by this point, most of which were dead. For a while I’d been pondering a fresh install for the machine, so this was the perfect opportunity. I decided to think about what I wanted it to do - this is what I came up with.

1. Obviously needs ADSL connection (via rather old, but working, USB modem)
2. I’d quite like a VPN connection to work for various (but not all) work servers
3. IPv6 routing both internally and out to the world
4. Internal NIC with my private and public address ranges
5. A second internal NIC for my wireless network
6. A better firewall setup (I decided on PF in the end)

Rather predictably I decided to do all this with FreeBSD. Nothing exciting about the install, other than I used gmirror this time. I’m still trying to find the best RAID solution on FreeBSD. So far I think gmirror has impressed me most compared to ataraid and gvinum.

So most of the things I wanted the router to do are things it did before. The new things were the VPN, IPv6 and PF. Those are what I’ll write about.

Setting up the VPN was straightforward. I installed the net/pptpclient port, bunged the sample config and my credentials in /etc/ppp/ppp.conf, and knocked up a quick RC script (let me know if you’d like a copy). I also added specific entries to ppp.conf for the hosts I wanted to route over the VPN, rather than letting it route whole subnets.

Something worth noting about ppp is the -unitN flag. Using this you can make sure ppp always uses the same numbered tun device. For example, my VPN connection has -unit1 ensuring it is always tun1. This makes firewall configuration a bit more manageable.

I’ve also knocked up a slightly better RC script for starting the ADSL connection (compared to the one provided with net/pppoa) that checks the line is up before returning. This allows subsequent startup scripts to be pretty much guaranteed access to the Internet. Again, let me know if you’d like a copy.

The next task was getting the IPv6 connection going. I decided to use the BT IPv6 Tunnel Broker service. In retrospect this might not have been the best choice; it’s been down for the last few days. I’ll let you know how I decide to proceed with that, but I’m reluctant to change because I’ll get a whole new address range. Getting this set up was pleasantly simple, particularly when compared with my past experiences trying to set up an IPv6 tunnel. Upon registering I was allocated an IP range and given a FreeBSD-compatible script to bring the link up. I decided to set things up more permanently using the excellent guide on the FreeBSD Diary website and the details from the broker’s script.

Surprisingly with the relevant tunneling, routing, and advertisments going setting up clients was a doddle. On my FreeBSD desktop machine I turned on ipv6_enable in rc.conf and it sprang to life (after a reboot). Even on our Windows systems it was as simple as running “ipv6 install”.

This finally left PF. Now that I’ve finished setting it up I can happily say it seems much nicer than IPFW, but I won’t pretend the journey was easy. It took a while to get my head around the differences, the main one being last-match versus first-match rules. I still need to figure out some of the ALTQ stuff though; my last attempt left me throttling internal traffic to 0.5Mb/s

I recently found a neat little tool whilst looking for applications that link against libstatgrab. It’s called bwm-ng and is written by a guy called Volker Gropp. The tool itself isn’t anything revolutionary (it’s influenced by the original bwm tool), it’s just a handy way of displaying current bandwidth usage across multiple interfaces.

This screenshot shows bwm-ng in action on my FreeBSD router.

It has a bunch of input methods to make it more portable, including libstatgrab which in theory might make it work on Windows. The default output method is the curses interface as shown in the above screenshot, but it’ll also do various textual formats including HTML.

The bwm-ng website gives links to a whole bunch of pre-packaged builds for various Linux distributions, and I’ve recently added it to the FreeBSD ports collection. Building from source is trivial too. 

I’m always on the lookout for handy little tools like this that just give you the raw facts in a simple and easily digested format. And it’s even better when they make use of libstatgrab