Today was a day we’ll remember for a while. Ruth & I got up early, but separately, and anxiously got ready. We met up with some family and friends in church, then headed out for a meal. The meal was concluded with some amusing anecdotes and some drinks. We enjoyed music in to the evening, danced a bit, ate some more food and drank plenty. Then exhausted we retired at the end of a memorable day.
Confused? This might clear things up…
Not that I’m counting, but it looks like I have 39 years, 1 month and 11 days to go until I retire.
Work out how long you have left using this handy calculator:
There are three credit referencing agencies used in the UK. They are Experian, Equifax and Callcredit. You have a statutory right to request a copy of the information they hold on you, for which you are charged £2. All of them offer full reports for around the £15 mark, but these just contain analysis and views on how lenders will interpret the information. There are times when a full report could be useful, but not for me at the moment.
I recently requested the statutory reports from all three agencies and they all showed up within a few days. Although they all contained similar information the Experian report showed that it was the agency lenders were primarily using to do checks with. It was interesting to see just how much information is contained in the report and how often you’ve had checks run against yourself. Did you know that some insurance companies do it when you get a quote?
So why would you want to do this? The main reason is to keep tabs on the information held about you and to check for anything suspicious. If someone has stolen your identity you might find things cropping up on your report that have nothing to do with you. They also give you the opportunity to find and correct mistakes, which is better than trying to deal with problems when a lender does a check on you.
For £6 in total it’s not worth worrying about the money, so I plan to get reports on a yearly or bi-yearly basis.
Hopefully this provides you with a little insight in to this area - I knew little about it until I went through this process.
Thursday 22 February. That’s the day it all went wrong.
I was on my way home from a shopping trip at Sainsburys. We’d been on a Thursday instead of a Friday because we had to go to Cornwall on the Friday afternoon for a funeral (that’s bad thing number one). Just after leaving Sainsburys we came across an area that was unusually dark. I guessed this was a power cut, so I was beginning to worry about the state of things at work. Then we hit an area that was lit up, then another that was dark, then another that was lit up. Unfortunately we got to University to find it in darkness. That’s bad thing number two.
I sent Ruth home with the shopping and headed in to see what I could do. Our core equipment has a couple of hours UPS provision and the non-core stuff has virtually no UPS provision, so there was no immediate rush - the machines were either fine or already off. This was when the third bad thing happen - the card locks had failed. Although I could get in to the Octagon lobby I was unable to get anywhere else in the building. After some wandering around the caretaker managed to find a way to the machine room bypassing the card locks, but this still left the challenge of entering the room itself.
I waited for someone else to show up, hopefully with a key for the machine room, but nobody did. In the end I gave up and went home to try and use the computer there to reach people. Success! Someone from operations was coming in with a key. I grabbed a quick bit of food and rushed out of the door to meet them. Suddenly I felt a seering pain in my finger. I glanced at it and wondered why it was purple and throbbing with pain. I’d stupidly managed to catch it in my large metal front door. There’s problem number four. After a few minutes of putting ice on it I headed back up to work.
The next part of the story is a bit dull. I shut everything down and went home to sleep. I’d figured I was better off coming in early the next day rather than staying late, particularly with the state of my finger. 7am next morning I headed back.
It took a good few hours to get everything going again. There were plenty of problems with the cluster which all turned out to be related to the last bad thing. The main disk array (a Sun StorEdge 3510) is a fully redundant unit - it has dual controllers, dual PSUs, dual fibre links, and a RAID 5 disk configuration. One of the power supplies is linked to our UPS and the other to the main machine room UPS. In a power failure the machine room UPS lasts only a few minutes so we drop down to only one PSU. So it would help if that PSU was functioning. It all appeared fine - lights were on, fans whiring - but the unit just wouldn’t power on with just that single PSU. Consequently when the power cut occured that disk array went down.
That problem was compounded by the way we’d set up our coordinator disks. We originally only had this one array so we had three coordinator disks on the one array. Later when we added two more arrays we added a coordinator disk on each of those but fataly left three on the original array. When this array turned off we were left with less than a majority of coordinator disks which caused all the cluster nodes to panic - they assumed they were cut off from the main part of the cluster. So even the services that should have remained running were killed off.
The disk array powering off also caused corruption on one of our filesystems. This was easily fixed with a fsck (thank goodness we had VxFS, otherwise all our ACLs would have vanished - last time I checked that was still a problem with UFS). It had also corrupted our MySQL databases, but after some rather long checks these were also fixed. One of the MySQL databases handled our email services, so we had a whole bunch of problems there too.
We’ve now had the PSU replaced by Sun, with unusually little fuss, so maybe it’s a known problem. But we can’t test it properly without potentially killing the array again, so we’re holding off doing it until we next power everything down. I’ve swapped the power cables over so another power failure shouldn’t land us in such a mess.
Isn’t it great when your highly available systems work? 
Back in the middle of last year I wrote about our plans to tackle our NFS performance issues by introducing a direct and dedicated network link to carry our NFS traffic between the clients and the servers. We’d done the tests so we just had to implement it.
First we waited for the financial year (1 August) to roll over so we had some money, then we went and purchased 2 gigabit switches and 4 quad port network cards for the client machines. We only really needed dual port cards, but the supplier gave a choice of single or quad. The server nodes, which are part of our cluster, already had sufficient ports (8 per node, 6 of which are now used!).
You’d think it’d be pretty simple from there? It wasn’t. We hit a couple of snags:
After much deliberation other jobs came along and consumed my time for the next few months.
So, a couple of months ago we took the plunge and made an important decision. We got blue cables. This would avoid confusion with our yellow cables (normal network), purple cables (crossover network), red cables (serial network), green cables (serial rolled) or grey cables (to be burnt alive). We also noticed some handy holes in the tops of the cabinets and neatly threaded the link cables to the servers through there.
With the hard work done we set about putting the new cards in to the client machines. The first machine was a good test case and I spent a while sorting our configuration and automounter setup to deal with the new link nicely. The remaining machines I’ve done on Tuesday mornings this month, with the last one being done today.
This leaves us with the important question. Is it actually quicker? My raw tests prove it is - ping times are halved and times to transfer large files are also halved. Loading my email in mutt is much quicker, as is listing my overly populated home directory.
Sorted then? Not quite - we still have a few users with Exmh slowdown problems. We’re still investigating that. Maybe Exmh is doing something pathalogically slow that doesn’t agree with NFS. Or maybe it’s just getting a bit slow in its twilight years. I’ll leave that to the boss (the only Exmh user in our group) to figure out
So, I don’t post for the best part of half a year and then all I can be bothered to do is link to a colleague’s Celebrity Gossip page. It’s a favour, OK? 
Oh, and he wanted to make sure that everyone was aware that this content is just for him and that he really enjoys reading it. I think.
So my current hosted server is getting a bit old. It’s not got enough RAM, and the disk in it is failing (yes, I did have RAID, more on that later). So it’s about time to get a replacement in.
The guys over at Netrino have just installed a new machine for me. I say new, but it’s not a brand spanking new bleeding edge state of the art all singing all dancing machine costing a million pounds. It’s just an Intel Celeron 2Ghz, with 1GB of RAM, and two 80GB hard disks. The main thing is the increased RAM, and two new (and hopefully working) disks.
Things didn’t get off to a good start on day 1 - they didn’t have a FreeBSD CD to hand. They “kindly” left me a USB dongle containing a variety of Linux installers, so I had a play with them in the hope that I could somehow bootstrap a FreeBSD install from one. After some googling I found the Depenguinator that claimed to do exactly what I needed. A few hours later I discovered it didn’t - probably because its not been updated for more recent FreeBSD versions.
On day 2 things got off to a better start - a FreeBSD 6.1 CD arrived in the CD-ROM drive and booted nicely. I had a quick play around to check everything - particularly the network card - worked, and thankfully it all did. Next came the installation.
One area I’ve had quite a few problems with on FreeBSD is the software RAID provision. You’ll see in one of my previous posts that I had some fatal problems with gvinum, and since then I’ve had other problems recovering a RAID 5 failure using it. Another alternative is ataraid, which worked fine for me up until FreeBSD 6. Since then I’ve not been able to get it to resync a failed disk properly - it hangs at 0% forever. So those two solutions are written off.
Other than hardware RAID this leaves me with a clear choice: gmirror. I’ve been using gmirror on another machine for some time now, and I’m pleased with the results. Following this guide it’s easy to apply it after installation, which is a definate selling point - I can’t stand solutions that require dumping and restoring. On my other machine its also had no problems resyncing, so another box ticked (or not ticked on the reasons-to-avoid list, actually).
I’m left wondering here how hard it would be to add support for gmirror (and maybe some of the other geom providers) to the FreeBSD sysinstall program. I experimented with setting up the mirror by hand before running the installer, but it failed to notice it. If this functionality could be added it’d be a real selling point for FreeBSD.
So, with the install complete, and the disks mirrored, I’m ready to move on to building and configuring. First up is updating the world and kernel, then installing all the software. I’ve not really figured out how I’ll copy everything across though…
Back in May I wrote about the performance problems we were having with our new NFS based user filestore. It’s been a while since then, and the problems have continued. We have noticed that it appears to be load related - not just the network, but also the machine. This suggests that our theories about IPsec causing the slow down may be correct.
Our original plan was to try a private network which would remove the need for IPsec and also remove any latency added by routing the traffic between our subnets. This still seemed like a good plan, so I asked around and another department kindly lent us a brand new gigabit switch. We’ve connected this to one of our NFS clients and to the cluster node that’s currently running our filestore.
So far we’ve noticed some serious performance boosts. There’s only a few of us using it, so it could just be that it’s a lightly loaded connection - time will tell on that one. The bottom line is that it seems to be quicker than the IPsec connection ever was, so hopefully we’re on to a winner. We’ve also got a few staff testing it out, and their responses have been positive so far.
The next step after this testing period is to look at the costs of doing this properly with our own equipment. One of the key things we’ve been doing recently is increasing the redundancy of our systems, so it’d be fairly daft to do this with just one switch. We’d need at least two, with every cluster node connected to both, and every client that we want optimum performance on connected to both. Obviously there’ll be other clients that are less important and they can continue to use the existing infrastructure.
Of course, I’ve got absolutely no idea where we’ll put these switches, or how we’ll wire them in - things are pretty tight in our racks at the moment. Suppose there’s got to be a challenge somewhere
My only worry with all this is what we’ll do if it doesn’t work. I don’t have any other ideas that’d make it go quicker - to be frank, you can’t really get any quicker than a directly connected switch. Lets hope we don’t have to worry about it.
In my last post about setting up a slimserver I said that I was having trouble getting slimp3slave working:
Whilst it doesn’t appear to have any problems, I didn’t have much success with the players. mpg123 got confused by the stream, and madplay kept skipping the beginnings of tracks when I hit next on the server. This could be a problem with slimp3slave - I’ll need to investigate.
The problem did turn out to be with slimp3slave. I discovered that when skipping a track the stream is restarted which caused slimp3slave to start up a new player. The problem was this is that it did it before the old one had exited, thus causing the new one to die because it couldn’t access the sound device. There’s another bug here - it didn’t notice the new player dying and tried to write to it, which resulted in lots of SIGPIPE messages.
So I looked at the code for shutting down the player and noticed that it wasn’t using the right close function. This change fixed it:
RCS file: /home/pdw/vcvs/repos/slimp3slave/slimp3slave.c,v
retrieving revision 1.10
diff -u -r1.10 slimp3slave.c
— slimp3slave.c 12 Apr 2004 08:04:52 -0000 1.10
+++ slimp3slave.c 22 Jun 2006 21:21:31 -0000
@@ -394,7 +394,7 @@
}void output_pipe_close(FILE * f) {
- fclose(f);
+ pclose(f);
}unsigned long curses2ir(int key) {
I have sent this change to the author, so maybe it’ll get integrated.
Now I have a working streaming system. The only remaining problem seems to be the wireless networking to the client dropping out from time to time - a wire would fix that one
And in the past couple of days I’ve even got a client (softsqueeze on Windows this time) running at work that’s streaming the music over my ADSL connection. Very handy!
This weekend I thought I’d have a go at setting up SlimServer, an application that streams music to various audio devices. It’s primarily design to work with the SqueezeBox, a hardware device that can wirelessly stream the music, but there are various bits of software that can use it too.
The server itself was a doddle to set up. There is a FreeBSD port of it that does all the work for you. Once installed you just run it then browse to port 9000 on the server to access it. It didn’t take long to index my MP3 collection, and then it was ready to go.
I started out by using Winamp to stream the music. That worked absolutely fine, but I wanted something I could run under FreeBSD. The idea was that I’d find an old piece of hardware that I could run headlessly in the lounge hooked up to our speaker system.
Various applications existed to do the job. The obvious choice is SoftSqueeze, a virtual SqueezeBox application provided in the SlimServer distribution. It’s java based, which makes it mildly more effort to get going on FreeBSD, but works pretty well. It has a headless mode too, which is ideal for what I want.
Next up there’s slimp3slave, which is a small C application that does the same job as SoftSqueeze. It uses an external application such as mpg123 or madplay to actually play the audio, so it’s a fairly small app. Whilst it doesn’t appear to have any problems, I didn’t have much success with the players. mpg123 got confused by the stream, and madplay kept skipping the beginnings of tracks when I hit next on the server. This could be a problem with slimp3slave - I’ll need to investigate.
Unfortunately SoftSqueeze isn’t faultless either. Whilst it plays fine, if you leave it idle for a long period of time something goes wrong and it refuses to play. I need to debug this further - it’s likely a FreeBSD related issue, since I know it works for other people on other platforms.
To finish the installation off I installed the MusicIP listener tool which can generate playlists based on any track you give it. When first started it does a scan of all your collection, which takes forever, and builds a database of information about tracks. It then uses this to match similar tracks together. It’s working surprisingly well so far.
The only problem with the MusicIP tool is that it’s a linux binary. This meant activating linux emulation on the server and installing the base linux port. To use the client application (not actually needed, though, since you can do everything through the server) you need java too - a linux one. I only had success with the blackdown 1.4 version.
This lot is controlled via a web browser. This is fine if you’re sitting at a PC and streaming to an application on your machine. But what about the headless machine? Fortunately I have an iPAQ with wireless, and it does the job of a remote control perfectly.
Longer term, if I can’t solve the problems with SoftSqueeze or slimp3slave I’ll consider buying a SqueezeBox. They’re expensive though; £170 for a wired version and £210 for a wireless one. Until I can justify the expense (ie. I’d acually use it) I won’t be forking out for one, though.