A Blog by Tim Bishop

This weekend I thought I’d have a go at setting up SlimServer, an application that streams music to various audio devices. It’s primarily design to work with the SqueezeBox, a hardware device that can wirelessly stream the music, but there are various bits of software that can use it too.

The server itself was a doddle to set up. There is a FreeBSD port of it that does all the work for you. Once installed you just run it then browse to port 9000 on the server to access it. It didn’t take long to index my MP3 collection, and then it was ready to go.

I started out by using Winamp to stream the music. That worked absolutely fine, but I wanted something I could run under FreeBSD. The idea was that I’d find an old piece of hardware that I could run headlessly in the lounge hooked up to our speaker system.

Various applications existed to do the job. The obvious choice is SoftSqueeze, a virtual SqueezeBox application provided in the SlimServer distribution. It’s java based, which makes it mildly more effort to get going on FreeBSD, but works pretty well. It has a headless mode too, which is ideal for what I want.

Next up there’s slimp3slave, which is a small C application that does the same job as SoftSqueeze. It uses an external application such as mpg123 or madplay to actually play the audio, so it’s a fairly small app. Whilst it doesn’t appear to have any problems, I didn’t have much success with the players. mpg123 got confused by the stream, and madplay kept skipping the beginnings of tracks when I hit next on the server. This could be a problem with slimp3slave - I’ll need to investigate.

Unfortunately SoftSqueeze isn’t faultless either. Whilst it plays fine, if you leave it idle for a long period of time something goes wrong and it refuses to play. I need to debug this further - it’s likely a FreeBSD related issue, since I know it works for other people on other platforms.

To finish the installation off I installed the MusicIP listener tool which can generate playlists based on any track you give it. When first started it does a scan of all your collection, which takes forever, and builds a database of information about tracks. It then uses this to match similar tracks together. It’s working surprisingly well so far.

The only problem with the MusicIP tool is that it’s a linux binary. This meant activating linux emulation on the server and installing the base linux port. To use the client application (not actually needed, though, since you can do everything through the server) you need java too - a linux one. I only had success with the blackdown 1.4 version.

This lot is controlled via a web browser. This is fine if you’re sitting at a PC and streaming to an application on your machine. But what about the headless machine? Fortunately I have an iPAQ with wireless, and it does the job of a remote control perfectly.

Longer term, if I can’t solve the problems with SoftSqueeze or slimp3slave I’ll consider buying a SqueezeBox. They’re expensive though; £170 for a wired version and £210 for a wireless one. Until I can justify the expense (ie. I’d acually use it) I won’t be forking out for one, though.

Looks like summer is finally here, but for me that isn’t a good thing.

The main reason I dislike the summer is hayfever. I get it quite badly which means I spend most of the summer bunged up, itchy, and miserable. Add to that the heat and dryness we get in Kent and I’m not a happy chappy.

Thankfully I have our portable aircon unit running at the moment which has reduced the temperature of this room to a relatively cool 24 degrees; the rest of the house is much warmer.

Roll on September…

This week we’ve finally seen the end of some things I’ve been trying to sort out for some time now.

• The old storage arrays (Sun T3s and A1000s) are finally gone. The T3 arrays in particular have caused us endless grief over the past few years, so I’m more than happy to see them go. It also marks the end of a year long project to centralise our filestore on our resilient cluster. No more losing access to our files when one machine goes down
• Our last Solaris 8 machine has been decommissioned. We’d stopped supporting it a while ago, but this finally puts the nail in the coffin. More importantly it means I can focus on moving towards Solaris 10, which I hadn’t done until now because I didn’t want to be running 3 different versions of Solaris at once!
• We’ve finally removed the last non-rackmountable machine from the racks. Actually, it wasn’t even in our racks, so it means we’re now entirely self-contained within our own area. This is something I’ve been trying to do for many years.

So I’m now spending some time looking at Solaris 10 and trying to see how we can integrate the new “features” in to our existing systems. The main problem areas seem to be the service management stuff. I’ll undoubtedly post more about that in the future.

I’ve never got on well with software RAID systems on FreeBSD. I’ve tried gvinum (previously I used vinum), gmirror, and ataraid, all with varying degrees of success. The latest machine I built is using gmirror, and so far I’m happy.

However, over the past few days I’ve been having problems with a system I built a couple of years ago. It originally used vinum on FreeBSD 5.2.1, but I recently upgraded it to 5.5 and switched to gvinum. A week or so ago I noticed that the second disk in the mirror was marked stale - I guessed it was an artifact of the upgrade to 5.5. So on Tuesday I decided to resync it.

It went fine to start with, until syncing one partition produced a disk read error. This marked the whole original disk as bad, and I’d only half synced to the second disk. Thinking back I knew this disk had an error on, and I’d fully intended to replace it. Shame I didn’t do it at the time. Next I rebooted the machine to recover the disk from dead to stale, so I could force it back online. This is where the problems started.

GEOM_VINUM: subdisk swap.p1.s0 state change: down -> stale
GEOM_VINUM: subdisk root.p1.s0 state change: down -> stale
GEOM_VINUM: subdisk var.p1.s0 state change: down -> stale
GEOM_VINUM: subdisk usr.p1.s0 state change: down -> stale

That’s what welcomed me during bootup. Not too bad I hear you say? Well, that’s all I saw after that - it didn’t boot any further. I tried various things such as unloading the geom_vinum module, booting single user, booting the other disk, pulling one disk, but nothing worked.

In desperation I booted an older kernel. It worked! Well, when I say worked, I mean it booted past this point and asked me for a root partition - but at least I could work with that. It wasn’t immediately obvious why it had worked; my theory is that it wasn’t the fact it was an older kernel, but that it was a different kernel version to the modules on the disk, making it refuse to load the geom_vinum module.

So after getting things running again I decided to update to 6.1. I figured help would be more limited when running 5.5, and I could see changes had gone in to gvinum in 6.1. After a few hours this was done, but the result was the same; I booted to single user, typed “gvinum start”, and got the same message. Oddly this time the machine wasn’t entirely dead - I could still reboot it. But maybe this was because I’d launched it manually.

Regardless of the cause of the problem I’m now stuck. I’ve got everything running off one disk fine, but I can’t get the RAID going. The only possibility I can see is redoing the RAID configuration, but to do this I’ll need to blast the existing config off the disks, and I’m nervous about that.

The other option I’m considering is replacing the machine and starting again (it’s getting old now anyway). Maybe this time I’ll go for a hardware RAID solution, though

I’ve known about Google Bookmarks for a while, but until recently couldn’t really see how they’d be useful to me. A single set of bookmarks on the web is great, but if you have to go to a webpage to find them it rapidly becomes too much effort. Compare this to a single click within your browser’s menu.

My Internet Explorer using friends have pointed out that it’s now integrated in to the Google Toolbar, but at the moment it’s only the IE version that has it. The new version for Firefox has many of the newer features in the IE version, but unfortunately not the bookmark functionality.

After giving up on waiting for Google to integrate this functionality in to their toolbar I decided to have a look for a Firefox extension to do it instead. Quite why I hadn’t done this before I don’t know. On the first page of search results I found this one.

It’s working pretty well so far. Sometimes it seems to be a bit fussy about the title of pages, and it’s not the best at automatic updating, but it does do the job pretty well. I’ve dropped the extension on to my menubar too, so it fits in quite nicely. It also works quite happily under all my different Firefox installations (Windows, FreeBSD, and Solaris).

This handily means I don’t need the Google Toolbar. Firefox already has a perfectly good search box, and this extension gives me the bookmark features. Sorted!

I’ve been playing a game called Entropia Universe (previously Project Entropia) recently, and I’m finding it quite addictive. It’s much like other MMORPGs at a first glance, but it boasts a “Real Cash Economy”. This means that there’s a direct exchange rate (fixed to the US dollar) and you can exchange money to and from the game currency just like you can with any other currency in the real world.

Within the game itself you learn skills, take on professions, and if you’re lucky you can even make a real profit. My experience so far, though, suggests that making real money is highly unlikely; but if you get good enough you might just about be able to fund yourself within the game - so no loss of real money either.

It’s worth giving the game a go. It’s a free (but large - approx 1Gb) download and it’s free to get an account. You can also get started in the game at no expense by “sweating” animals, although it gets tedious after a while.

I’m still fairly new, so I’ve only deposited a bit of real money to help progress me along a bit. I’ve just bought my first gun, and I’m now running around killing the animals instead of just sweating them. I have a reasonable amount of confidence in my ability not to get carried away and soak too much money in to this game

There’s a wealth of information on the game website, on wikipedia, and on the forums. There’s not much point in me repeating it all, so go take a look for yourselves. Maybe I’ll see you in the game?

We’ve recently moved to having our filestore NFS exported from a cluster. This provides almost complete resilience from hardware failures, and moves us away from depending on individual end-user systems with locally attached filestore.

Given the inherent insecurities with NFS we opted to use IPsec authentication (but not encryption) between the hosts involved. The NFS server only accepts connections from a list of hosts, and we know those hosts are who they say they are by relying on the IPsec authentication. We’ve also made it use privileged ports to ensure local users don’t try any spoofing

The trade-off here appears to be latency. I’ve done some completely unscientific tests that involved shovelling UDP data at a fixed rate between two machines. These are the ”jitter” figures they produced:

• 0.10ms - direct
• 0.30ms - via router
• 0.70ms - via router with IPsec

Bear in mind that those figures might not bear any relation to the latencies involved with NFS packets, but it should give an idea of the relative delays added by routing and IPsec.

We could, to some extent, reduce those figures by replacing hardware. Quicker routers would undoubtedly remove some of the routing latency, and quicker machines could perform the IPsec calculations faster. But this probably isn’t the cheapest solution.

The first test I want to try is adding a private network between the NFS server and NFS client, with no routing involved. Seeing as it’s private we can reasonably trust that people won’t be able to spoof packets on that network and remove the IPsec authentication. In theory, these differences could signficantly reduce the latencies involved.

We’ll continue to monitor this for a while first, though. We need to keep an eye on loading on the NFS server, network usage, and so on. But, at the moment, it seems likely the problems are in the network part of NFS communication process.

I’d finally finished migrating everything off the old myrtle disk arrays, so I was feeling quite pleased. I’d just unplugged the last array from myrtle and plugged it in to the test machine for wiping. Then I tried to log in to the machine room SunRay, but strangely it didn’t work.

I checked the console logs for myrtle and was surprised to see it counting “12%… 13%… 14%”. I glanced up and saw my colleagues attempting to come in to the machine room and tell me something, but for some reason were unable to open the door. Scrolling back over the console logs I saw what it was up to:

panic[cpu2]/thread=2a100105d40: md: Panic due to lack of DiskSuite state database replicas. Fewer than 50% of the total were available, so panic to ensure data integrity.

That made immediate sense to me, and I gave myself a bit of a kick. The RAID system we use for internal disks, DiskSuite (actually Volume Manager now, but it seems they haven’t updated this error message), has state databases stored on every disk. On myrtle we had 6 - two on the internal disks, and one on each of the four disk arrays. You need at least 50% for things to work.

A week or so ago I removed the first pair of arrays without any problems. At that point we had 4 out of 6 databases. Today I removed the last 2 giving us only 2 remaining, which is less than 50%, and the machine dutifully paniced itself.

Fixing it was made tricky by the fact that it could no longer mount the root filesystem because the RAID wouldn’t start. Thankfully the arrays were still to hand, so I just plugged them back in. After booting I removed the databases from the arrays, and added an additional one on each of the internal disks - this gives us 4 in total, 2 on each disk, which is what we normally do.

I also used the handy opportunity to mount the new filestore directly on /home and /proj, rather than using symlinks.

I’ll end this post with a bit of a rant. I can understand why the system won’t boot with less than 50% of the state databases - it has no way of knowing if they represent the correct state of things. But, what I don’t understand is why it needs to panic the system when it has less than 50%. It knows the remaining ones are valid because they’re currently in use. In fact panicing just makes it harder for the sysadmin to deal with the problem. Or am I missing something?

The Computer Science department got it’s new website (more of a re-skin, actually) yesterday, so I decided it was about time to update my staff page.

I’ve brought it in-line with the new look of the main website, although it doesn’t completely follow the standard templates. It’s also XHTML 1.1 valid, which makes the pedantic side of me feel much better.

There’s not much else to say - go take a look (unless you’re already there), and let me know what you think.

A few days ago one of our users reported that they couldn’t change their password. The error coming out of the passwd command was confusing in itself - it said ‘bad old password’, or similar, which turns out to be a bug in our wrapper script.

After some investigation we discovered that neither kadmin or kpasswd worked:

tdb [~] % kadmin -p tdb/admin
Enter Password:
kadmin: Operation failed for unspecified reason while
initializing kadmin interface

tdb [~] % kpasswd
kpasswd: Changing password for tdb.
Old password:
kpasswd: Cannot establish a session with the Kerberos
administrative server for realm CS.UKC.AC.UK.
Operation failed for unspecified reason.

The completely unhelpful bit there is the “failed for unspecified reason” error message. How are you meant to even begin debugging that? After a couple of hours digging I logged the call with Sun.

It turns out that there is a known bug:

Document ID:6410919
Title:Patch 112908-24 will cause the kadmin -p kws/admin to exit with a error message

The solution presented was to remove patch 112908-24. This time I’m willing to do that, but from past experience I’d like to see them actually fix the problem rather than just back it out. Or, at the very least, remove the patch from cluster patches. Otherwise in 6 months time I’m left staring at the same problem.

What I’ve found most interesting in all this is that it took the best part of a month for anyone to notice passwords couldn’t be changed