A Blog by Tim Bishop

I’ve made a couple of posts lately about the problems we’ve been having with one of our Sun 3511 disk arrays. Sun got back to me today with what they thought the problem was. Here’s the gory detail (slightly truncated to fit this blog post):

* SES

Ch  Id Chassis Vendor/Product ID        Rev  PLD
-------------------------------------------------
 2  28 092131  SUN StorEdge 3511F A     0430 1000
 3  28 092131  SUN StorEdge 3511F D     0420 1100*

* indicates SES or PLD firmware mismatch.

This appears in a few places, so I’ve only quoted the one Sun spotted. It boils down to the components in the array having inconsistent firmware revisions. This could very well have caused the crash we saw yesterday.

This is something I blame Sun for. Firstly, they shipped out a controller with mismatched firmwares on it. I guess this sort of thing might happen though, but the field engineer should really have spotted the mistake when he was onsite and getting the replacement controller configured.

Today Sun wanted to send out another engineer to get the firmware updated, and when I came back from shifting some stuff around I had a voicemail from dispatch. It’s good to see them being so proactive at fixing the issue, although I wonder if it’s because they realised it was their fault?

However, being as I am a sysadmin, I figured I could save everyone a lot of time and hassle if I did the upgrade myself. Sun gave me the link and a couple of hours later it was all sorted (and I even managed to shift some furniture around the building in the middle of it). We’ve agreed with Sun to wait until next week to ensure things are working as they should, then we’ll close the case.

In the meantime I have our largest volume resyncing, but have some familiar looking problems cropping up with the other. Somehow I fear this saga isn’t quite over yet…

After all the fun the other day I was hoping for some time to work on other stuff this week. By the end of the weekend the array had finishing syncing and I’d remirrored all the volumes back on to it. It was all ticking over nicely, until this morning…

Unrecoverable Controller Error Encountered !
Resetting Controller !

After which the array disappeared. On arrival at work I power cycled the unit and it came back up without any problems (albeit needing resyncing, again). But this isn’t good enough - the unit is still faulty.

So I’ve logged the case with Sun, again. They’re being remarkably slow to respond today, despite me logging it as high priority. But, as you’d expect, they just got it within their 4 hour SLA window.

So it continues…

“Any idea WTF is going on?” is what I read on my phone as I stumbled out of bed this morning. It was from one of my colleagues who, for some reason I can’t understand, seems to like getting in to work at a ridiculous hour in the morning.

Still half asleep I plodded through to my desk and sat down at my computer. I tried to check my email but nothing was responding. Then I saw the message “NFS server resfs.fs.cs not responding”… and woke up rather quickly. This meant either our network was shafted, or more likely, the cluster had blown up again.

I discovered one of the cluster nodes was offline and marked as failed, and the service group that manages our filestore was also marked as failed. That was odd, but it had happened before. I dug a bit further and found a screenful of SCSI errors. This was bad - something must have gone wrong with the storage.

Next I checked the arrays. The first one I checked had numerous errors on it; failed disks, missing disks, and drive not ready messages. I can’t stress enough how important this data is - it holds files, email and shared areas for all the staff and researchers in our department, and I really didn’t want to explain to them that we’d lost it all (well, we do have backups…). I nervously moved on to check the second array - they were mirrored, so as long as one was OK we’d be fine - and I was delighted to find no error messages.

So, now I knew that the likely cause of the problems was an array failing. It turned out later on to be the controller in this array, which was a good thing because Sun managed to send the wrong disks anyway. The next steps were to get the array fixed and to get things back online. I asked my colleague, who was already in the office, to disconnect the fibres from the failed array (to keep it completely out of the loop whilst it was fixed) and get on to Sun to fix it. Whilst he did that I, still at home, not dressed, and without breakfast, got on with getting things back online.

This, in theory, should have been the easy part. We had a mirrored setup so the plan was to just bring the volume back online with only half of the mirror. No problem, I thought. Except when it wouldn’t come online. When the initial problem had occured the cluster software (VCS) had failed to unmount the disk from the node it was on. It had decided that it needed to do this to bring it online on another node (little did it know that it wouldn’t work on any other node either), so as a last resort it asked the machine to panic. This is something akin to asking it to commit suicide. It duely did it’s job, but in the process left the disks in an odd state.

When I tried to mount these disks on one of the other nodes I got errors from the volume manager telling me a split brain had occured (this happens when a live cluster splits in two, but neither half can see the other). I knew that wasn’t the case, so I tried to force the mount. That failed with write errors. After a lot of head scratching I realised it was probably the I/O fencing stopping this node from accessing the disk. Whilst frustrating, it was nice to see the software behaving as it should - in a real split brain situation this is exactly what you want.

A while later I figured out how to clear the SCSI3 reservations on the disks (-o clearreserve option to vxdg import). This was nearly enough. Another issue with the split brain was that the configuration data stored on the disks didn’t quite match (I’m not 100% sure why, but I believe the node that paniced hadn’t managed to consistently update the metadata). After dumping the configuration it was clear that they were identical, bar a revision number, so by using -o selectcp we were able to get the diskgroup imported.

vxdg -fC -o clearreserve -o selectcp=1128804183.107.qetesh \
    import ResFS

Success! The diskgroup was online. From here it was just a case of waiting for fsck to confirm everything looked OK and then unleashing VCS to bring the service group back online.

By this point Sun had sent out an engineer and parts to fix the other array (we get a good service from them, thankfully). That’s currently resyncing its disks, which will take a day or two. Once that’s done we’ll hook it back in to the fibre fabric and bring things back online. It’ll take just as long again to resync the data, but all I have to do is sit and watch

Finally, after hours of investigation I finally found out the cause of all the problems. We’ve just ordered a newer, bigger array. The old ones are just jealous.

(And a quick thanks to Pete for his help in debugging things this morning )

I’ve done a bit of work on my FreeBSD ports lately. Firstly, after building my new server, I got round to upgrading from SlimServer to SqueezeCenter. This also meant sorting out ports for all the plugins I use. This didn’t take too long, and you can find them all over here. So far I’m liking SqueezeCenter, and I’d highly recommend it (and a SqueezeBox, of course).

I also maintain a port for a suite of software called KRoC. KRoC is written and maintained where I work, so apart from making it available to FreeBSD users I also have an interest in supporting the work done by our department. I’ve been waiting some time for a 1.5.x release of KRoC, but I finally got impatient. I automated the production of snapshots from their stable branch, and updated the port to build from that. I also run a FreeBSD 7 machine in their buildbot system to further test KRoC on my favourite operating system

And in other FreeBSD news, I cast my vote in the FreeBSD Core elections. It’s hard to know who to vote for, but I gave their statements a good read and made a decision. Good luck to them all!

A few weeks back I started having problems with my file server at home. This machine is fairly important to us; it holds all our photos, music and other files. For years I’ve been bodging it together with various old parts scavenged from other machines and some new parts when needed. But, once again, it’d started to break. Disks were dropping out of the RAID unexpectedly, and the replacements were refusing to rebuild. Unsure of where the problem was I uttered the fateful words “I’ll build a new server; it’s got to be easier than patching up the old one…”. My colleagues were sceptical, but I ploughed on anyway. Maybe I should have listened to them?

It took the best part of a week to work out what I wanted. There were so many decisions to make: which RAID card, disks, motherboard, CPU, RAM, case, etc. I researched each one as much as I could, but there’s a bottomless pit of information on the Internet. Eventually I settled on a 3ware 9690SA RAID card with 4 Seagate ST31000340NS disks. The other bits were fairly decent to make sure the machine would have a good life, but not excessive.

The reason for choosing a hardware RAID solution over software RAID was simple - reliability. Now, I’m not knocking software RAID in principal (look at ZFS, for example), but the implementations for RAID 3-5 on FreeBSD aren’t great (yes, it has ZFS, but I’m not in the mood for trailblazing this time round). I wanted to stick with FreeBSD so I opted for the well known reliablity that 3ware cards provide. And the 5 year warranty on Seagate disks made them an attractive choice.

The purchasing process wasn’t as simple as it could have been. I ordered from dabs.com, span.com (they specialise in storage stuff) and overclockers.co.uk. I’ve used all three companies before, so I wasn’t too concerned about problems. The bulk of it was ordered from Dabs - it looks like they’re back to being competitive on prices. The problems started almost immediately; Dabs held my order over an issue with my address. It’s happened once before and that put me off Dabs for some time, but we use them all the time at work, so I had hopes they’d be better now. It took a working day to resolve that issue… and then next day I get an email to say my credit card company has declined the order. On the phone to them and through to their security department; seems buying lots of stuff online is unusual… not for me it isn’t. Anyway, that was resolved and then I had more waiting for Dabs to try the transaction again. Eventually I got impatient and tried their online chat thing and the matter was resolved in minutes. Meanwhile the parts ordered from the other two suppliers were sitting on my desk.

Eventually it all arrived and I took it home. Ruth wasn’t overly impressed when I cleared off the dining room table and covered it in computer parts, but I assured her it wasn’t for long. That was a couple of weeks ago - it’s all still there.

I spent a weekend putting things together and testing it all out. I routed every cable neatly and tied them carefully to the case to ensure nothing moved about. Airflow was good and the additional fans in the case were doing a great job of keeping things cool (not sure about their blue LEDs though…). All was looking good and I was enjoying the process.

Then I tried to use the RAID card. The first problems hit when I turned on the motherboard’s RAID, which I’d intended to use to mirror the system disks, whilst the 9690SA was plugged in. I’d gone for a Asus P5E3 and expected both RAID systems to work happily together, but sadly I was wrong. I experienced unusual problems such as the machine hanging on the Intel Matrix Storage (the onboard RAID) screen and disks randomly disappearing from both arrays. In the end I gave up and turned off the onboard RAID; I figured the FreeBSD RAID 1 (gmirror) is pretty solid, so I’d use that.

Thinking I’d got over the worst of the problems I moved on to setting up the 9690SA. Things looked good for a while; the interface was clear and everything was easy to set up. It wasn’t until I started trying to put data on that I noticed problems. Here’s a snippet from the error log (largely for the benefit of Google):

E=0200 T=08:26:00 : Cable CRC error
SATA Device. port = 0x0
task file written out : cd dh ch cl sn sc ft
                      : 00 70 00 00 00 1200 00
  task file read back : st dh ch cl sn sc er
                      : 00 00 00 00 00 8441 00
E=0200 T=08:26:00 P=0h: Soft reset drive
E=0200 T=08:26:00 P=0h: exitCode = 1013
Port retry not allowed
E=0200 T=08:26:00 P=0h: Prepare for command retry
exitCode = 1013

At first I wasn’t sure what to make of this. Maybe it was the cable or connection, but on all four drives? It was a special 4-in-1 (SFF8087) cable, but it still seemed odd. I logged the case with 3ware’s technical support and got back a response suggesting I try another cable. Well, duh, I could have figured that myself. I was hoping they might be able to point out any other less obvious potential causes.

So, I purchased another cable. It took a couple of days to arrive and did absolutely nothing to resolve the problem. Sigh. At the same time as this was going on I had another problem - it’s only with hindsight that I know to separate the two:

E=0204 T=18:34:36     : Port timeout (ext)
SATA Device. port = 0x2
task file written out : cd dh ch cl sn sc ft
                      : 00 04 00 00 00 00 00
Send AEN (code, time): 0x9, 06/10/2008 18:34:36
Drive timeout detected
(EC:0x09, SK=0x04, ASC=0x00, ASCQ=0x00, SEV=01, Type=0x71)
phy=6
  task file read back : st dh ch cl sn sc er
                      : 00 00 00 00 00 00 00
E=0204 T=18:34:36 P=2h: Soft reset drive
E=0204 T=18:34:36 P=2 : Inserting Set UDMA command
E=0204 T=18:34:36 P=2h: Check power cycles, initial=40, current=40
E=0204 T=18:34:36 P=2h: exitCode = 1013
Port retry not allowed
E=0204 T=18:34:36 P=2h: Prepare for command retry
exitCode = 1013
E=0204 T=18:34:36 U=0 : Retrying command

These errors happened less frequently, but eventually caused I/O to hang and the controller to reset. Again I logged this with 3ware’s technical support and got back a bunch of not so helpful responses. They suggested moving the card in the machine, testing the disks, checking the power supply, and so on. All valid points, but what annoyed me was they could only ask me to check one at a time… and they could only reply to me once a day. Plus I’d already done everything they suggested. It took a week to go through this nonsense.

In the mean time I spent a lot of time experimenting, fiddling, and web searching. Eventually I found the following two pages, although it took me a while to realise their significance:

http://www.3ware.com/KB/article.aspx?id=15385
http://www.3ware.com/kb/article.aspx?id=15171

The first of the articles explicitly mentions my controller card and drives, so it seemed to be the right thing to do. But I had the SN04 firmware on my drives and they wanted me to apply AN05. I asked both 3ware and Seagate to clarify the differences, but neither gave satisfactory answers. Seagate managed to give me the SN05 firmware to try, but it didn’t help. In slight desperation, and without anyone giving me much help, I decided to take a punt on the AN05 firmware.

IT WORKED!

There was a lot of tension for the next few hours whilst I continued testing, but eventually I was satisfied that the AN05 firmware solved the problem. Later attempts to clarify with Seagate why SN05, which they gave me, didn’t work and AN05, which 3ware pointed me at, did work, got nowhere. Seagate support actually admitted that they basically don’t know.

So on to the next issue. The second article suggested limiting the speed of the drives to work around the drive timeout issue. It’s definately a workaround, but it was worth a shot. I’d already removed the jumpers from the drives that limited them to 1.5 Gb/s, and they were a nightmare to do - I’ve never seen such small and fiddly jumpers on a disk… it was completely unnecessary given the available space. This time I decided to do the limiting in the 9690SA’s software.

ONCE AGAIN, IT WORKED!

So at this point I’m happy. Things are looking good. That last fix is definately a workaround, and I’ve told 3ware they need to fix it. It’s a bug, and bugs need fixing. I’m now using the array to store my data on, it’s nice and quick (a 512MB write cache helps!), and I have plenty of space. And Ruth might get the dining room table back soon… assuming I can work out how to lift this massive machine (did I mention the case was quite big?).

But I’d like to finish this post with a rant. It turned out that the solutions to my problems were both in the 3ware knowledge base. Now maybe I should have searched harder initially, but it took me some time to find these articles. But more to the point, 3ware support should definately have known about these issues and should have directed me straight to them. I wasted a week of my time messing around with them, and I’m not happy about it. The card is great (apart from the aforementioned bug), but the support sucks. It will seriously make me think twice about going with 3ware again.

I hope this post will fill in the whole story to those I’ve been ranting at recently, and maybe it’ll help someone else on the Internet out if/when they hit the same problem. That’s assuming they can read this lengthy post in less time that in takes to figure out the solution themselves ;-).

Good night.

Since we spend quite a bit of time in Falmouth I thought I’d just quickly list some of the places we like to eat. This was originally just going to be a post about our favourite place, but I figured I’d list a few others as well.

Our number one place to go is Five Degrees West at the Arwenack Street end of town (for those that remember the old Pirate pub, it’s where that used to be). It’s a modern pub looking pub with a fairly relaxed atmosphere. The staff are really friendly and nearly always manage to say hello and goodbye as you walk in or out. The food is excellent for a pub (and probably beats a lot of restaurants too), and it has a decent selection of beverages. A lot of the food and drink is sourced locally too. Price wise it’s probably more than your average pub, but below that of a restaurant.

I’ve rated Five Degrees West as our number one place because it’s a great all rounder. It’s good for lunch, an evening meal or just a relaxing drink. It even has free Wi-Fi. So I recommend you take a visit if you’re in Falmouth, and I further recommend the Bacon and Cornish Brie Ciabatta

We also have a couple of favourite restaurants which by chance are right on top of each other. They are The Warehouse Bistro and Clarks Restaurant, which are located on Custom House Quay. Warehouse has a smaller, older and cosier feel to it, and has a lovely fillet steak (I recommend the Stilton sauce). Clarks is more modern and has some nice food on the menu. Both are obviously more pricey than Five Degrees West, but worth a visit once when we’re in Falmouth.

I’d also like to briefly mention The Hut. We went there last year and had a good time, but haven’t had the chance to go again.

On the cheaper side we have a Weatherspoons at the other end of town. It was good when it first opened, but it’s starting to look a bit shabby now. The food can be variable and likewise for the service. We still go in on occasions, but usually in the evening for a drink rather than food.

No post about eating out in Cornwall would be complete without a mention of the good old Cornish Pasty (thanks pao for spotting this glaring omission!). There’s only one place to get a Pasty in Falmouth - that’s Rowes Bakery near the Prince of Wales pier. Well, there’s absolutely loads of places, but I only go there

Falmouth is awash with restaurants, and more seem to be opening all the time (the old Post Office building has turned in to an Italian!). I haven’t been to a lot of them, but if you’ve been somewhere good and you happen to be reading this, feel free to leave a recommendation

It took me a while to figure this code out, and there seemed to be a lack of complete examples on the web to do exactly this, so I thought I’d document it.

I needed to connect to an LDAP server using a Kerberos principal for authentication from within a Perl script. This meant that it needed to do it without any external input, so it couldn’t rely on a password being entered or someone doing a kinit first.

The code is fairly simple. It basically gets the right credentials using a pre-initialised keytab and then sets up the relevant objects and uses them to bind to an LDAP server.

#!/usr/local/bin/perl -w    

# How to connect to an LDAP server using GSSAPI Kerberos auth.    

use strict;    

use Net::LDAP;
use Authen::SASL qw(Perl);
# This module makes doing the kinit much easier
use Authen::Krb5::Easy qw(kinit kdestroy kerror);    

# Location of the keytab which contains testuser's key
# exported in kadmin by: ktadd -k /tmp/test.keytab testuser
my $keytab = '/tmp/test.keytab';
# Where to store the credentials
my $ccache = '/tmp/test.ccache';    

$ENV{KRB5CCNAME} = $ccache;    

# Get credentials for testuser
kinit($keytab, 'testuser@CS.UKC.AC.UK') || die kerror();    

# Set up a SASL object
my $sasl = Authen::SASL->new(mechanism => 'GSSAPI') || die "$@";    

# Set up an LDAP connection
my $ldap = Net::LDAP->new('ldap.cs.kent.ac.uk') || die "$@";    

# Finally bind to LDAP using our SASL object
my $mesg = $ldap->bind(sasl => $sasl);    

# This should say "0 (Success)" if it worked
print "Message is ". $mesg->code ." (". $mesg->error .").\n";    

# Clear up the credentials
kdestroy();

Hopefully this will help someone else out. Comments welcome

It’s been nearly 2 years since I intially set up my IPv6 connectivity, and back then I had some problems with the BT Exact IPv6 tunnel broker. Now it seems that without much notice the service has been taken down permanently, so I’ve just spent quite a few hours moving over to a new provider - SixXS.

My initial impression of SixXS is that it’s much more polished than the BT service was. They have many PoPs (although a tunnel is only associated with one nearby PoP), a decent website, and all the facilities that I need. They work on a credit based system which means to use a facility you need credits. You get some credits when you set up an account, and you gain more by running a reliable tunnel. It’s an interesting idea… it encourages you to look after your setup.

Handily I got some bonus credits for my work on some Open Source projects, so I got both my home network and my colo server setup in one go. The process was as good as identical to the BT service, so there were no real problems - just the tediousness of updating configs, DNS entries and firewalls.

So there we have it - SixXS++

Yesterday we went and bought a new camera. I’ve been trying to decide what to get for ages now, and I’ve managed to go all the way back round to my first choice.

We picked up a shiny new Nikon D80 with the Nikon 18-200mm VR lens from Jessops in Canterbury. I’d already purchased a pair of Sandisk 2GB SD cards and a Hoya 72mm Protector filter online (Play.com and PurelyGadgets.co.uk respectively), since they were considerably cheaper than Jessops.

I wasn’t sure what to expect from Jessops. I’ve become quite cynical in recent years and I’ve come to expect being messed about and ripped off. But, to my delight, Jessops was great. I was served by a guy called Chris in the Mercery Lane store who thankfully knew what he was doing. He was also more than happy to price match a couple of magazines I’d taken in (containing the Jessops price match promise). I made a saving of around £150 from their list prices, and Nikon are offering a further £50 cash back. So I’m a happy man.

Now I would go on to say how great the camera is, but I can’t just yet. The first reason is that it’s meant to be a Christmas present, so I really ought to leave it alone for a couple of weeks. But, ignoring the first reason, it’s been either dark or pouring with rain since I left the store, so I haven’t had much opportunity.

Roll on Christmas!

Recently I changed my wife’s email address and user ID on eBay. It was pretty painless using their web interface… at least, that’s what I thought.

The problems came a couple of weeks later when she was still receiving solicited promotional material to her old email address. I figured it wouldn’t be that hard to find out why, so I filled in a web form asking if they could check things out. This was their first reply:

Since you have completed the change of address request, be assured that all the eBay emails are sent to your registered email address. The only possibility in this situation is that your ISP (AOL in your case) might have linked both the email addresses to your account. So, we’d suggest you to contact your ISP and confirm if this is the case. However, if this is not the case on their end, then you will need to send us an eBay email with the header.

Interesting. I had a word with her ISP, which was pretty easy given it’s me. Last time I checked I’m pretty sure I don’t run AOL either (thank goodness!). I took a look at the headers, and they look pretty conclusive to me (interesting bits only):

Received: from smfcamppool09.emailebay.com ([66.135.215.238]
	helo=smfitemap04.smf.ebay.com) ...
DomainKey-Signature: s=main; d=reply3.ebay.com; c=nofws; q=dns;
	b=NR1bQ5kTLijbb5Mc3TmFcKdB+BLWEb1YZvYiyvzns2iWz8iyi
	JVBCXP3ERh+lxAYiwwR3kbd94Zg3xyPvcW8CDscQaHYizuzh5vd
	59IOlVCKr1qwAYNvDHTmxMx5RL18;
From: "eBay" <eBay-INTL@reply3.ebay.com>
Subject: [her userid], knock his Christmas socks off this year
	with eBay
Received-SPF: pass (carrick.bishnet.net: domain of
	reply3.ebay.com designates 66.135.215.238 as permitted
	sender) client-ip=66.135.215.238;
	envelope-from=eBay-INTL.403108935.71560.0@reply3.ebay.com;
	helo=smfitemap04.smf.ebay.com;

So I sent that off to them and awaited their next reply. Here’s what they said:

Thank you for your reply. I understand that you are concerned about changing of the email address on eBay.

While checking your account status, I noticed that you have successfully changed your email address from ‘[old address]‘ to ‘[new address]‘ on Oct 27, 2007.

Your new email address ‘[new address]‘ is now enabled on eBay.

Looks like they’ve completely missed the point and have decided just to state the obvious instead. So, once again, I explain that the problem is that email is still going to the old address.

In comes the next reply:

I really want to help you resolve this issue because I know how important it is for you to have this matter settled. However, your message didn’t include the email header, which I need in order to take action.

Now it looks like they’re repeating themselves. Funny thing is, if you scroll down their email you’ll see they’ve quoted the last time I said I provied the headers. So, with a few rants about their inability to read the case history, I give them the entire email, headers and all, again.

And here’s were it starts getting really good. I had to read this a few times to believe they actually said it:

Thank you for writing back again regarding the unsolicited email you received. I’m sorry that this matter hasn’t yet been resolved.

I’ve checked the information you sent us and I can confirm that the email was not sent by eBay, and is not endorsed by eBay in any way. However, it appears to have been sent by another eBay member.

How to cut down on spam emails …

Now hold on a minute. For a start, I’ve never said it’s unsolicited, it’s just going to the wrong address. And now, to top it all off, they’re trying to say they never sent it and that another member did!

I took a few minutes to cool down before calmly asking them to explain how exactly they came to that conclusion. I also suggested that if they can’t answer my questions they should consider escalating the query to someone who can.

Finally I manage to make contact with one of the (I summise) 20% of their staff who know what they’re talking about:

Please understand that when you change your email address on eBay it will remain in our database for the next 30 days and once this period is over, you won’t receive any email at your old registered email address as our system releases that email address from the database.

It took a week and 10 emails for that conclusion to be reached. It’s not rocket science, is it?

All I can say is that I’m glad it wasn’t something important…