A Blog by Tim Bishop

Recently I changed my wife’s email address and user ID on eBay. It was pretty painless using their web interface… at least, that’s what I thought.

The problems came a couple of weeks later when she was still receiving solicited promotional material to her old email address. I figured it wouldn’t be that hard to find out why, so I filled in a web form asking if they could check things out. This was their first reply:

Since you have completed the change of address request, be assured that all the eBay emails are sent to your registered email address. The only possibility in this situation is that your ISP (AOL in your case) might have linked both the email addresses to your account. So, we’d suggest you to contact your ISP and confirm if this is the case. However, if this is not the case on their end, then you will need to send us an eBay email with the header.

Interesting. I had a word with her ISP, which was pretty easy given it’s me. Last time I checked I’m pretty sure I don’t run AOL either (thank goodness!). I took a look at the headers, and they look pretty conclusive to me (interesting bits only):

Received: from smfcamppool09.emailebay.com ([66.135.215.238]
	helo=smfitemap04.smf.ebay.com) ...
DomainKey-Signature: s=main; d=reply3.ebay.com; c=nofws; q=dns;
	b=NR1bQ5kTLijbb5Mc3TmFcKdB+BLWEb1YZvYiyvzns2iWz8iyi
	JVBCXP3ERh+lxAYiwwR3kbd94Zg3xyPvcW8CDscQaHYizuzh5vd
	59IOlVCKr1qwAYNvDHTmxMx5RL18;
From: "eBay" <eBay-INTL@reply3.ebay.com>
Subject: [her userid], knock his Christmas socks off this year
	with eBay
Received-SPF: pass (carrick.bishnet.net: domain of
	reply3.ebay.com designates 66.135.215.238 as permitted
	sender) client-ip=66.135.215.238;
	envelope-from=eBay-INTL.403108935.71560.0@reply3.ebay.com;
	helo=smfitemap04.smf.ebay.com;

So I sent that off to them and awaited their next reply. Here’s what they said:

Thank you for your reply. I understand that you are concerned about changing of the email address on eBay.

While checking your account status, I noticed that you have successfully changed your email address from ‘[old address]‘ to ‘[new address]‘ on Oct 27, 2007.

Your new email address ‘[new address]‘ is now enabled on eBay.

Looks like they’ve completely missed the point and have decided just to state the obvious instead. So, once again, I explain that the problem is that email is still going to the old address.

In comes the next reply:

I really want to help you resolve this issue because I know how important it is for you to have this matter settled. However, your message didn’t include the email header, which I need in order to take action.

Now it looks like they’re repeating themselves. Funny thing is, if you scroll down their email you’ll see they’ve quoted the last time I said I provied the headers. So, with a few rants about their inability to read the case history, I give them the entire email, headers and all, again.

And here’s were it starts getting really good. I had to read this a few times to believe they actually said it:

Thank you for writing back again regarding the unsolicited email you received. I’m sorry that this matter hasn’t yet been resolved.

I’ve checked the information you sent us and I can confirm that the email was not sent by eBay, and is not endorsed by eBay in any way. However, it appears to have been sent by another eBay member.

How to cut down on spam emails …

Now hold on a minute. For a start, I’ve never said it’s unsolicited, it’s just going to the wrong address. And now, to top it all off, they’re trying to say they never sent it and that another member did!

I took a few minutes to cool down before calmly asking them to explain how exactly they came to that conclusion. I also suggested that if they can’t answer my questions they should consider escalating the query to someone who can.

Finally I manage to make contact with one of the (I summise) 20% of their staff who know what they’re talking about:

Please understand that when you change your email address on eBay it will remain in our database for the next 30 days and once this period is over, you won’t receive any email at your old registered email address as our system releases that email address from the database.

It took a week and 10 emails for that conclusion to be reached. It’s not rocket science, is it?

All I can say is that I’m glad it wasn’t something important…

I started working on CSProjects quite a few months ago.

Problems started early on. I began by bringing our software up-to-date. This included Apache, Python, Subversion, Trac and mod_python. It took some time, but I didn’t experience any problems… until I tried to run them. Seemingly at random, but quite frequently, the Apache children would get a Bus Error. I googled around and discovered this was a fairly common problem, but none of the solutions (mostly involving library versions, particularly expat) seemed to make any difference.

After a few weeks of recompiling, stripping things down to the bare bones, turning on debugging and staring endlessly in to the output of gdb, I struck upon a solution. And annoyingly it wasn’t in any of the things I’d be staring it, but instead it appeared in the form of mod_wsgi. This wonderful piece of code does a similar job to mod_python, so I dropped it in and hoped for the best. Nope, it still crashed. But what saved me was the documentation - the author wrote, and I quote:

Do note though that some versions of the Subversion Python bindings apparently have problems when being used from within secondary Python sub interpreters rather than the main Python interpreter. The result of this will be strange Python exceptions or the Apache child processes could even crash.

To avoid such problems, the Trac application should be forced to run within the main Python interpreter. This can be done using the WSGIApplicationGroup directive with the value ‘%{GLOBAL}’.

This was precisely my problem. So I did as suggested and to much relief everything worked. And mod_wsgi a’int half good too… in my opinion it’s much better than mod_python.

At this point I had all the software working. So I took a month off. Literally.

When I returned I had to move all of our frickin’ servers. But after that I got back to CSProjects.

With the help of Adam Sampson I got down to the business of bringing these software packages together in to something we could offer our users. We did a lot of coding and a few weeks later the final CSProjects was published. Then we had to change it all and another week later CSProjects was published again. Today we launched it to our users and we already have a whole bunch of people using it. Which brings a satisfying end to a few months of work.

Oh, and the logo. I did that (mostly - I got a little bit of help). Sometimes the simple things work best…

Today was a day we’ll remember for a while. Ruth & I got up early, but separately, and anxiously got ready. We met up with some family and friends in church, then headed out for a meal. The meal was concluded with some amusing anecdotes and some drinks. We enjoyed music in to the evening, danced a bit, ate some more food and drank plenty. Then exhausted we retired at the end of a memorable day.

Confused? This might clear things up…

Not that I’m counting, but it looks like I have 39 years, 1 month and 11 days to go until I retire.

Work out how long you have left using this handy calculator:

State Pension Age Calculator

There are three credit referencing agencies used in the UK. They are Experian, Equifax and Callcredit. You have a statutory right to request a copy of the information they hold on you, for which you are charged £2. All of them offer full reports for around the £15 mark, but these just contain analysis and views on how lenders will interpret the information. There are times when a full report could be useful, but not for me at the moment.

I recently requested the statutory reports from all three agencies and they all showed up within a few days. Although they all contained similar information the Experian report showed that it was the agency lenders were primarily using to do checks with. It was interesting to see just how much information is contained in the report and how often you’ve had checks run against yourself. Did you know that some insurance companies do it when you get a quote?

So why would you want to do this? The main reason is to keep tabs on the information held about you and to check for anything suspicious. If someone has stolen your identity you might find things cropping up on your report that have nothing to do with you. They also give you the opportunity to find and correct mistakes, which is better than trying to deal with problems when a lender does a check on you.

For £6 in total it’s not worth worrying about the money, so I plan to get reports on a yearly or bi-yearly basis.

Hopefully this provides you with a little insight in to this area - I knew little about it until I went through this process.

Thursday 22 February. That’s the day it all went wrong.

I was on my way home from a shopping trip at Sainsburys. We’d been on a Thursday instead of a Friday because we had to go to Cornwall on the Friday afternoon for a funeral (that’s bad thing number one). Just after leaving Sainsburys we came across an area that was unusually dark. I guessed this was a power cut, so I was beginning to worry about the state of things at work. Then we hit an area that was lit up, then another that was dark, then another that was lit up. Unfortunately we got to University to find it in darkness. That’s bad thing number two.

I sent Ruth home with the shopping and headed in to see what I could do. Our core equipment has a couple of hours UPS provision and the non-core stuff has virtually no UPS provision, so there was no immediate rush - the machines were either fine or already off. This was when the third bad thing happen - the card locks had failed. Although I could get in to the Octagon lobby I was unable to get anywhere else in the building. After some wandering around the caretaker managed to find a way to the machine room bypassing the card locks, but this still left the challenge of entering the room itself.

I waited for someone else to show up, hopefully with a key for the machine room, but nobody did. In the end I gave up and went home to try and use the computer there to reach people. Success! Someone from operations was coming in with a key. I grabbed a quick bit of food and rushed out of the door to meet them. Suddenly I felt a seering pain in my finger. I glanced at it and wondered why it was purple and throbbing with pain. I’d stupidly managed to catch it in my large metal front door. There’s problem number four. After a few minutes of putting ice on it I headed back up to work.

The next part of the story is a bit dull. I shut everything down and went home to sleep. I’d figured I was better off coming in early the next day rather than staying late, particularly with the state of my finger. 7am next morning I headed back.

It took a good few hours to get everything going again. There were plenty of problems with the cluster which all turned out to be related to the last bad thing. The main disk array (a Sun StorEdge 3510) is a fully redundant unit - it has dual controllers, dual PSUs, dual fibre links, and a RAID 5 disk configuration. One of the power supplies is linked to our UPS and the other to the main machine room UPS. In a power failure the machine room UPS lasts only a few minutes so we drop down to only one PSU. So it would help if that PSU was functioning. It all appeared fine - lights were on, fans whiring - but the unit just wouldn’t power on with just that single PSU. Consequently when the power cut occured that disk array went down.

That problem was compounded by the way we’d set up our coordinator disks. We originally only had this one array so we had three coordinator disks on the one array. Later when we added two more arrays we added a coordinator disk on each of those but fataly left three on the original array. When this array turned off we were left with less than a majority of coordinator disks which caused all the cluster nodes to panic - they assumed they were cut off from the main part of the cluster. So even the services that should have remained running were killed off.

The disk array powering off also caused corruption on one of our filesystems. This was easily fixed with a fsck (thank goodness we had VxFS, otherwise all our ACLs would have vanished - last time I checked that was still a problem with UFS). It had also corrupted our MySQL databases, but after some rather long checks these were also fixed. One of the MySQL databases handled our email services, so we had a whole bunch of problems there too.

We’ve now had the PSU replaced by Sun, with unusually little fuss, so maybe it’s a known problem. But we can’t test it properly without potentially killing the array again, so we’re holding off doing it until we next power everything down. I’ve swapped the power cables over so another power failure shouldn’t land us in such a mess.

Isn’t it great when your highly available systems work?

Back in the middle of last year I wrote about our plans to tackle our NFS performance issues by introducing a direct and dedicated network link to carry our NFS traffic between the clients and the servers. We’d done the tests so we just had to implement it.

First we waited for the financial year (1 August) to roll over so we had some money, then we went and purchased 2 gigabit switches and 4 quad port network cards for the client machines. We only really needed dual port cards, but the supplier gave a choice of single or quad. The server nodes, which are part of our cluster, already had sufficient ports (8 per node, 6 of which are now used!).

You’d think it’d be pretty simple from there? It wasn’t. We hit a couple of snags:

1. How do we get the cables directly from one cabinet to another? All our cabling patches back to a central network cabinet, but we don’t have enough patch panel ports to send them all there and back.
2. What colour cables should we use? We can’t use the normal colour!

After much deliberation other jobs came along and consumed my time for the next few months.

So, a couple of months ago we took the plunge and made an important decision. We got blue cables. This would avoid confusion with our yellow cables (normal network), purple cables (crossover network), red cables (serial network), green cables (serial rolled) or grey cables (to be burnt alive). We also noticed some handy holes in the tops of the cabinets and neatly threaded the link cables to the servers through there.

With the hard work done we set about putting the new cards in to the client machines. The first machine was a good test case and I spent a while sorting our configuration and automounter setup to deal with the new link nicely. The remaining machines I’ve done on Tuesday mornings this month, with the last one being done today.

This leaves us with the important question. Is it actually quicker? My raw tests prove it is - ping times are halved and times to transfer large files are also halved. Loading my email in mutt is much quicker, as is listing my overly populated home directory.

Sorted then? Not quite - we still have a few users with Exmh slowdown problems. We’re still investigating that. Maybe Exmh is doing something pathalogically slow that doesn’t agree with NFS. Or maybe it’s just getting a bit slow in its twilight years. I’ll leave that to the boss (the only Exmh user in our group) to figure out

So, I don’t post for the best part of half a year and then all I can be bothered to do is link to a colleague’s Celebrity Gossip page. It’s a favour, OK?

Oh, and he wanted to make sure that everyone was aware that this content is just for him and that he really enjoys reading it. I think.

So my current hosted server is getting a bit old. It’s not got enough RAM, and the disk in it is failing (yes, I did have RAID, more on that later). So it’s about time to get a replacement in.

The guys over at Netrino have just installed a new machine for me. I say new, but it’s not a brand spanking new bleeding edge state of the art all singing all dancing machine costing a million pounds. It’s just an Intel Celeron 2Ghz, with 1GB of RAM, and two 80GB hard disks. The main thing is the increased RAM, and two new (and hopefully working) disks.

Things didn’t get off to a good start on day 1 - they didn’t have a FreeBSD CD to hand. They “kindly” left me a USB dongle containing a variety of Linux installers, so I had a play with them in the hope that I could somehow bootstrap a FreeBSD install from one. After some googling I found the Depenguinator that claimed to do exactly what I needed. A few hours later I discovered it didn’t - probably because its not been updated for more recent FreeBSD versions.

On day 2 things got off to a better start - a FreeBSD 6.1 CD arrived in the CD-ROM drive and booted nicely. I had a quick play around to check everything - particularly the network card - worked, and thankfully it all did. Next came the installation.

One area I’ve had quite a few problems with on FreeBSD is the software RAID provision. You’ll see in one of my previous posts that I had some fatal problems with gvinum, and since then I’ve had other problems recovering a RAID 5 failure using it. Another alternative is ataraid, which worked fine for me up until FreeBSD 6. Since then I’ve not been able to get it to resync a failed disk properly - it hangs at 0% forever. So those two solutions are written off.

Other than hardware RAID this leaves me with a clear choice: gmirror. I’ve been using gmirror on another machine for some time now, and I’m pleased with the results. Following this guide it’s easy to apply it after installation, which is a definate selling point - I can’t stand solutions that require dumping and restoring. On my other machine its also had no problems resyncing, so another box ticked (or not ticked on the reasons-to-avoid list, actually).

I’m left wondering here how hard it would be to add support for gmirror (and maybe some of the other geom providers) to the FreeBSD sysinstall program. I experimented with setting up the mirror by hand before running the installer, but it failed to notice it. If this functionality could be added it’d be a real selling point for FreeBSD.

So, with the install complete, and the disks mirrored, I’m ready to move on to building and configuring. First up is updating the world and kernel, then installing all the software. I’ve not really figured out how I’ll copy everything across though…

Back in May I wrote about the performance problems we were having with our new NFS based user filestore. It’s been a while since then, and the problems have continued. We have noticed that it appears to be load related - not just the network, but also the machine. This suggests that our theories about IPsec causing the slow down may be correct.

Our original plan was to try a private network which would remove the need for IPsec and also remove any latency added by routing the traffic between our subnets. This still seemed like a good plan, so I asked around and another department kindly lent us a brand new gigabit switch. We’ve connected this to one of our NFS clients and to the cluster node that’s currently running our filestore.

So far we’ve noticed some serious performance boosts. There’s only a few of us using it, so it could just be that it’s a lightly loaded connection - time will tell on that one. The bottom line is that it seems to be quicker than the IPsec connection ever was, so hopefully we’re on to a winner. We’ve also got a few staff testing it out, and their responses have been positive so far.

The next step after this testing period is to look at the costs of doing this properly with our own equipment. One of the key things we’ve been doing recently is increasing the redundancy of our systems, so it’d be fairly daft to do this with just one switch. We’d need at least two, with every cluster node connected to both, and every client that we want optimum performance on connected to both. Obviously there’ll be other clients that are less important and they can continue to use the existing infrastructure.

Of course, I’ve got absolutely no idea where we’ll put these switches, or how we’ll wire them in - things are pretty tight in our racks at the moment. Suppose there’s got to be a challenge somewhere

My only worry with all this is what we’ll do if it doesn’t work. I don’t have any other ideas that’d make it go quicker - to be frank, you can’t really get any quicker than a directly connected switch. Lets hope we don’t have to worry about it.