Increasing our storage provision

Wednesday, October 14th, 2009 in Computing, Work

During the summer we started getting tight on storage availability. It seems that usage on our home directory areas constantly increases – people never delete stuff (me included!). We were running most of our stuff through our Veritas Cluster from a pair of Sun 3511 arrays and a single 3510 array. Between them (taking mirroring in to account) we had around 3TB of space.

Now, it’s a well known fact with maintenance contracts that the cost goes up over time (parts get more scarce and more costly). So we did the sums on the cost we were paying for the old arrays and realised that over a sensible lifetime period it was cheaper to replace them. So we got a pair of Sun 2540 arrays with a 12TB capacity each.

Since our data is absolutely precious we mirror these arrays and use RAID 6. This gives us just under 10 TB of usable space, which is a fair amount more than we started with.

The next stage was bring this online. Because we use Veritas Volume Manager and the Veritas File System we were able to do this almost transparently. The new arrays were provisioned and added to the relevant diskgroups. The volumes were then mirrored on to them and then the filesystems expanded. Finally the old arrays were disconnected. All of this was done without any downtime or interruption to our users or services.

I said almost transparently though. It seems it’s not possible to change the VCS coordinator disks without taking the diskgroups offline and back online (this might be improved in VCS 5). So I rebooted the whole cluster last weekend and it was all finished.

The problem with all this clever technology? Nobody knows we’ve done it. After weeks of work we grew the filesystems just before they completely filled and without any noticable downtime. We’d probably get more gratitude if we’d let it fill up first ;-)

  • Share/Bookmark

Getting the indexes right for OpenLDAP when using NSS

Wednesday, October 14th, 2009 in Computing

I recently deployed a Linux system which used the libnss-ldap module to get its passwd and group information. This all worked fine except group lookups (in particular when logging in) which were extremely slow. We have about 600 groups in our directory, which isn’t massive, but is more than the average system.

Clearly this wasn’t right. Initially I tried nscd, which helped, but only after it had cached the data. Then I realised it was probably the indexes in OpenLDAP. Googling didn’t turn up much of use (hence this post), but I did find this page on the OpenLDAP site.

This fairly quickly pointed me at the problem; I was missing indexes on memberUid and uniqueMember. Adding these fixed the problem completely.

So here’s the indexes I’ve ended up with:

index   objectClass     eq
index   cn,uid          eq
index   uidNumber       eq
index   gidNumber       eq
index   memberUid       eq
index   uniqueMember    eq
index   entryCSN        eq
index   entryUUID       eq

(the last two are for replication)

I’m actually quite surprised how much the indexes matter. It makes a huge difference, even on a small setup. So if you’re setting up a directory take the time to read the Tuning section of OpenLDAP Admin Guide first.

  • Share/Bookmark

PAM – Locking out accounts when using external authentication

Friday, October 9th, 2009 in Computing

One of the projects I’ve been working on lately has required me to investigate the way PAM handles external authentication. The setup is a bunch of different boxes running Solaris, Linux and FreeBSD, but all authenticating from a single central Kerberos domain. There’s also LDAP in the mix, but it’s not actually relevant, so I’ll ignore it for now.

The problem comes when you want to lock out an account on a machine. By this, I mean the account still exists in the passwd file (for whatever reason, it needs to stay there) and the account still exists within the Kerberos domain because it can log on to other machines.

What are the options? The PAM setup probably looks a little something like this:

auth    sufficient pam_krb5
auth    required   pam_unix

account required   pam_krb5
account required   pam_unix

This setup allows the user to authenticate with just their Kerberos credential, but they must pass both modules in the account section (the pam_krb5 one here doesn’t do much though).

So we can put whatever we want in the shadow password entry on the machine, it won’t affect the authentication – it never even gets there. But what about the account section? Can this do anything?

After some investigation and source code checking I came up with the following:

  • On Solaris one can put *LK* in the shadow password field. This causes the pam_unix module to reject the login in the account stage.
  • On FreeBSD one can do the same as Solaris but with *LOCKED*.
  • On Linux you can put what you want in the field but pam_unix won’t take any notice.

If you’re thinking right now “hey, you can just put an exclamation mark in the shadow password field to lock the account out” you’re wrong. What this does on a standalone system is ensure that no possible password can decrypt to match this string which results in locking the account out. But, when we have external authentication it’s meaningless.

So what’s the solution? There’s another field we can use – the shadow password expiry field. This is a time stamp in seconds since the epoch at which the password expires. The default setting of 0 means no expiry. Simply setting this to 1 (or any number smaller than now) results in the account being locked out. This seems like it’ll work across all three operating systems I’m testing, although I’m yet to finish my analysis.

I am a little disappointed with the Linux PAM module. It’s a simple test to do, so I can’t see any reason why it shouldn’t be added.

Update 2nd November 2009:

I’ve since discovered that what I said above about the “default setting of 0 means no expiry” is incorrect. In fact, it varies on different operating systems. I got misled by the behaviour on Ubuntu 9.04 because of a Debian patch which made 0 mean no expiry, but which has been removed in Ubuntu 9.10 (details here).

Consequently my advice now is to set the shadow password expiry field to 1 to lock the account, and to unset it to unlock the account. This appears to work on all the operating systems I’ve tested it on.

References:

  • Share/Bookmark

A new look CS website

Wednesday, July 29th, 2009 in Work

A little over 3 years ago I posted about the reskin the Computer Science website at the University of Kent had just received. Well, we’re there again, although we’re now called the School of Computing.

I’ve been working on this project for the last few months. It started out with some meetings with the Information Services web team. They talked us through their current systems and then did a good job of selling their new system (which later became known as Pantheon). We decided to go with the new one, but hit a slight snag – it wasn’t quite written yet.

After a bit of waiting we finally got our hands on the code and I began the integration work. Although we’re deploying their system, we’re running our own copy of it on our servers. This is a good mix; it keeps us in sync with the central web development, but allows us to maintain the flexibility we’ve had for years. So this first integration step was a little slow, and it took some lengthy IRC conversations and a session around my desk to finally get it sorted.

Over the next month or so I spent more time tying this in to our system. I found various bugs and issues with the code and fed my fixes back to the development team where necessary. I also tried to bring across the look from our blog, and sorted out the configuration and menu structures. Then, with a working system in place, and new theme from the web team called Bacchus, I was ready to do the final integration step – tying it in to our existing build system.

At this point time was becoming more of an issue. We could have just chucked all our old ways of doing this out of the window. This might have been less effort for me, but it would have been more for everyone else. The decision was made to stick with our old infrastructure and tie the new system in to our templates. This didn’t turn out to be too hard, and only took a couple of weeks to complete.

What we’re left with is an interesting system. Our build scripts take user generated content and wrap them up with our templates to generate a HTML file. Then, at runtime, this HTML file (which contains special markup) is fed through the Pantheon system which uses the Bacchus template to render the final page. So we’ve got a baked and fried system all in one :D .

The final step was a week long effort to rebuild pages and tweak the various build scripts and jobs to generate reasonable looking and valid XHTML pages. This was more tedious than technical I suppose, but a satisfying finaly step to the project. Then the work passed over to our web editors to start reorganising and rebuilding the content.

And there we have it – the new Computer Science website. Just don’t dig too deep or you’ll find the pages we haven’t finished with yet :-) .

  • Share/Bookmark

Implementing SRS with Exim and SQLite

Wednesday, July 29th, 2009 in Computing, Work

Due to issues with SPF-style restrictions I decided I’d take a look at implementing SRS (the Sender Rewriting Scheme, a good description of which is over here) with Exim. I thought it’d be fairly straightforward and well documented, but it wasn’t. I’m left wondering if anybody actually does it like this?

To start with I built Exim on FreeBSD with the SPF and SRS libraries (I used libsrs_alt). This was straightforward enough – although where are the options in the FreeBSD port? – and would be standard across most operating systems.

Since this was a trial run I thought I’d take the lightweight approach and use SQLite rather than set up a full database like MySQL. This again was a straightforward install.

Lets look quickly and the SQLite tables. The instructions on the libsrs_alt site don’t talk about creating these, so I just figured it out for myself. This is what I ended up with:

# sqlite3 /var/tmp/srs.db .dump
BEGIN TRANSACTION;
CREATE TABLE SRS(Key TEXT, Address TEXT, Time INTEGER);
COMMIT;
#

That’s pretty simple. The database should probably live somewhere a bit more permanent though :-) .

Right, back to the instructions. Basically we need to add a bit of config and a new router to the exim configuration. Starting with the default configuration I added the following option in the global section:

hide srs_secrets = asecurestringofareasonablelength

And then the following routers:

srs_reverse:
  driver = redirect
  domains = +local_domains
  srs = reverseandforward
  srs_dbinsert = ${lookup sqlite{/var/tmp/srs.db \
    INSERT INTO SRS ('Key', 'Address', 'Time') \
    VALUES ('${srs_db_key}', '${srs_db_address}', \
    strftime('%s','now'))}}
  srs_dbselect = ${lookup sqlite{/var/tmp/srs.db \
    SELECT Address FROM SRS \
    WHERE Key = '${srs_db_key}' \
    AND Time > strftime('%s','now','-30 days') \
    LIMIT 1}}
  data = ${srs_recipient}
srs_forward:
  driver = redirect
  domains = +local_domains
  srs = forward
  srs_dbinsert = ${lookup sqlite{/var/tmp/srs.db \
    INSERT INTO 'SRS' ('Key', 'Address', 'Time') \
    VALUES ('${srs_db_key}', '${srs_db_address}', \
    strftime('%s','now'))}}
  data = $local_part@example.org

As you can see in this case, I’m simply forwarding all email to another domain. That’s not the most useful setup, but again I’m just testing.

Note that I had to put these in the opposite order to the instructions. The first router is looking for addresses that are already SRS encoded (so bounces for forwarded messages, etc). If it doesn’t find one, it just passes on to the next one. So this way round work best for me.

That’s it really. At a simple level this works. I’ve not looking at proper integration in to the forwarding setup or any kind of database maintenance yet. But given the lack of useful documentation online I thought I’d post my findings.

If you’re reading this and thinking “why on earth is he doing it like that?” please drop me a comment below and enlighten me :-) .

  • Share/Bookmark

Vodafone ordering fiasco

Monday, April 20th, 2009 in General

Around 2 months ago my wife ordered an upgrade to a Nokia 5800 from Vodafone. This should have been a straight-forward every day thing for them, but somehow it ended up taking 2 months and various departments before she finally received the phone.

It all started with a mistake when giving the delivery address over the phone. Once that was corrected the problems started:

  1. Wait for phone for up to a week, receive no dispatch text and no phone.
  2. Phone up.
  3. Get told order was never placed or was cancelled.
  4. Get promised phone will be reordered (sometimes with a promise of a call back to confirm).
  5. Go back to step 1.

Admittedly there were stock problems at the time since this was a new phone, but the delays were still unacceptable. After about the 12th phone call my wife enquired as to why it seemed impossible to actually order the phone. This is how they (2 people said the same thing) said it works:

  • Phones can only be “ordered” during working hours.
  • If the warehouse receives an order and none are in stock they cancel the order.
  • When new stock comes in they all sell out within minutes.

This seemed like an almost unimaginable system – I honestly couldn’t believe it worked like this. How do they sell out so quickly if you can only order when they’re in stock? Are the entire Vodafone staff poised ready to order the second they come in? Madness!

So, after getting authority on my wife’s account, I head on to the Vodafone eForum with this post. They assure me the system can’t be as described, and do their best to help. This is when I discover that they have great difficulty sending emails, half of them getting lost or receiving no response. I’ll admit the staff were polite and helpful, but after a few weeks we still didn’t have the phone.

We decided enough was enough and decided to lodge a complaint. I spent a while trying to find an address to write to on their site, but failed (it is buried in there, I discovered eventually, but it’s well hidden). Whilst googling though I came across the following link:

http://campaigns.vodafone.co.uk/complaints/

Interestingly this was a paid advert on Google, so it seems at least somebody cares enough to deal with the problems. We submitted the form on the weekend and within a day or two got a response. We had one member of staff throughout who was extremely helpful and polite. She confirmed that the system is as crazy as described, but that there’s resistence to change. Within a few days though the phone arrived.

I find it hard to understand how a person in the complaints department can solve the problem so quickly and everybody else can fail. Still, it shows that complaining works :-) .

To help others I thought I’d mention this on the eForum, but my post was rejected! The reason given was:

Whilst we really do appreciate your contribution to the eForum, can we please ask you to refrain from posting this link as it detracts from our desire to help out first hand! We trust this is clear now, and hope to see your contributions to the eForum again soon!

Given that kind of cenorship I don’t think I’ll bother in future…

  • Share/Bookmark

Petition: Savings account summary box

Tuesday, March 31st, 2009 in Finance

Martin Lewis, of moneysavingexpert.com, has launched a petition on the Number 10 government petition site. He’s petitioning the Prime Minister to mandate savings account providers to include the current interest rates on every statement.

I think this is a worthy cause. Recently I’ve been trying to find the rates of my savings accounts and it’s certainly not as easy as it should be. It gets particularly tricky when a provider has multiple accounts with similar names, often the same account but different issues. Working out which you have can take some time.

So, if you agree please take a moment to sign the petition.

Thanks!

  • Share/Bookmark

Wordpress plugin recommendations

Friday, March 27th, 2009 in Computing

I’ve been using Wordpress for a few years and I’m now using quite a few plugins. Since I made use of other’s blog posts to find them I thought I’d return the favour and list the ones I’m using. So, here they are:

Plugins to make life easier for visitors

Visitors are the main reason blogs exist. If nobody is looking at your blog why bother writing it? So here’s a few plugins that make things a little easier for your visitors.

Yet Another Related Posts Plugin – website, wordpress directory

This plugins adds a list of related posts to the end of each post. It’s almost magical in how it works; it somehow just does the “right thing” without any configuration or input from you. It also adds related posts to your RSS feed, which is a great way to pull people back in to your site.

Since I’ve been adding excerpts to my posts lately I’ve configured this plugin to display excerpts in their entirety. I suppose if I write many more posts I might want to shorten them.

Subscribe To Comments – website, wordpress directory

Do you find it annoying having to check back on blogs you’ve commented on to see if others have commented too? If so, this plugin is almost for you. It will allow visitors to subscribe to comments being added to a post on your blog; when someone else comments they’ll receive an email update.

Sadly it won’t do anything to make your life easier visiting other blogs, but the more people that take it up the better. In fact, I’d go as far as to say this should be integrated in to Wordpress itself.

Top 10 – website, wordpress directory

This is a handy little plugin that’ll keep track of visits to your site and allow you to display a list of the top 10 visited posts. It’s pretty basic but does the job. I’d like to see it have an expiry on it so you could list the top 10 posts over the last 30 days, for example (this is already possible – see the first comment below).

I did try and use StatPress Reloaded (see below) to do this, but its listing was pretty naff (didn’t display post titles) and it worked on a URL basis rather than a post basis.

Add to Any: Share/Save/Bookmark Button – website, wordpress directory

There’s a whole load of sites out there for bookmarking and discussing other sites, so it makes sense to allow readers to quickly link to your blog posts. This plugin does just that – it adds a button to the end of each post which allows visitors to share, bookmark and email your posts to all sorts of services and people.

In addition, there’s a Subscribe Button plugin, which I don’t use, that allows readers to quickly subscribe to your blog in a multitude of readers.

Search Engine Optimisation

To get visitors your blog needs to be found. As well as the automatic pings that Wordpress sends out we need to make sure our blogs are easily found and indexed by search engines. These plugins help out with that process.

All in One SEO Pack – website, wordpress directory

This is the most popular plugin in the wordpress directory, and with good reason. It does various small tweaks to your site such as page titles and metadata, which in my experience do have a positive effect on the number of visitors. It’s surprising to see how little changes can make such a difference, but I won’t complain!

The nice thing about this plugin is that it’s all set to go after installation. You don’t even need to do anything to configure it, although it has plenty of options if you want to.

I highly recommend this plugin to all Wordpress users.

Google XML Sitemaps – website, wordpress directory

I don’t have much experience with this plugin, or Google sitemaps for that matter, but given Google’s dominance of the search market it makes sense to play along with it. This plugin creates a map of your site (actually just a list of posts, categories, etc) and submits it to Google. In theory this allows Google to properly index your site, but I’d have thought it’d do a pretty good job of that on it’s own.

It’s pretty straight-forward to set up, although if you have your blog under a sub-directory you might want to add a few static entries for the rest of your site. Then fire up the Google webmaster tools and submit the URL.

It turns out Google can also use your RSS feeds as sitemaps, so I did that as well.

Plugins to make things better for you!

We’ve looked at plugins to make life easier for your visitors and to bring more visitors to your site, but what about you? I’ve haven’t forgotten about the hard working blogger behind the site. Here’s a few plugins to help you out.

StatPress Reloaded – website, wordpress directory

This is an alternative version of the StatPress plugin. It gathers statistics about visitors including which posts they visit, where they came from, and what search terms they used to get there. It’s all information that’s available through Google Analytics (I’ve added that to my template, although there is a plugin to do it for you), but it’s nice to have it easily accessible within your admin area.

The information it provides is pretty interesting. I’ve found a few sites talking about my stuff and linking to me that I wouldn’t have found if the stats weren’t so readily available. I recommend giving it a try.

All in One Adsense and YPN – website, wordpress directory

Pretty much every personal site you visit these days has some form of advertising on it. Most overdo it, but done subtly it can be unobtrusive and provide an income stream. Don’t get me wrong, blogging isn’t a good way to make money – I certainly don’t make much – you have to put in a lot of work before you get anything worthwhile out of it. But if you enjoy writing blog posts why not make a little out of it?

This plugin makes it easy to insert adverts within the text of posts. In fact, there’s probably one somewhere within this post. It’s fully configurable and does most of the hard work for you. However, you will need an Adsense or YPN (which I don’t use) account to do it.

I have had some reservations about the way the donation code works, but it turned out to be just badly written code. Still, it gives me reason to not feel completely confident with this plugin, but until I find an alternative I’ll stick with it.

Math Comment Spam Protection – website, wordpress directory

Up until recently I had a graphical captcha plugin to help weed out spammers’ comments. After the problems I had with it I decided to switch to MCSP instead. It’s a simple plugin – it gives readers a simple mathematical sum to solve when posting a comment. That’s enough to keep the spammers at bay, but simple enough that any human should be able to do it.

I’m still in two minds about this one. On the one hand it is a lot easier than a graphical captcha, but at the same time those are what people are used to these days. I’ll give it some time and see how it goes, but the problem is that you never know if someone has given up on a comment because they couldn’t figure out the captcha.

Theme Test Drive – website, wordpress directory

The theme I currently have is based on the Wordpress default and I’ve had it for a few year now. Lately I’ve been thinking about getting a new theme, but I was left with the problem of how to test them. That’s where this plugin comes in – it allows you, as an admin, to see a different theme on your site to your normal visitors. This allows for testing and development of new themes without making your site look like a mess whilst you do it.

I guess this plugin isn’t something that everyone would need, unless you’re addicted to changing your theme of course. But it’s a good one to have around, maybe disabled, until you do need it.

Plugins to help out lost visitors

My last group of plugins are those for dealing with people landing on a 404 page (this happens when the page they’re looking for doesn’t exist). If we can get the visitor to the page they’re looking for they’ll be much happier!

Useful 404’s – website, wordpress directory

This is the most useful of the 404 plugins that I have installed. It analyses what’s happened and both notifies the user of the problem and sends you an email to let you know. It distinguishes between broken internal links, broken incoming external links and out of date search results, and acts accordingly for each situation.

On its own this plugin only does half the job, but it’s still a useful part of the overall 404 solution.

Smart 404 – website, wordpress directory

This plugin sits nicely with Useful 404. It attempts to figure out possible posts the visitor could have been looking for and gives a list of those for them to choose from. It’s not perfect – its algorithms seem quite basic – but I guess it doesn’t have much information to work from.

Google 404 – wordpress directory

This plugin is fairly similar in purpose to Smart 404, but instead uses the Google 404 widget to provide visitors with useful information. However, at the moment it’s not doing anything useful for me, but I’ll give it some more time before I decide to drop it.

That’s the end of my plugin recommendations. I hope you’ve found them useful.

  • Share/Bookmark

Broken pingbacks – what’s to blame?

Tuesday, March 24th, 2009 in Computing

For a while I’ve been noticing a lack of pingbacks from people linking to me. At first I put it down to people just not linking to me, but then after discussions with a colleague today I realised I didn’t have that annoying problem of them happening when I link to my own posts. I tested further and realised they weren’t working at all.

I debugged the issue using the good old “print debug messages all over the place” method – crude but effective. I started in xmlrpc.php and discovered that it was stopping at this line:

$comment_ID = wp_new_comment($commentdata);

This meant it was getting as far as trying to post the comment. It was then that I had a lightbulb moment. What would be stopping comments being posted? My captcha to stop comment spam (has it really been three years since I set that up?), that’s what!

My colleague has been using a numerical captcha-style plugin called Math Comment Spam Protection (MCSP) instead. I did a quick comparison between it and the SecureImage plugin and fairly quickly found the difference. This is from the MCSP plugin:

if (  ( !isset($user_ID) ) && ( $comment_data['comment_type'] == '' ) ) {

// Do not check if the user is registered & do not check trackbacks/pingbacks

By comparison, the SecureImage plugin only checked if the user is registered:

// If the user is not logged in check the security code

if ( !$user_ID ) {

The fix looks simple. But, given the lack of updates for SecureImage I decided it was time to move on. So now I have the MCSP plugin instead, and pingbacks are once again working.

I just have one request to make. Can anyone who’s linked to me in the last three years please try again? :-)

  • Share/Bookmark

UK Bank Rate cut to 0.5%

Thursday, March 5th, 2009 in Finance

Today the Bank of England cut the bank rate (commonly known as the interest rate) to 0.5%, the lowest value ever recorded. They also plan to inject more money in to the system to try and get the economy going again. The governor said this policy would “eventually work”.

Now I get the reasons behind this. The economy worldwide is in a bad state at the moment and governments are forced to make tough choices to try and rectify things. But I’m selfish, and all I see is that the interest on my savings will be reduced yet again.

I’m in the position of many people these days. House prices were soaring and getting that first step on to the housing ladder was hard. So, we took the decision to rent instead of buy and to save our money for a future house deposit. This worked well when rates were high; we had no expensive mortgage to pay and our savings were working hard for us (although, inflation meant that looked better than it really was).

But now after numerous rate cuts we’re seeing no benefit, we’re just seeing our monthly interest payments dwindle. Looking back 12 months I think we were earning roughly 3 times what we are now. That’s a huge drop! Fortunately we don’t depend on the interest to live, but I feel sorry for those that do.

This has led me, and many others, to start searching for better ways to invest our money. For short to medium term maybe corporate or government bonds are a good choice? Whilst equities are likely still the best long term choice. Maybe Zopa is good for medium term as well. But these all bring risk, and now probably isn’t the time to gamble with your savings (at least not all of it).

Next month brings the start of the next tax year. We’re already starting to see some attractive cash ISA deals, so come April I’ll be hunting out the best deals for our money – I’ll be sure to post the results!

So where does this leave rates now? Hopefully this will be the lowest they can go. Is it actually possible they’ll drop to 0.25% or even 0%? Will banks start charging savers rather than paying interest? Time will tell…

  • Share/Bookmark