Archive for the ‘Computing’ Category

Increasing our storage provision

Wednesday, October 14th, 2009 in Computing, Work

During the summer we started getting tight on storage availability. It seems that usage on our home directory areas constantly increases – people never delete stuff (me included!). We were running most of our stuff through our Veritas Cluster from a pair of Sun 3511 arrays and a single 3510 array. Between them (taking mirroring in to account) we had around 3TB of space.

Now, it’s a well known fact with maintenance contracts that the cost goes up over time (parts get more scarce and more costly). So we did the sums on the cost we were paying for the old arrays and realised that over a sensible lifetime period it was cheaper to replace them. So we got a pair of Sun 2540 arrays with a 12TB capacity each.

Since our data is absolutely precious we mirror these arrays and use RAID 6. This gives us just under 10 TB of usable space, which is a fair amount more than we started with.

The next stage was bring this online. Because we use Veritas Volume Manager and the Veritas File System we were able to do this almost transparently. The new arrays were provisioned and added to the relevant diskgroups. The volumes were then mirrored on to them and then the filesystems expanded. Finally the old arrays were disconnected. All of this was done without any downtime or interruption to our users or services.

I said almost transparently though. It seems it’s not possible to change the VCS coordinator disks without taking the diskgroups offline and back online (this might be improved in VCS 5). So I rebooted the whole cluster last weekend and it was all finished.

The problem with all this clever technology? Nobody knows we’ve done it. After weeks of work we grew the filesystems just before they completely filled and without any noticable downtime. We’d probably get more gratitude if we’d let it fill up first ;-)

  • Share/Bookmark

Getting the indexes right for OpenLDAP when using NSS

Wednesday, October 14th, 2009 in Computing

I recently deployed a Linux system which used the libnss-ldap module to get its passwd and group information. This all worked fine except group lookups (in particular when logging in) which were extremely slow. We have about 600 groups in our directory, which isn’t massive, but is more than the average system.

Clearly this wasn’t right. Initially I tried nscd, which helped, but only after it had cached the data. Then I realised it was probably the indexes in OpenLDAP. Googling didn’t turn up much of use (hence this post), but I did find this page on the OpenLDAP site.

This fairly quickly pointed me at the problem; I was missing indexes on memberUid and uniqueMember. Adding these fixed the problem completely.

So here’s the indexes I’ve ended up with:

index   objectClass     eq
index   cn,uid          eq
index   uidNumber       eq
index   gidNumber       eq
index   memberUid       eq
index   uniqueMember    eq
index   entryCSN        eq
index   entryUUID       eq

(the last two are for replication)

I’m actually quite surprised how much the indexes matter. It makes a huge difference, even on a small setup. So if you’re setting up a directory take the time to read the Tuning section of OpenLDAP Admin Guide first.

  • Share/Bookmark

PAM – Locking out accounts when using external authentication

Friday, October 9th, 2009 in Computing

One of the projects I’ve been working on lately has required me to investigate the way PAM handles external authentication. The setup is a bunch of different boxes running Solaris, Linux and FreeBSD, but all authenticating from a single central Kerberos domain. There’s also LDAP in the mix, but it’s not actually relevant, so I’ll ignore it for now.

The problem comes when you want to lock out an account on a machine. By this, I mean the account still exists in the passwd file (for whatever reason, it needs to stay there) and the account still exists within the Kerberos domain because it can log on to other machines.

What are the options? The PAM setup probably looks a little something like this:

auth    sufficient pam_krb5
auth    required   pam_unix

account required   pam_krb5
account required   pam_unix

This setup allows the user to authenticate with just their Kerberos credential, but they must pass both modules in the account section (the pam_krb5 one here doesn’t do much though).

So we can put whatever we want in the shadow password entry on the machine, it won’t affect the authentication – it never even gets there. But what about the account section? Can this do anything?

After some investigation and source code checking I came up with the following:

  • On Solaris one can put *LK* in the shadow password field. This causes the pam_unix module to reject the login in the account stage.
  • On FreeBSD one can do the same as Solaris but with *LOCKED*.
  • On Linux you can put what you want in the field but pam_unix won’t take any notice.

If you’re thinking right now “hey, you can just put an exclamation mark in the shadow password field to lock the account out” you’re wrong. What this does on a standalone system is ensure that no possible password can decrypt to match this string which results in locking the account out. But, when we have external authentication it’s meaningless.

So what’s the solution? There’s another field we can use – the shadow password expiry field. This is a time stamp in seconds since the epoch at which the password expires. The default setting of 0 means no expiry. Simply setting this to 1 (or any number smaller than now) results in the account being locked out. This seems like it’ll work across all three operating systems I’m testing, although I’m yet to finish my analysis.

I am a little disappointed with the Linux PAM module. It’s a simple test to do, so I can’t see any reason why it shouldn’t be added.

Update 2nd November 2009:

I’ve since discovered that what I said above about the “default setting of 0 means no expiry” is incorrect. In fact, it varies on different operating systems. I got misled by the behaviour on Ubuntu 9.04 because of a Debian patch which made 0 mean no expiry, but which has been removed in Ubuntu 9.10 (details here).

Consequently my advice now is to set the shadow password expiry field to 1 to lock the account, and to unset it to unlock the account. This appears to work on all the operating systems I’ve tested it on.

References:

  • Share/Bookmark

Implementing SRS with Exim and SQLite

Wednesday, July 29th, 2009 in Computing, Work

Due to issues with SPF-style restrictions I decided I’d take a look at implementing SRS (the Sender Rewriting Scheme, a good description of which is over here) with Exim. I thought it’d be fairly straightforward and well documented, but it wasn’t. I’m left wondering if anybody actually does it like this?

To start with I built Exim on FreeBSD with the SPF and SRS libraries (I used libsrs_alt). This was straightforward enough – although where are the options in the FreeBSD port? – and would be standard across most operating systems.

Since this was a trial run I thought I’d take the lightweight approach and use SQLite rather than set up a full database like MySQL. This again was a straightforward install.

Lets look quickly and the SQLite tables. The instructions on the libsrs_alt site don’t talk about creating these, so I just figured it out for myself. This is what I ended up with:

# sqlite3 /var/tmp/srs.db .dump
BEGIN TRANSACTION;
CREATE TABLE SRS(Key TEXT, Address TEXT, Time INTEGER);
COMMIT;
#

That’s pretty simple. The database should probably live somewhere a bit more permanent though :-) .

Right, back to the instructions. Basically we need to add a bit of config and a new router to the exim configuration. Starting with the default configuration I added the following option in the global section:

hide srs_secrets = asecurestringofareasonablelength

And then the following routers:

srs_reverse:
  driver = redirect
  domains = +local_domains
  srs = reverseandforward
  srs_dbinsert = ${lookup sqlite{/var/tmp/srs.db \
    INSERT INTO SRS ('Key', 'Address', 'Time') \
    VALUES ('${srs_db_key}', '${srs_db_address}', \
    strftime('%s','now'))}}
  srs_dbselect = ${lookup sqlite{/var/tmp/srs.db \
    SELECT Address FROM SRS \
    WHERE Key = '${srs_db_key}' \
    AND Time > strftime('%s','now','-30 days') \
    LIMIT 1}}
  data = ${srs_recipient}
srs_forward:
  driver = redirect
  domains = +local_domains
  srs = forward
  srs_dbinsert = ${lookup sqlite{/var/tmp/srs.db \
    INSERT INTO 'SRS' ('Key', 'Address', 'Time') \
    VALUES ('${srs_db_key}', '${srs_db_address}', \
    strftime('%s','now'))}}
  data = $local_part@example.org

As you can see in this case, I’m simply forwarding all email to another domain. That’s not the most useful setup, but again I’m just testing.

Note that I had to put these in the opposite order to the instructions. The first router is looking for addresses that are already SRS encoded (so bounces for forwarded messages, etc). If it doesn’t find one, it just passes on to the next one. So this way round work best for me.

That’s it really. At a simple level this works. I’ve not looking at proper integration in to the forwarding setup or any kind of database maintenance yet. But given the lack of useful documentation online I thought I’d post my findings.

If you’re reading this and thinking “why on earth is he doing it like that?” please drop me a comment below and enlighten me :-) .

  • Share/Bookmark

Wordpress plugin recommendations

Friday, March 27th, 2009 in Computing

I’ve been using Wordpress for a few years and I’m now using quite a few plugins. Since I made use of other’s blog posts to find them I thought I’d return the favour and list the ones I’m using. So, here they are:

Plugins to make life easier for visitors

Visitors are the main reason blogs exist. If nobody is looking at your blog why bother writing it? So here’s a few plugins that make things a little easier for your visitors.

Yet Another Related Posts Plugin – website, wordpress directory

This plugins adds a list of related posts to the end of each post. It’s almost magical in how it works; it somehow just does the “right thing” without any configuration or input from you. It also adds related posts to your RSS feed, which is a great way to pull people back in to your site.

Since I’ve been adding excerpts to my posts lately I’ve configured this plugin to display excerpts in their entirety. I suppose if I write many more posts I might want to shorten them.

Subscribe To Comments – website, wordpress directory

Do you find it annoying having to check back on blogs you’ve commented on to see if others have commented too? If so, this plugin is almost for you. It will allow visitors to subscribe to comments being added to a post on your blog; when someone else comments they’ll receive an email update.

Sadly it won’t do anything to make your life easier visiting other blogs, but the more people that take it up the better. In fact, I’d go as far as to say this should be integrated in to Wordpress itself.

Top 10 – website, wordpress directory

This is a handy little plugin that’ll keep track of visits to your site and allow you to display a list of the top 10 visited posts. It’s pretty basic but does the job. I’d like to see it have an expiry on it so you could list the top 10 posts over the last 30 days, for example (this is already possible – see the first comment below).

I did try and use StatPress Reloaded (see below) to do this, but its listing was pretty naff (didn’t display post titles) and it worked on a URL basis rather than a post basis.

Add to Any: Share/Save/Bookmark Button – website, wordpress directory

There’s a whole load of sites out there for bookmarking and discussing other sites, so it makes sense to allow readers to quickly link to your blog posts. This plugin does just that – it adds a button to the end of each post which allows visitors to share, bookmark and email your posts to all sorts of services and people.

In addition, there’s a Subscribe Button plugin, which I don’t use, that allows readers to quickly subscribe to your blog in a multitude of readers.

Search Engine Optimisation

To get visitors your blog needs to be found. As well as the automatic pings that Wordpress sends out we need to make sure our blogs are easily found and indexed by search engines. These plugins help out with that process.

All in One SEO Pack – website, wordpress directory

This is the most popular plugin in the wordpress directory, and with good reason. It does various small tweaks to your site such as page titles and metadata, which in my experience do have a positive effect on the number of visitors. It’s surprising to see how little changes can make such a difference, but I won’t complain!

The nice thing about this plugin is that it’s all set to go after installation. You don’t even need to do anything to configure it, although it has plenty of options if you want to.

I highly recommend this plugin to all Wordpress users.

Google XML Sitemaps – website, wordpress directory

I don’t have much experience with this plugin, or Google sitemaps for that matter, but given Google’s dominance of the search market it makes sense to play along with it. This plugin creates a map of your site (actually just a list of posts, categories, etc) and submits it to Google. In theory this allows Google to properly index your site, but I’d have thought it’d do a pretty good job of that on it’s own.

It’s pretty straight-forward to set up, although if you have your blog under a sub-directory you might want to add a few static entries for the rest of your site. Then fire up the Google webmaster tools and submit the URL.

It turns out Google can also use your RSS feeds as sitemaps, so I did that as well.

Plugins to make things better for you!

We’ve looked at plugins to make life easier for your visitors and to bring more visitors to your site, but what about you? I’ve haven’t forgotten about the hard working blogger behind the site. Here’s a few plugins to help you out.

StatPress Reloaded – website, wordpress directory

This is an alternative version of the StatPress plugin. It gathers statistics about visitors including which posts they visit, where they came from, and what search terms they used to get there. It’s all information that’s available through Google Analytics (I’ve added that to my template, although there is a plugin to do it for you), but it’s nice to have it easily accessible within your admin area.

The information it provides is pretty interesting. I’ve found a few sites talking about my stuff and linking to me that I wouldn’t have found if the stats weren’t so readily available. I recommend giving it a try.

All in One Adsense and YPN – website, wordpress directory

Pretty much every personal site you visit these days has some form of advertising on it. Most overdo it, but done subtly it can be unobtrusive and provide an income stream. Don’t get me wrong, blogging isn’t a good way to make money – I certainly don’t make much – you have to put in a lot of work before you get anything worthwhile out of it. But if you enjoy writing blog posts why not make a little out of it?

This plugin makes it easy to insert adverts within the text of posts. In fact, there’s probably one somewhere within this post. It’s fully configurable and does most of the hard work for you. However, you will need an Adsense or YPN (which I don’t use) account to do it.

I have had some reservations about the way the donation code works, but it turned out to be just badly written code. Still, it gives me reason to not feel completely confident with this plugin, but until I find an alternative I’ll stick with it.

Math Comment Spam Protection – website, wordpress directory

Up until recently I had a graphical captcha plugin to help weed out spammers’ comments. After the problems I had with it I decided to switch to MCSP instead. It’s a simple plugin – it gives readers a simple mathematical sum to solve when posting a comment. That’s enough to keep the spammers at bay, but simple enough that any human should be able to do it.

I’m still in two minds about this one. On the one hand it is a lot easier than a graphical captcha, but at the same time those are what people are used to these days. I’ll give it some time and see how it goes, but the problem is that you never know if someone has given up on a comment because they couldn’t figure out the captcha.

Theme Test Drive – website, wordpress directory

The theme I currently have is based on the Wordpress default and I’ve had it for a few year now. Lately I’ve been thinking about getting a new theme, but I was left with the problem of how to test them. That’s where this plugin comes in – it allows you, as an admin, to see a different theme on your site to your normal visitors. This allows for testing and development of new themes without making your site look like a mess whilst you do it.

I guess this plugin isn’t something that everyone would need, unless you’re addicted to changing your theme of course. But it’s a good one to have around, maybe disabled, until you do need it.

Plugins to help out lost visitors

My last group of plugins are those for dealing with people landing on a 404 page (this happens when the page they’re looking for doesn’t exist). If we can get the visitor to the page they’re looking for they’ll be much happier!

Useful 404’s – website, wordpress directory

This is the most useful of the 404 plugins that I have installed. It analyses what’s happened and both notifies the user of the problem and sends you an email to let you know. It distinguishes between broken internal links, broken incoming external links and out of date search results, and acts accordingly for each situation.

On its own this plugin only does half the job, but it’s still a useful part of the overall 404 solution.

Smart 404 – website, wordpress directory

This plugin sits nicely with Useful 404. It attempts to figure out possible posts the visitor could have been looking for and gives a list of those for them to choose from. It’s not perfect – its algorithms seem quite basic – but I guess it doesn’t have much information to work from.

Google 404 – wordpress directory

This plugin is fairly similar in purpose to Smart 404, but instead uses the Google 404 widget to provide visitors with useful information. However, at the moment it’s not doing anything useful for me, but I’ll give it some more time before I decide to drop it.

That’s the end of my plugin recommendations. I hope you’ve found them useful.

  • Share/Bookmark

Broken pingbacks – what’s to blame?

Tuesday, March 24th, 2009 in Computing

For a while I’ve been noticing a lack of pingbacks from people linking to me. At first I put it down to people just not linking to me, but then after discussions with a colleague today I realised I didn’t have that annoying problem of them happening when I link to my own posts. I tested further and realised they weren’t working at all.

I debugged the issue using the good old “print debug messages all over the place” method – crude but effective. I started in xmlrpc.php and discovered that it was stopping at this line:

$comment_ID = wp_new_comment($commentdata);

This meant it was getting as far as trying to post the comment. It was then that I had a lightbulb moment. What would be stopping comments being posted? My captcha to stop comment spam (has it really been three years since I set that up?), that’s what!

My colleague has been using a numerical captcha-style plugin called Math Comment Spam Protection (MCSP) instead. I did a quick comparison between it and the SecureImage plugin and fairly quickly found the difference. This is from the MCSP plugin:

if (  ( !isset($user_ID) ) && ( $comment_data['comment_type'] == '' ) ) {

// Do not check if the user is registered & do not check trackbacks/pingbacks

By comparison, the SecureImage plugin only checked if the user is registered:

// If the user is not logged in check the security code

if ( !$user_ID ) {

The fix looks simple. But, given the lack of updates for SecureImage I decided it was time to move on. So now I have the MCSP plugin instead, and pingbacks are once again working.

I just have one request to make. Can anyone who’s linked to me in the last three years please try again? :-)

  • Share/Bookmark

Automating tarsnap backups

Wednesday, January 28th, 2009 in Computing, FreeBSD

In my last post I wrote about backing up my dedicated server and why I decided to use tarsnap. After a couple of months of running tarsnap manually I decided it was way past the time to properly automate it.

The main issue is how many snapshots do you want to store? On the one hand it’s nice to be able to go back in time as far as possible, but on the other hand there’s the issue of how large your archives get (and consequently the cost).

There are three different charges for tarsnap; data sent, data received and data stored. Each is charged on a daily basis and subtracted from a total in your account (you keep an account in credit rather than being billed). If you’re doing backups on a daily basis the data sent and received will be approximately the same regardless of how long you retain the archives for. So the figure to consider is the cost for storing the data.

I decided to go for a model where I had X daily backups, Y weekly backups and Z monthly backups. I also decided I wanted to back up only certain directories, and that I wanted to keep them as separate archives (because I’m dealing with large numbers of files, and this breaks it down a bit – I don’t think it affects costs).

So I went about scripting this. First step was to write a “fake” tarsnap. The reasoning behind this was that it’d allow me to do quick backup runs without any time used for archiving or any costs. It’s basically just a perl script that adds and removes archives from a database file.

Next I wrote a backup script. It’s pretty basic at the moment, but fully automates the creation of archives and deletion of expired ones. You provide it with a list of directories to back up, and how many daily, weekly and monthly archives you want to keep. Then stick it in cron and off it goes.

It’s a bit tailored to my setup, and may only work on FreeBSD (are the date flags the same on other operating systems?). Also, its cleaning of old archives is primitive; it’s based on the number of archives, rather than the age.

I welcome feedback on these scripts and improvements, but bear in mind they’re very much a work in progress.

  • Share/Bookmark

Machine backups using tarsnap

Wednesday, January 28th, 2009 in Computing, FreeBSD

I’ve got a dedicated server that I’ve been backing up for the past few years. My crude backup system involved taring everything to local disk and then rsyncing it to a remote server. It worked well at first, but as the amount of data grew it was taking half a day to run. Add to that the amount of disk space being used by the local copy and I had to find a better solution.

I started off by looking at replacements for rsyncing to my remote server. rdiff-backup seemed an obvious choice, but some friends had previously had problems with it, so I decided to give it a miss. There were various other similar tools that involved copying my data to another server (of mine), possibly with incremental features, but I decided to expand my search criteria and came across online backup services.

These services involve paying someone else to store your data remotely. In principal there didn’t seem to be much point; I had somewhere else to put my data, so why pay for it? One solution that seemed quite nice was brackup – a set of perl scripts that backed up to Amazon S3. Still not quite what I wanted though.

Then, I remembered a project Colin Percival was working on – tarsnap. It’d be a while since I last checked up on progress so I was quite pleased to see that he’d just launched the public beta. I signed up and handed over some money. What made me decide on this solution over others? Well,

1. It works pretty much like tar, except the archives are remotely stored. So that made it simple to use and fit it to my way of working.

2. It has all the benefits of snapshots with all the benefits of incremental backups. By this I mean that you can access each archive as if it was a full copy, but behind the scenes it only transmits and stores the differences (and charges you accordingly). You can delete any archive in the set without worrying about things like needing all the incrementals to the last full backup, and it sorts it all out for you.

3. Colin is a FreeBSD guy, so that pretty much guarantees it’s going to work well on FreeBSD and be supported in the future.

4. After doing the sums I realised the charges are pretty insignificant compared with the cost of running my own dedicated server.

5. There’s a whole load of stuff about security. It doesn’t particularly interest me though, but it’s nice to know my data is safe (nobody can read it – Colin included – without my key).

So I’ve been using tarsnap now for a few months and on the whole I’ve been very happy. There was a minor issue with the large number of files (Maildir email storage can get insane :-) ) I had to backup, but once I discovered the --lowmem flag all was fine. And Colin’s support has been great – he’s always been quick to answer if I’ve had a question or problem.

If you’re looking for an online backup service that’s geared towards Unix systems I highly recommend giving tarsnap a try.

(How about reading my post on Automating Tarsnap Backups?)

  • Share/Bookmark

Extracting album art from iTunes

Wednesday, December 17th, 2008 in Computing

My music collection is primarily played using SqueezeCenter. But, I also have iTunes available for syncing to my iPod which I use in the car. One thing iTunes does fairly well is fetching album artwork, and I’ve been looking for a way of getting this in to SqueezeCenter without embedding the images in the MP3 files.

I found this script (there’s a download link towards the bottom) which does this job perfectly. It runs through my library and saves out all the iTunes artwork in a single file per album called cover.jpg. It’s written in javascript, so it’s not at all pretty. You just run it and wait until the dialogue appears saying it’s finished.

One gotcha I found was that I had to do a complete rescan of SqueezeCenter to make it notice that the album artwork had appeared. It was sporadic before that. Caching maybe?

That’s one problem I can tick off with a nice easy solution. :-)

  • Share/Bookmark

Exhange 2007 OOF (Out Of Office) and Free/Busy Availability, fixed

Tuesday, November 18th, 2008 in Computing

I’ve spent the past couple of weeks on and off trying to figure out why the OOF (Out Of Office) and Free/Busy availability information were broken in our Outlook 2007 clients. They generated errors saying the service was not available. They worked fine in OWA.

After some digging I thought I’d try some of the EWS URLs manually in a web browser. I’d tried some of the others and although they produced odd things, they at least did something. The EWS URLs just produced a 404.

Looking in IIS there were clearly files in the EWS directory with the correct names, so something else must have been missing. I started a comparison with another reference system. Eventually I noticed a difference in the web.config file located in C:\Program Files\Microsoft\Exchange Server\ClientAccess\exchweb\ews.

The copy on our production system contained entries like this:

<codeBase version=”0.0.0.0″ href=”file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Common.IL.dll” />

Whilst our reference system contained entries like this:

<codeBase version=”0.0.0.0″ href=”file:///C:\Program Files\Microsoft\Exchange Server\bin\Microsoft.Exchange.Common.IL.dll” />

As far as I could tell %ExchangeInstallDir% was not set on either system. Further, looking at the autodiscover and OWA web.config files I noticed they did it the same way as the reference system.

So I ducked over to a Unix system, did a proper comparison of the production file versus the reference one and determined that the only difference was the expansion of the variable. So I simply dropped the reference one in to place on the production system and restarted IIS.

And it worked!

I’m not sure how we got in to that state, but I’m pleased it’s sorted out. I’m not particularly getting on with this Exchange stuff, and peculiar issues like this really don’t help. Given we installed both the production and reference systems in the same way I can’t understand how this happened.

Now to fix the remaining issues… :-(

Update: Now that I know what the issue was I can google for it. I found this post (better formatted here for non-IE users) which shows that the issue occured with the accidently released update for Exchange a while back. We got this update in the small window in which it was available and later uninstalled it. Looks like that caused the problem.

  • Share/Bookmark