Archive for October, 2009

Increasing our storage provision

Wednesday, October 14th, 2009 in Computing, Work

During the summer we started getting tight on storage availability. It seems that usage on our home directory areas constantly increases – people never delete stuff (me included!). We were running most of our stuff through our Veritas Cluster from a pair of Sun 3511 arrays and a single 3510 array. Between them (taking mirroring in to account) we had around 3TB of space.

Now, it’s a well known fact with maintenance contracts that the cost goes up over time (parts get more scarce and more costly). So we did the sums on the cost we were paying for the old arrays and realised that over a sensible lifetime period it was cheaper to replace them. So we got a pair of Sun 2540 arrays with a 12TB capacity each.

Since our data is absolutely precious we mirror these arrays and use RAID 6. This gives us just under 10 TB of usable space, which is a fair amount more than we started with.

The next stage was bring this online. Because we use Veritas Volume Manager and the Veritas File System we were able to do this almost transparently. The new arrays were provisioned and added to the relevant diskgroups. The volumes were then mirrored on to them and then the filesystems expanded. Finally the old arrays were disconnected. All of this was done without any downtime or interruption to our users or services.

I said almost transparently though. It seems it’s not possible to change the VCS coordinator disks without taking the diskgroups offline and back online (this might be improved in VCS 5). So I rebooted the whole cluster last weekend and it was all finished.

The problem with all this clever technology? Nobody knows we’ve done it. After weeks of work we grew the filesystems just before they completely filled and without any noticable downtime. We’d probably get more gratitude if we’d let it fill up first ;-)

  • Share/Bookmark

Posted in Computing, Work | No Comments »

Getting the indexes right for OpenLDAP when using NSS

Wednesday, October 14th, 2009 in Computing

I recently deployed a Linux system which used the libnss-ldap module to get its passwd and group information. This all worked fine except group lookups (in particular when logging in) which were extremely slow. We have about 600 groups in our directory, which isn’t massive, but is more than the average system.

Clearly this wasn’t right. Initially I tried nscd, which helped, but only after it had cached the data. Then I realised it was probably the indexes in OpenLDAP. Googling didn’t turn up much of use (hence this post), but I did find this page on the OpenLDAP site.

This fairly quickly pointed me at the problem; I was missing indexes on memberUid and uniqueMember. Adding these fixed the problem completely.

So here’s the indexes I’ve ended up with:

index   objectClass     eq
index   cn,uid          eq
index   uidNumber       eq
index   gidNumber       eq
index   memberUid       eq
index   uniqueMember    eq
index   entryCSN        eq
index   entryUUID       eq

(the last two are for replication)

I’m actually quite surprised how much the indexes matter. It makes a huge difference, even on a small setup. So if you’re setting up a directory take the time to read the Tuning section of OpenLDAP Admin Guide first.

  • Share/Bookmark

Tags: , , , , , ,
Posted in Computing | No Comments »

PAM – Locking out accounts when using external authentication

Friday, October 9th, 2009 in Computing

One of the projects I’ve been working on lately has required me to investigate the way PAM handles external authentication. The setup is a bunch of different boxes running Solaris, Linux and FreeBSD, but all authenticating from a single central Kerberos domain. There’s also LDAP in the mix, but it’s not actually relevant, so I’ll ignore it for now.

The problem comes when you want to lock out an account on a machine. By this, I mean the account still exists in the passwd file (for whatever reason, it needs to stay there) and the account still exists within the Kerberos domain because it can log on to other machines.

What are the options? The PAM setup probably looks a little something like this:

auth    sufficient pam_krb5
auth    required   pam_unix

account required   pam_krb5
account required   pam_unix

This setup allows the user to authenticate with just their Kerberos credential, but they must pass both modules in the account section (the pam_krb5 one here doesn’t do much though).

So we can put whatever we want in the shadow password entry on the machine, it won’t affect the authentication – it never even gets there. But what about the account section? Can this do anything?

After some investigation and source code checking I came up with the following:

  • On Solaris one can put *LK* in the shadow password field. This causes the pam_unix module to reject the login in the account stage.
  • On FreeBSD one can do the same as Solaris but with *LOCKED*.
  • On Linux you can put what you want in the field but pam_unix won’t take any notice.

If you’re thinking right now “hey, you can just put an exclamation mark in the shadow password field to lock the account out” you’re wrong. What this does on a standalone system is ensure that no possible password can decrypt to match this string which results in locking the account out. But, when we have external authentication it’s meaningless.

So what’s the solution? There’s another field we can use – the shadow password expiry field. This is a time stamp in seconds since the epoch at which the password expires. The default setting of 0 means no expiry. Simply setting this to 1 (or any number smaller than now) results in the account being locked out. This seems like it’ll work across all three operating systems I’m testing, although I’m yet to finish my analysis.

I am a little disappointed with the Linux PAM module. It’s a simple test to do, so I can’t see any reason why it shouldn’t be added.

Update 2nd November 2009:

I’ve since discovered that what I said above about the “default setting of 0 means no expiry” is incorrect. In fact, it varies on different operating systems. I got misled by the behaviour on Ubuntu 9.04 because of a Debian patch which made 0 mean no expiry, but which has been removed in Ubuntu 9.10 (details here).

Consequently my advice now is to set the shadow password expiry field to 1 to lock the account, and to unset it to unlock the account. This appears to work on all the operating systems I’ve tested it on.

References:

  • Share/Bookmark

Tags: , ,
Posted in Computing | 2 Comments »