Today we saw a strange problem on our Solaris hosts that NFS mount VxFS filestore from our Veritas cluster. The users were seeing “Disc quota exceeded” messages, whilst the quota command wasn’t showing they’d hit their limit. After some digging on the cluster node we found the following error message:
Sep 12 11:04:33 bes vxfs: [ID 702911 kern.warning] WARNING: msgcnt 10 mesg 089: V-2-89: quotas on /cluster/ResFS invalid; disk usage for group id 2805 exceeds 2147483646 sectors
Ah-ha! Group quota! We hadn’t even set group quotas, but it appears the system tracks the usage anyway when you mount with -o quota. Some googling turned up the following document:
http://seer.entsupport.symantec.com/docs/277535.htm
So it turns out there’s a 1TB maximum limit when using quotas. Since we weren’t using group quotas the simple option was to disable them:
vxquotaoff -gv /cluster/ResFS
Then I edited the Mount resource and changed the quota mount option to usrquota.
This only alleviates the problem for a while. Eventually someone will need to use 1TB of storage for themselves, but hopefully that’s a little way off yet. Maybe we’ll be using ZFS by then anyway 
Related posts:
- Impending doom (for our filesystems, anyway) Over the past year or so the space usage on our research and web filesystems has pretty much doubled to the point where we’re dangerously close to running out of space. There’s currently about 1TiB of filestore available of which less than 10% remains unused. Teaching filestore, however, has barely grown at all during the last [...]...
- Now what? It’s too scary to use… Its been months in the making, but it’s finally done. We have our new filestore ready to go. There’s still plenty to do, like rolling it out for the teaching machines and web filestore, but at least we’ve got the main part done. So why has it taken so long? I spent a long time researching [...]...
- “Any idea WTF is going on?” “Any idea WTF is going on?” is what I read on my phone as I stumbled out of bed this morning. It was from one of my colleagues who, for some reason I can’t understand, seems to like getting in to work at a ridiculous hour in the morning. Still half asleep I plodded through to [...]...
- Increasing our storage provision During the summer we started getting tight on storage availability. It seems that usage on our home directory areas constantly increases – people never delete stuff (me included!). We were running most of our stuff through our Veritas Cluster from a pair of Sun 3511 arrays and a single 3510 array. Between them (taking mirroring [...]...
- NFS+IPsec Performance We’ve recently moved to having our filestore NFS exported from a cluster. This provides almost complete resilience from hardware failures, and moves us away from depending on individual end-user systems with locally attached filestore. Given the inherent insecurities with NFS we opted to use IPsec authentication (but not encryption) between the hosts involved. The NFS server [...]...