CSLab Support: Disk Space

Understanding Disk Space at CSLab

At CSLab, your data is stored in filesystems, such as the filesystem your home directory is in (which will be /h/NNN for some number), the web directory filesystem your web pages are in (which start with /w), and workdir filesystems (which also start with /w). Most filesystems are part of a specific shared pool of disk space that is arranged by your professor or your research group. Every filesystem in the pool of shared space can draw on that shared space, up to a configured size limit for each filesystem. On the one hand, this allows the available space to be flexibly used where it's needed and useful, without people having to decide in advance how much space to give to each filesystem. On the other hand, this means that someone using disk space in another filesystem in your pool of shared space can cause the filesystem you're in to run out of disk space too, because both filesystems are drawing on the same pool of free space.

If your home directory filesystem runs out of space for any reason, this will almost always be reported to you as an error message about hitting a quota limit (this happens for technical reasons). This is also the case for other filesystems not having any available space left. One of the things stored in your home directory is your IMAP mail folders, so if there's no space left for your home directory filesystem you won't be able to refile messages or save email messages you send to your Sent folder, and your mail client will probably report that the IMAP server said you hit a quota limit.

On CSLab's servers, such as our general login servers, the df command (for example, 'df -h .') will accurately tell you how much disk space a filesystem is currently using and how much more it could use right now if nobody else used any more space from the pool of shared disk space it draws on. For example:

Filesystem                Size  Used Avail Use% Mounted on
fs1.cs.toronto.edu:/h/81  416G  138G  278G  34% /h/81

This filesystem is using 138 GBytes of space and could use another 278 GBytes of space if no other filesystem used it. You can ignore the 'Size' field (and the related Use% field); it's always the sum of Used and Avail.

You can see all of the pools of disk space in the special directory /cs/pools, and if you use df on one of them, the Avail number will tell you how much free space is still available for general use by filesystems in that pool of disk space. For example:

$ df -h /cs/pools/fs1-corestaff-01
Filesystem                            Size  Used Avail Use% Mounted on
fs1.cs.toronto.edu:/fs1-corestaff-01  278G     0  278G   0% /cs/pools/fs1-corestaff-01

This pool of disk space has 278 GBytes free in total. It's not a coincidence that both examples show 278 GBytes free; /h/81 is part of the fs1-corestaff-01 pool of disk space, so at the moment the amount of free space /h/81 can use is being limited by the amount of free space left in the pool of disk space.

If your home directory filesystem or one of the work directory filesystems you use is running short of space, you can see who is using the space (and recent changes in that space usage) either through services.cs or with the hogs command, which is located in /local/bin and is normally on your $PATH. This command attempts to work out whether you're being limited by the free space available in the shared pool of disk space or by the size limit of the filesystem, and show you the usage of the relevant one. If you're sure you want to see who is using space across the shared pool of disk space you're part of, you can invoke hogs with a -p argument, 'hogs -p'. If you need more information about current and past space usage, you can get in touch with your Point of Contact.

If you prefer not to log in to one of our servers, you can also get this information through services.cs, using the "Disk usage for your home directory" and "Your disk usage everywhere" options. The latter uses /local/bin/myhogs to show you where you're using disk space and recent changes in that.

If you and other people in your group are persistently running short of necessary space, it is possible for your professor or research group to arrange for more space. Contact your Point of Contact for more details.

(It's possible for a filesystem to have more available space than its pool of disk space; space can be specifically reserved for a filesystem, which removes the reserved space from the pool's shared space. It's also possible for a filesystem to have less available space than the pool's shared space, because filesystems have size limits, as mentioned.)

Technical details

All of CSLab's homedir and workdir filesystems are on our NFS fileservers, where we use ZFS. What we're calling "pools of disk space" are ZFS pools, which have the cryptic names you see in /cs/pools. Every regular homedir and workdir filesystem has some ZFS filesystem quota limit that caps its maximum size, and it may have a ZFS filesystem reservation to reserve some amount of space for it. Each group or professor has arranged a certain amount of space for their ZFS pools, and that is the total amount of disk space available in that pool.

CSLab doesn't normally use ZFS snapshots, because they will use up some of the limited disk space in ZFS pools and CSLab Core normally assumes that people want to use that space for their current files. The owners of a ZFS pool can decide that they are willing to accept the space overheads of snapshots and have them enabled on some or all of the pool's filesystems. Pool owners can contact their Point of Contact for details.