- What Disks to Use
- Disk Quotas
- Disk Configuration
- Disk Usage Monitoring
- NetApp Snapshots: How to Recover Old or Deleted Files
- SSD and Local Disk Space
(this is still a draft, some items need to be fixed)
1. What Disks to Use
All the useful disk space available on the cluster is mounted off a dedicated device (aka appliance or server), a NetApp filer.
The available disk space is divided in several area (aka partitions):
- a small partition for basic configuration files and small storage, the
/home
partition,
- a set of medium size partitions, one for SAO users, one for non-SAO users, the
/data
partitions, - a set of large partitions, one for SAO users, one for non-SAO users, the
/pool
partitions, - a second set of large partitionsfor temporary storage, the
/scratch
partitions.
Note that:
- we impose quotas: limit on how much can be stored on each partition by each user, and
- we monitor disk usage;
/home
should not be used to keep large files, use/pool
instead;/scratch
is for temporary storage (i.e., while a job is running) if you need more than what you can store under/pool
.
We will implement an automatic scrubber: old stuff will be deleted to make space, using a FIFO model.- None of the disks on the cluster are for long term storage, please copy your results back to your "home" computer and
delete what you don't need any longer. - Once you reach your quota you won't be able to write anything on that partition until you delete stuff.
- We are in the process of adding local SSDs (solid state disks) to some compute nodes (not all - stay tuned), and
for special cases it may be OK to use disk space local to the compute node.
Contact us if your jobs can benefit for either SSDs or local disk space (missing link).
2. Disk Quotas
To prevent the disks to fill up and hose the cluster, there is a limit (aka quota) on
- how much disk space and
- how many files (in fact "
inodes
": the sum of number of files and number of directories)
each user can keep.
Each quota type has a soft limit (warning) and a hard limit (error) and is specific to each partition. In other words exceeding the soft limit produces warnings; while exceeding the hard limit is not allowed, and results in errors.
3. Disk Configuration
Maximum | Quotas per user | NetApp | |||
---|---|---|---|---|---|
disk | disk space |
| snapshots | ||
Disk name | capacity | soft/hard | soft/hard | enabled? | Purpose |
|
| 50/ | 1.8/2M |
| For your basic configuration files, scripts and job files - your limit is low but you can recover old stuff up to 4 weeks. |
|
|
| 1/2M |
| For important but relatively small files like final results, etc. - your limit is medium, you can recover old stuff, but disk space is not released right away. For SAO users. |
|
| 1.0/2.0TB | 1/2M |
| For important but relatively small files like final results, etc. - your limit is medium, you can recover old stuff, but disk space is not released right away. For non-SAO users. |
/ |
|
| 4/5M |
| For the bulk of your storage - your limit is high, and disk space is released right away, for SAO users. |
|
| 1.8/2.0TB | 1.8/2M |
| For the bulk of your storage - your limit is high, and disk space is released right away, for non-SAO users. |
|
|
| 1/1M |
| For temporary storage, if you need more than what you can keep in - SAO/non-SAO user should use The FIFO model (first in first out) purging has yet to be implemented as we tune the system. |
Notes
The notation
- 2.8/3.0TB means that the soft limit is 2.8TB and the hard limit is 3.0TB of disk space, while
- 4/5M means that the soft limit is 4 millions
inodes
and the hard limit is 5 millions.
It is inefficient to store a slew of small files and if you do you may reach your inodes
quota before you space quota.
While some of the tool(s) you use may force you to be inefficient while jobs are running, you should remember to
- remove useless files when jobs have completed,
- compress file that can benefit from compression, and
- archive a slew of file into a tar-ball or a zip, as follow
% zip archive.zip dir/
or% tar czf archibe.tgz dir/
both examples archive the content of the directorydir/
into asingle zi
p or atgz
file. You can then delete the content ofdir/
with% rm -rf dir/
- You can unpack each type of archive with
% unzip archive.zip
or% tar xf archive.tgz
The sizes of the partitions (aka the various disks) on the NetApp will "auto-grow" until they reach the listed maximum capacity, so the size shown by the traditional Un*x command, like df
does necessarily not reflect the maximum size.
Once we secure more space for /scratch
, we will implement a FIFO (first in first out) model, where old files are deleted without warning to make space.
- There will be a minimum age limit, meaning that only files older that (let's say) 3 months will be deleted.
- Older files will be deleted before the newer ones (FIFO),
- We will run a scrubber an a regular interval
4. Disk Monitoring
The following tools can be used to monitor your disk usage.
You can use the following Un*x commands:
du | show disk use |
df | show disk free |
or you can use Hydra-specific home-grown tools, (these require that you load the tool/local-hpc
module)
dus-report.pl | run du and parse its output in a more user friendly format |
disk-usage.pl | run df and parse its output in a more user friendly format |
Disk usage
The output of du
can be very long and confusing. It is best used with the option "-hs
" to show the sum ("-s
") and to print it in a human readable format ("-h
").
If there is a lot of files/directory, du
can take a while to complete.
For example:
% du -sh dir/ 136M dir/
The output of df
can be very long and confusing.
You can use it to query a specific partition and get the output in a human readable format ("-h
"), for example:
% df -h /pool/sao Filesystem Size Used Avail Use% Mounted on 10.61.10.1:/vol_sao 20T 15T 5.1T 75% /pool/sao
You can compile the output of du
into a more useful report with the dus-report.pl
tool. This tool will run du
for you (can take a while) and parse its output to produce a more concise/useful report.
For example, to see the directories that hold the most stuff in /pool/sao/hpc
:
% dus-report.pl /pool/sao/hpc 612.372 GB /pool/sao/hpc capac. 20.000 TB (75% full), avail. 5.088 TB 447.026 GB 73.00 % /pool/sao/hpc/rtdc 308.076 GB 50.31 % /pool/sao/hpc/rtdc/v4.4.0 138.950 GB 22.69 % /pool/sao/hpc/rtdc/vX 137.051 GB 22.38 % /pool/sao/hpc/rtdc/vX/M100-test-oob-2 120.198 GB 19.63 % /pool/sao/hpc/rtdc/v4.4.0/test2 120.198 GB 19.63 % /pool/sao/hpc/rtdc/v4.4.0/test2-2-9 83.229 GB 13.59 % /pool/sao/hpc/c7 83.229 GB 13.59 % /pool/sao/hpc/c7/hpc 65.280 GB 10.66 % /pool/sao/hpc/sw 64.235 GB 10.49 % /pool/sao/hpc/rtdc/v4.4.0/test1 49.594 GB 8.10 % /pool/sao/hpc/sw/intel-cluster-studio 46.851 GB 7.65 % /pool/sao/hpc/rtdc/vX/M100-test-oob-2/X54.ms 46.851 GB 7.65 % /pool/sao/hpc/rtdc/vX/M100-test-oob-2/X54.ms/SUBMSS 43.047 GB 7.03 % /pool/sao/hpc/rtdc/vX/M100-test-oob-2/X220.ms 43.047 GB 7.03 % /pool/sao/hpc/rtdc/vX/M100-test-oob-2/X220.ms/SUBMSS 42.261 GB 6.90 % /pool/sao/hpc/c7/hpc/sw 36.409 GB 5.95 % /pool/sao/hpc/c7/hpc/tests 30.965 GB 5.06 % /pool/sao/hpc/c7/hpc/sw/intel-cluster-studio 23.576 GB 3.85 % /pool/sao/hpc/rtdc/v4.4.0/test2/X54.ms 23.576 GB 3.85 % /pool/sao/hpc/rtdc/v4.4.0/test2-2-9/X54.ms 23.576 GB 3.85 % /pool/sao/hpc/rtdc/v4.4.0/test2/X54.ms/SUBMSS 23.576 GB 3.85 % /pool/sao/hpc/rtdc/v4.4.0/test2-2-9/X54.ms/SUBMSS 22.931 GB 3.74 % /pool/sao/hpc/rtdc/v4.4.0/test2/X220.ms 22.931 GB 3.74 % /pool/sao/hpc/rtdc/v4.4.0/test2-2-9/X220.ms report in /tmp/dus.pool.sao.hpc.hpc
You can rerun dus-report.pl
with different options on the same intermediate file, like
% dus-report.pl -n 999 -pc 1 /tmp/dus.pool.sao.hpc.hpc
to get a different report, to see the list down to 1%. Use
% dus-report.pl -help
to see how else you can use it.
The tool disk-usage.pl
runs df
and presents its output in a more friendly format:
% disk-usage.pl Filesystem Size Used Avail Capacity Mounted on NetApp.1:/vol_home/hpc 2.70T 1.02T 1.68T 38%/17% /home/hpc NetApp.1:/vol_sao 20.00T 14.91T 5.09T 75%/34% /pool/sao NetApp.1:/vol_sao_atmos 18.00T 13.80T 4.20T 77%/7% /pool/sao_atmos NetApp.1:/vol_sao_rtdc 1.00T 167.51G 856.49G 17%/1% /pool/sao_rtdc NetApp.1:/vol_genomics 20.00T 17.12T 2.88T 86%/57% /pool/genomics NetApp.1:/vol_data_genomics 1.90T 10.78G 1.89T 1%/3% /data/genomics NetApp.1:/vol_data/sao 3.60T 3.27T 336.17G 91%/37% /data/sao NetApp.1:/vol_data/admin 3.60T 3.27T 336.17G 91%/37% /data/admin NetApp.1:/vol_old_pools/cluster0 17.00T 15.60T 1.40T 92%/22% /pool/cluster0 NetApp.1:/vol_old_pools/cluster1 17.00T 15.60T 1.40T 92%/22% /pool/cluster1 NetApp.1:/vol_old_pools/cluster2 17.00T 15.60T 1.40T 92%/22% /pool/cluster2 NetApp.1:/vol_old_pools/cluster3 17.00T 15.60T 1.40T 92%/22% /pool/cluster3 NetApp.1:/vol_old_pools/cluster4 17.00T 15.60T 1.40T 92%/22% /pool/cluster4 NetApp.1:/vol_old_pools/cluster5 17.00T 15.60T 1.40T 92%/22% /pool/cluster5 NetApp.1:/vol_old_pools/cluster7 17.00T 15.60T 1.40T 92%/22% /pool/cluster7 NetApp.1:/vol_sylvain 14.18T 12.75T 1.43T 90%/51% /pool/sylvain NetApp.1:/vol_scratch/genomics 25.00T 19.53T 5.47T 79%/5% /scratch/genomics NetApp.1:/vol_scratch/sao 25.00T 19.53T 5.47T 79%/5% /scratch/sao NetApp.5:/vol/a2v1/genomics01 20.00T 0.05G 20.00T 1%/1% /scratch/genomics01 NetApp.5:/vol/a2v1/sao01 20.00T 0.05G 20.00T 1%/1% /scratch/sao01
Use
% disk-usage.pl -help
to see how else to use it. You can, for instance, get the disk quotas and the max size with:
% disk-usage.pl -quotas quotas: disk usage #(inodes) max Filesystem Size Used Avail Capacity soft/hard soft/hard size Mounted on NetApp.1:/vol_home/hpc 2.70T 1.02T 1.68T 38%/17% 50G/100G 1.8M/2.0M 8T /home/hpc NetApp.1:/vol_sao 20.00T 14.91T 5.09T 75%/34% 1.8T/2.0T 4.0M/5.0M 60T /pool/sao NetApp.1:/vol_sao_atmos 18.00T 13.80T 4.20T 77%/7% 8.0T/10.T 9.0M/10.M 22T /pool/sao_atmos NetApp.1:/vol_sao_rtdc 1.00T 167.51G 856.49G 17%/1% 2.8T/3.0T 2.5M/3.0M 11T /pool/sao_rtdc NetApp.1:/vol_genomics 20.00T 17.12T 2.88T 86%/57% 1.8T/2.0T 1.8M/2.0M 50T /pool/genomics NetApp.1:/vol_data_genomics 1.90T 10.78G 1.89T 1%/3% 1.0T/2.0T 1.0M/2.0M 10T /data/genomics NetApp.1:/vol_data/* 3.60T 3.27T 336.17G 91%/37% 2.8T/3.0T 35.M/40.M 20T /data/sao:admin NetApp.1:/vol_old_pools/cluster* 17.00T 15.60T 1.40T 92%/22% /pool/cluster0:1:2:3:4:5:7 NetApp.1:/vol_sylvain 14.18T 12.75T 1.43T 90%/51% 14.T/15.T 63.M/65.M 15T /pool/sylvain NetApp.1:/vol_scratch/* 25.00T 19.53T 5.47T 79%/5% 2.8T/3.0T 1.0M/1.0M 50T /scratch/genomics:sao NetApp.5:/vol/a2v1/* 20.00T 0.05G 20.00T 1%/1% 14.T/15.T 2.0M/2.0M 20T /scratch/genomics01:sao01
Monitoring Quota Usage
The Linux command quota
is not (yet) working with the NetApp filer.
We compile a daily quota report and provide tools to parse the quota report. (need to insert links to these tools)
5. NetApp Snapshots: How to Recover Old or Deleted Files.
Some of the disks on the NetApp filer have the so called "snapshot mechanism" enabled:
- This allow users to recover deleted files or access an older version of a file.
- Indeed, the NetApp filer makes a "snapshot" copy of the file system (the content of the disk) every so often and keeps these snapshots up to a given age.
- So if we enable hourly snapshot and set a two weeks retention, you can recover a file as it was hours ago, days ago or weeks ago, but only up to two weeks ago.
- The drawback of the snapshot is that when files are deleted, the disk space is not freed until the deleted files age-out. like 2 or 4 weeks later.
How to Use the NetApp Snapshots:
To recover an old version or a deleted file, foo.dat, that was (for example) in /data/genomics/frandsen/important/results/
:
- If the file was deleted:
cd /data/genomics/.snapshot/XXXX/frandsen/important/results cp -pi foo.dat /data/genomics/frandsen/important/results/foo.dat
- If you want to recover an old version:
cd /data/genomics/.snapshot/XXXX/frandsen/important/results cp -pi foo.dat /data/genomics/frandsen/important/results/old-foo.dat
- The
-p
will preserve the file creation date and the-i
will prevent overwriting an existing file. - The
XXXX
is to be replaced by either:hourly.YYYY-MM-DD_HHMM
daily.YYYY-MM-DD_0010
weekly.YYYY-MM-DD_0015
whereYYY-MM-DD
is a date specification (i.e.,2015-11-01
)
- The files under
.snapshot
are read-only:- they be recovered using
cp
,tar
orrsync
; but - they cannot be moved (
mv
) or deleted (rm
).
- they be recovered using
6. SSD and Local Disk Space
We are in the process of adding local SSDs (solid state disks) to some compute nodes (not all - stay tuned), and
for special cases it may be OK to use disk space local to the compute node.
Until we post here detailed instruction, you should contact us if your jobs can benefit for either SSDs or local disk space (missing link).
Last Updated SGK.