1. Configuration
As of
| Quotas per user | ||||||
|---|---|---|---|---|---|---|
| Disk | Space | No. of files | Purpose | |||
Name | size | soft/hard | soft/hard | Snapshots? | Scrubbed? | Public disks |
| see note (1) | (2) | (3) | (4) | (5) | (6) | |
|
|
| 9/10M | yes: 4w | no | For your basic configuration files, scripts and job files
|
|
|
|
| yes: 2w | no | For important and smaller files like final results, configurations, etc.
|
/
|
|
|
| no | yes | For storing large files
|
|
|
| 25/26M | no | yes | For storing very large file (more space than in
|
| | | Project specific disks (/pool) | |||
/pool/kozakk | 11T | 10.5/11.0T | 26/278M | no | no | STRI/Krzysztof Kozak |
/pool/nmnh_ggi | 21T | 15.0/15.8T | 37/40M | no | no | NMHN/GGI |
/pool/sao_access | 21T | 15.0/15.8T | 37/40M | no | no | SAO/ACCESS |
/pool/sao_rtdc | 11T | 2.8/3.0T | 2.5/3.0M | no | no | SAO/RTDC |
/pool/sylvain | 30T | 29/30T | 71/75M | no | no | SAO/Sylvain Korzennik |
| Project specific disks (/scratch) | ||||||
/scratch/bradys |
| - | - | no | no | NMNH/Seán Brady/BRADY_LAB |
/scratch/kistlerl | 100T | 98/100T | 254/260M | no | no | NMNH/Logan Kistler |
/scratch/meyerc | 25T | 24/25T | 52/62M | no | no | NMNH/Christopher Meyer |
/scratch/nmnh_corals | 20T | 18/20T | 52/54M | no | MNNH/Corals | |
/scratch/nmnh_ggi | 77T | 75/77T | 195/200M | no | no | NMNH/GGI |
/scratch/nmnh_lab | 25T | 4/5T | 10/12M | no | no | NMNH/LAB |
/scratch/nmnh_mammals | 25T | 24/25T | 66/68M | no | NMNH/Mammals | |
/scratch/nzp_ccg | 35T | 34/35T | 89/92M | no | no | NZP/Michael Campana/CCG |
/scratch/sao_atmos | 250T | 245/250T | 252/261M | no | no | SAO/ATMOS |
/scratch/sao_cga | 25T | 7/8T | 18/20M | no | no | SAO/CGA |
/scratch/sao_tess | 50T | 36/40T | 94/210M | no | no | SAO/TESS |
/scratch/schultzt | 25T | - | - | no | no | NMNH/Ted Schultz/SCHULTZ_LAB |
/scratch/stri_ap | 25T | 4/5T | 10/12M | no | no | STRI/W. Owen McMillan/STRI_AP |
/scratch/sylvain | 70T | 68/70T | 176/182M | no | no | SAO/Sylvain Korzennik |
/scratch/usda_sel | 25T | 24/25T | 52/62M | no | no | NMNH/Christopher Owen/USDA_SEL |
/scratch/wrbu | 40T | 38/40T | 99/100M | no | no | WRBU |
| Near line (/store) | ||||||
/store/admin |
| - | n/a | yes: 8w | no | Sys Admin |
/store/public | 175T | 5/5T | n/a | yes: 8w | no | Public, available upon request |
/store/bradys |
| - | n/a | yes: 8w | no | NMNH/Sean Brady/BRADY_LAB |
/store/nmnh_ggi | 90T | - | n/a | yes: 8w | no | NMNH/GGI |
/store/sao_atmos | 300TB | - | n/a | yes: 8w | no | SAO/ATMOS |
/store/sylvain | 100TB | - | n/a | yes: 8w | no | SAO/Sylvain Korzennik |
/store/schultzt | 40TB | - | n/a | yes: 8w | no | NMNH/Ted Schultz/SCHULTZ_LAB |
/store/wrbu | 40TB | - | n/a | yes: 8w | no | WRBU |
| Extra | ||||||
|
| no | no | sys admin | ||
/scratch/dbs | 10T | no | no | data bases | ||
/scratch/tmp | 100T | no | no | scratch space for jobs that need a big temp, via -l bigtem=XX | ||
Notes
- The notation /x/{y|z} stands for /x/y or /x/z
- Sizes are indicative, since we adjust them as needed
- the notation 1.8/2.0TB means that the soft limit is 1.8TB and the hard limit is 2.0TB of disk space
- the notation 4/5M means that the soft limit is 4 million
inodesand the hard limit is 5 millions - Snapshots allow to restore deleted files, up to that many weeks
- Public space under
/pooland/scratchis scrubbed weekly - files older than 180 days are removed.
2. Storage Efficiency
- It is inefficient to store a slew of small files and if you do you may reach your
inodesquota before your space quota (too many small files).- Some of the disk monitoring tools show the
inodeusage. - If your
%(inode)>%(space)your disk usage is inefficient,
consider archiving your files intoziportar-compressedsets.
- Some of the disk monitoring tools show the
- While some of the tool(s) you use may force you to be inefficient while jobs are running, you should remember to
- remove useless files when jobs have completed,
- compress files that can benefit from compression (with
gzip,bzip2orcompress), and - archive a slew of files into a
zipor atar-compressed set, as follows:% zip archive.zip dir/
or% tar -czf archive.tgz dir/
both examples archive the content of the directorydir/into asingle zip or atgzfile. You can then delete the content ofdir/with% rm -rf dir/
- You can unpack each type of archive with
% unzip archive.zip
or% tar xf archive.tgz
3. Scrubbing of Public Disks
- We have implement a FIFO (first in first out) scrubbing model, where old files are deleted, aka scrubbed, to keep free space on the public disks.
- There is an age limit, meaning that only files older than 180 days get deleted.
- Older files get deleted before the newer ones (FIFO),
- We run a scrubber on a weekly basis.
- Consult the Scrubber and How to Request Scrubbed Files to be Restored page for additional information.
- In any case, we ask you to remove from
/pooland/scratchfiles that you do not need for active jobs.
4. Dedicated Project Disk Space
- For projects that would benefit from dedicated disk space, such space can be secured with project's specific funds when we expand the disk farm (contact us).
Last Updated SGK/PBF.