2019 Cluster Upgrade to Hydra-5

   Read the Hydra-5 Migration Notes and Reset Your Password    

The Hydra cluster has been upgraded as follows:

• Software
  • Upgrade OS to CentOS 7.6,
  • Change our management tool (from Rocks 6.x to Bright Cluster, aka BCM 8.2),
  • Switch from SGE to UGE (Univa) version 8.6.6, a commercially supported version of SGE (backward compatible).

• Hardware
  • Added 16 new nodes (40c/380GB) for a total of 640 cores,
  • Decommissioned oldest nodes (old compute-6* and compute-4-*),
  • Add ed1.5PB of high performance storage (parallel file system GPFS, aka Spectrum Scale),
  • This is in addition to the 500TB of NetApp and the  950TB of near-line storage (NAS),
  • The /scratch disk has been moved to the GPFS and increase in size from 100TB to 900TB (450TB+450TB for /scratch/genomics and /scratch/sao)
  • Moved all the nodes to a 10Gb/s Ethernet connection (was 1Gb/s).

 Please read this page carefully, and after the upgrade, do not hesitate to contact us (at SI-HPC-Admin@si.edu) if something is not working as it should any longer.

• For biology and genomics software issues and/or incompatibilities that arise, please contact SI-HPC@si.edu,
• SAO users please contact Sylvain at hpc@cfa.harvard.edu.

List of Software Changes 

• Implementation of user account management using LDAP
  • The procedure to change your password has changed:
    • You can use passwd  on either the login node and that's it, or
    • Use the Self Service Password (SSP) page, listed at https://hydra-5.si.edu
  • Email SI-HPC-Admin@si.edu only if this fails.

• Switching from Rocks to BCM
  • BCM locates things differently from Rocks;
    • for example /opt  is no longer used, instead BCM uses /cm  
  • If you use modules, everything should work the same;
    • but if you hardwired locations, things may no longer work the same;
  • The compute nodes will be renamed as follows:
    • compute-NN-MM, 
    • i.e. the "N", or logical rack number, and
    • M the node index are both two-digit numbers;
  • You should use modules as much as possible.

• Job Scheduler
  • We switched to UGE, or Univa Grid Engine; that
    • is backward compatible with SGE;
    • offers some additional features;
    • although the output of some commands will look different, and

    • some have different options .

•  The list of queues and their limits will not changed,
  • a few complex values will change (to use local SSD and GPU usage)

  •  One important change  is that 

    the memory reservation value will no longer be specified per slots, but per jobs:

    the specification  "-pe mthread 10 -l mres=20G,h_data=20G,h_vmem=20G,himem"
    must be changed to "-pe mthread 10 -l mres=200G,h_data=20G,h_vmem=20G,himem"
    to reserve 200GB of memory for this job, asking the scheduler to start on a node that has 20G per thread of free memory

• Compilers
  • The default compiler versions has changes, and
  • they had to be reloaded.
  • For MPI jobs the compiler/flavor/version combos has also changed.

• Local tools
  • The tools accessible with the tools/local module have been split into two groups and
  • some of the names have been changed or simplified
  • Use module help tools/local and module help tools/local+ to see what the split is and what the names are.
  • Use module help tools/local-bc to see the correspondence
  • Loading tools/local-bc will
    • load tools/local and tools/local+ and
    • create aliases to be backward compatible (-bc)
  • use the new names whenever possible

Last updated 06 Sep 2019  SGK.