Page tree
Skip to end of metadata
Go to start of metadata
  1. IDL & GDL
    1. NUMPY multi-threading
  4. JAVA
  5. R
  6. Anaconda and miniconda
  7. Julia

[what is stricken out needs to be updated]

1. IDL & GDL

We have 8 interactive licenses and 128 run-time licenses for IDL, and have installed GDL version 0.9.5, an open source (read no licenses needed) idl-like package.

1.a IDL

  • The interactive licenses are available for all versions of IDL, from v6.1 to v8.7, and should be used only on the login nodes for pre- or post processing.
  • to access IDL, simply load the idl module:
    % module load idl
  • To view all the available versions, use:
    % ( module -t avail ) | & grep idl
  • Running IDL the normal way (i.e., interactively) in jobs submitted to the queue will quickly exhaust the available pool of licenses.
  • Instead you must use the run-time licenses.

Using IDL with Run-Time Licenses

  • To run IDL with a run-time license you must first compile your IDL code, and all the ancillary procedures and functions it may use and save it in a save set.

  • The following example shows how to compile a procedure called reduce, stored in the file, to a complete save set that will be called reduce.sav

    How to compile and save it as a save set
    % module load idl/6.1
    % idl
    IDL> .run reduce
    IDL> resolve_all
    IDL> save, /routine, file='reduce.sav'
  • After creating a save set, changes to any segment of the code will not be reflected in the .sav file; you must recompile the code each time you modify it.

  • To run IDL in run-time mode, you load the idl/rt module and use the -rt=XXX flag when invoking IDL.

  • To let the GE know that you will pull an IDL run-time license, use the -l idlrt=1 flag when qsub'ing.
    The GE will keep track of the number of licenses used and will limit accordingly the number of concurrent jobs using them to the number of available licenses.

  • Here is an example of a job file reduce.job that will run the reduce procedure:

    IDL run-time example job file
    # /bin/csh
    #$ -cwd -j y -o reduce.log -N reduce
    #$ -l idlrt
    echo + `date` job $JOB_NAME started in $QUEUE with jobID=$JOB_ID on $HOSTNAME
    module load idl/rt
    idl -rt=reduce.sav
    echo = `date` job $JOB_NAME done
  • You then run that job with:
    % qsub reduce.job


  • The name of the save file must match the name of the top procedure to run, and
  • there is no way to pass arguments to that top level procedure, when started in with -rt=name.sav, i.e.:
    If you want to run the procedure make_my_model, you should write and compile it to a make_my_model.sav file.
  • IDL procedures can read parameters from a file, or from standard input (stdin), for example
  • Example of reading from stdin (standart input)
    idl -rt=make_my_model.sav<<EOF

    the corresponding procedure would look like this

    Corresponding procedure
    procedure make_my_model, temperature, density, name
    ;; this procedure computes a model using a given temperature and density
    ;;   and saves it to a file whose name is derived from given string
    if n_params() eq 0 then
      ;; no parameters passed, so we need to read the parameters from stdin
      ;; let's initialize them to set the variable type (double and string)
      temperature = 0.D0
      density = 0.0D0
      name = ''
      read, temperature
      read, density
      read, name
    fullName = name + '.dat'
    print, 'running a model for T=', temperature,', rho=', density
    print, 'saving it to "'+fullName+'"'
    [the code goes here]
    • If the reference to the .sav file is to a different (sub)directory, IDL will execute a cd to that directory, namely
        idl -rt=src/make_my_model
      is in fact equivalent to
        cd src
        idl -rt=make_my_model

    • Some IDL procedures will use more than one thread (more than one CPU) if some operations can be parallelized.
      In fact IDL queries how many CPUs there are on the current machine to decide on the maximum number of threads it may start, assuming effectively that you are the sole user of that computer.
      (warning) This is not appropriate for a shared resource and it should not be used.
      To avoid this you must add the following instruction:
         CPU, TPOOL_NTHREADS = 1
      to limit IDL to using only one thread (one CPU). 

      Alternatively, you can request several slots (CPUs, threads) on a single compute node when submitting a job (qsub -pe mthread N, where N is a number between 2 and 64),
      and tell IDL to use that same number of threads with the CPU command.
      Of course, if your IDL code is not using N threads all the time, you will be grabbing more resources than you will be using, something that should be avoided.

    • You can check if IDL was started in run-time mode with
        IF lmgr(/runtime) THEN message, /info, 'runtime mode is on'

  • Some IDL instructions are not allowed in run-time mode to prevent emulating interactive mode, consult the manual.

  • Currently our run-time licenses allow us to run only IDL version 6.1 (and earlier).

    This was the latest version of IDL available when we bought these run-time licenses, as we decided not to pay for s/w maintenance on these.
    We will consider upgrading these licenses if there is a need.

  • You can check how many run-time licenses the GE thinks are still available with
    %  qhost -F idlrt -h global

  • You can check how many licenses are used by loading one of the idl module and issuing either command (to check interactive use) :
    % module load idl
    % $IDL_DIR/bin/lmstat -c /share/apps/idl/license/license.dat -f idl
    or (to check run-time license use)
    % module load idl
    % $IDL_DIR/bin/lmstat -c /share/apps/idl/license/license.dat -f idl_rt
    Note that the number of licenses returned by lmstat has to be divided by 6 (for historical reasons, each instance of IDL grabs 6 of these license tokens.)

1.b GDL

  • GDL is an open source package that is compatible with IDL (version 7.1), see

  • This is a free software (you get what you paid for) and thus there is no licensing nor run-time limitations.

  • The latest version of GLD (0.9.5) is available on Hydra, and is accessible by loading the tool/gdl module:
    % module load tools/gdl
    The module will set variable GDL_STARTUP to either ~/.gdl.startup.0.9.5 or ~/.gdl.startup, if either file exists (checked in that order).
    Any GDL commands in the startup file will be executed every time GDL is started as if typed at the prompt.

  • Like IDL, some GDL procedures will use more than one thread (more than one CPU) if some operations can be parallelized.
    GDL queries how many CPUs there are on the current machine to decide on the maximum number of threads it may start, assuming effectively that you are the sole user of that computer.
    (warning) This is not appropriate for a shared resource and it should not be used. To avoid this you must add the following instruction:
    to limit GDL to using only one thread (one CPU).

    Alternatively, you can request several slots, as described for IDL above, with the same caveats.


  • Full fledged MATLAB is not available on Hydra
    • we would need to purchase licenses for Hydra,
    • SAO users can use the MATLAB compiler to produce run-time version of their MATLAB code.

  • The MATLAB run-time environment is available on Hydra
    • to access it, load the right module:

      Name of moduleDescription
      matlab/R2014a2014 first release
      matlab/R2017b02017 second release (SAO/CF equivalent)
      matlab/R20147b2017 second release, with (latest) update (# 9)
      matlab/R2019a2019 first release (latest version as of April 2019)
      matlab/rt → R2017b0default run-time is set to use R2017b0

      Note that:

      • you must compile your MATLAB application to run it on Hydra,
      • SAO has a single (concurrent) seat license for the MATLAB compiler, available on all CF-managed machines.

3. Python

  • By default Rocks 6.1.1 installs python version 2.6.6
    • To access additional python packages, load the module tools/python2.6-x, (-x for extra) ie:
      % module load tools/python2.6-x
  • Python version 2.7.6 is available by loading the module opt-python, ie:
    % module load opt-python
  • (warning) You should unload the tools/python2.6-x module before loading the opt-python one.

NUMPY Multi-Threading

  • By default, NUMPY is build to use multi-threading, namely some numerical operations will use all the available CPUs on the node it is running on.
    • (warning) This is NOT the way to use a shared resource, like Hydra,
    • The symptom is that your job is oversubscribed.
  • The solution is to tell NUMPY how many threads to use, using the respective module:

    serial case
    load tools/single-thread-numpy


    multi-thread case
    load tools/mthread-numpy

    use module show <module-name> to see what is done.

    1. Example:

      # this example uses 4 threads
      #$ -pe mthread 4
      #$ -cwd -j y -o demo-mthread-numpy.log -N demo-mthread-numpy
      echo + `date` $JOB_NAME started on $HOSTNAME in $QUEUE with id=$JOB_ID
      echo NSLOTS = $NSLOTS
      module load tools/mthread-numpy
      echo = `date` $JOB_NAME done.

4. Java

  • Java is available on Hydra, version 1.7 and version 1.8,

    by loading the appropriate module:
       % module load java/1.7


       % module load java/1.8

  • (warning) Java does not play nice w/ GE:

    • Java, by default, wants to starts as many threads as possible and grab as much memory as there is.
    • By not specifying some memory related parameters, java fails in every submitted job, with the following message:

      Error occurred during initialization of VM
      Could not reserve enough space for object heap
    • (lightbulb) You should always start java with the following options:

      java -d64 -server -XX:MaxHeapSize=1g

      where the value "1g" should be adjusted to the memory needed by the application and for the job to fit within the queue and the requested resources memory constraints.

      The total amount of memory used by java is not just the maximum heap size.

    • If you need more memory, be sure to adjust the memory resource request accordingly (-l memres=X,h_data=X,h_vmem=X), see the section about memory reservation in the Available Queues page.

  • (grey lightbulb) The complete documentation for all of java options (v1.8) is posted here.

5. R

  • R version 3.2.1 (built with gcc 4.9.2) is available by loading the corresponding module:
    % module load tools/R

  • Installing Packages: 

    Users can install packages their own packages in a user-specific library. When the install.packages() command is used, there will be a notification that the system-wide library is not writable and you will prompted to create a personal library in your home folder. All future packages that you install will be installed into your personal library. You will then be prompted to choose a repository.

    Packages only need to be installed one time to be accessible to all of your R jobs on all nodes on the cluster. Because compute nodes don't have internet access, you will need to run the install.packages() command interactively from the login node.

  • $NSLOTS in R scripts
    It is best practice to use the environmental variable NSLOTS in your scripts to specify the number of cores to use for R commands that supports multiple cores. By using NSLOTS rather than hardcoding the number of cores in your R script, your job will utilize the requested number of slots, even if you change your qsub submission parameters.

    (lightbulb) Use the R Base function Sys.getenv() to access the value of $NSLOTS from within your script.

    ## Script using Sys.gentenv() to read the value of $NSLOTS
    numcores <- Sys.getenv("NSLOTS")
  • Proper parallelization of makeCluster():
    R packages that incorprate the makeCluster()  function must specify type="FORK" as an argument. Without this, Hydra's scripts that kill zombie jobs will terminate the R processes created by makeCluster()

    cl <- makeCluster(no_cores, type="FORK")

6. Anaconda and miniconda

  • Users may find it convenient to use anaconda or miniconda to install and manage software in their user space. It is acceptable to use these systems, but like compiling and running any software on Hydra, the user should be familiar with how the software functions and uses resources (memory, CPUs, disk) so as not to overutilize cluster resources.
  • Miniconda, rather than the full anaconda, is preferred on Hydra because it initially installs minimal packages reducing disk usage; required dependencies will be installed by conda.
  • To install miniconda, download a current 64-bit miniconda installer from here. Then follow these instructions to install in your user space. The default location to unpack the software is your home directory. This works well because /home is not scrubbed and the disk space needed by miniconda typically is within the  user quotas for /home.
  • After installation, you can use conda install to install software. See these instructions for full conda documentation:
  • To use software installed via conda in submitted job, you can create a personal module file to add your miniconda bin directory to your PATH. See here for instructions.

7. Julia

Last update    SGK/MPK

  • No labels