SingularityContainers

To make SL6/SL7/CC8/EL9 flavours available for everyone while the cluster is using another version, we make use of containers. This also works for SL8 and EL9 on the new HTCondor CC7 Cluster, just replace SL7 with the version you want.

Prerequisites

Helper scripts are in /group/userscripts/, so make sure either it is in your $PATH variable with:

export PATH=$PATH:/group/userscripts/

or just call the full path:

/group/userscripts/sl6

To test code on the mX UIs

Simply go into an EL7 environment:

sl7 bash

This should give you a prompt, where your SL7 code should work.

Just to convince yourself, you can cross-check the OS release:

cat /etc/redhat-release

To use SL7 inside a cluster job

You can simply ask your script to be run inside the sl7 container:

sl7 /user/$USER/MYSUPERSCRIPT.sh

To send it to the cluster, the job.sh file you now will send to qsub should contain:

/group/userscripts/sl7 /user/$USER/MYSUPERSCRIPT.sh

Please note that you then need, in your script, to go to TMPDIR.
It is HIGHLY recommended to play with this first, by printing path and env variables, to check everything inside the container.

To be more exhaustive, what the script sl7 does is first to add your PATH & LD_LIBRARY_PATH to singularity, and then launch your script using the following line:

singularity exec -B /cvmfs -B /pnfs -B /user -B /scratch /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el7:latest MYSCRIPT.sh

where:

• exec: action to do for singularity, here will just execute your command in the container specified
• -B /mountpoint: is used to have the path present in the container if needed.
• /cvmfs/.../cms:rhel7: the path on /cvmfs of the container used.
• MYSCRIPT.sh: script or command to execute

N.B. You can look at more containers inside /cvmfs/singularity.opensciencegrid.org/opensciencegrid/.
Or you can also create your own container, with for instance specific software versions, it is quite simple.
You can use the guide here, or google singularity container in your favorite search engine :)

To use EL9 inside a cluster job

As Centos 7 is soon to be EOL (30/06/2024), we plan on changing the OS of the cluster to an EL9 flavor. Until then, you can use the following to get access to a containter in EL9:

singularity exec -B /cvmfs -B /pnfs -B /user -B /scratch /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el9:latest MYSCRIPT.sh

If you need grid commands

The above containers work perfectly on our cluster. However, they have the drawback of not being updated often. Unfortunately, to be able to work in a grid environment, the CA (certificate authority) files need to be kept up-to-date.
For this reason, CMS also provides some containers. Unfortunately, not all software is inside (python in el9 is lacking atm). But the grid commands do work. For them, run the scrip:

/cvmfs/cms.cern.ch/common/cmssw-el8
or
/cvmfs/cms.cern.ch/common/cmssw-el9

Some more info on these containers can be found here