OtherSoftware: Difference between revisions

From T2B Wiki
Jump to navigation Jump to search
 
(2 intermediate revisions by the same user not shown)
Line 6: Line 6:
=== General purpose area ===
=== General purpose area ===


!!! N.B.: For now, the recommended release is [http://lcginfo.cern.ch/release/98python3/ '''LCG_98python3'''] with platform '''x86_64-centos7-gcc10-opt''', with a list of all packages included [http://lcginfo.cern.ch/release_packages/x86_64-centos7-gcc10-opt/98python3/ here] !!!
!!! N.B.: For now, the recommended release is [https://lcginfo.cern.ch/release/102/ '''LCG_102'''] with platform '''x86_64-centos7-gcc11-opt''', with a list of all packages included [https://lcginfo.cern.ch/release_packages/x86_64-centos7-gcc11-opt/102/ here] !!! Just source the softare area like this:
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_102 x86_64-centos7-gcc11-opt




Instead of trying to explain in details how the /cvmfs sub-directories are organized, we will limit ourselves in this wiki to give some relevant links about the general purpose area:
====The LCG environments, a group of packages neatly built into a coherent environment ====
Instead of trying to explain in details how the /cvmfs/sft.cern.ch sub-directories are organized, we will limit ourselves in this wiki to give some relevant links and commands about the general purpose area.


[http://lcginfo.cern.ch Highlevel description of the software packages and how they are grouped in LCG Configurations] <br>
A high-level description of the software packages and how they are grouped in LCG Configurations can be found here: [http://lcginfo.cern.ch lcginfo.cern.ch].<br>
[http://ep-dep-sft.web.cern.ch/ Website of the SFT group managing what's in /cvmfs/sft.cern.ch]
You can search by LCG release, platforms, but more importantly by package name.<br>
Then from the package, you can select the specific version you want, and find which LCG release contains it.<br>
Finally just find a arch-OS-compiler combo that suits you  (see below), and voila ! <br>


==== Load a specific software repository ====
==== Load a specific software repository ====


The software packages are grouped in LCG Configurations. In each of these configurations, you will normally find set of software versions that are compatible with each others. To get the list of LCG Configurations :
The software packages are grouped in LCG Configurations. In each of these configurations, you will normally find a set of software versions that are compatible with each others. To get the list of LCG Configurations :
<pre>/cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh</pre>
<pre>/cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh</pre>


Now, let's say you want to know more about the "LCG_93c" configuration :
Now, let's say you want to know more about the '''"LCG_102"''' configuration, and get what architecture/OS/compiler are available for it :
<pre>/cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_93c
<pre>/cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_102
Available <arch-os-compiler> for LCG_93c :  
Available <arch-os-complier> for LCG_102 :  
   x86_64-centos7-gcc62-opt
  aarch64-centos7-gcc11-opt
   x86_64-centos7-gcc7-dbg
  arm64-mac12-clang131-opt
   x86_64-slc6-gcc62-dbg
  x86_64-centos7-clang12-dbg
   x86_64-slc6-gcc62-opt
  x86_64-centos7-clang12-opt
   x86_64-slc6-gcc7-dbg
  x86_64-centos7-gcc11-dbg
   x86_64-centos7-gcc11-opt
   x86_64-centos7-gcc8-dbg
   x86_64-centos7-gcc8-opt
  x86_64-centos8-gcc11-dbg
   x86_64-centos8-gcc11-opt
   x86_64-centos9-gcc11-dbg
  x86_64-centos9-gcc11-opt
  x86_64-mac11-clang120-opt
  x86_64-ubuntu2004-gcc9-opt
  x86_64-ubuntu2204-gcc11-opt
</pre>
</pre>


If you are on an SL6 UI and we want gcc62, we have to issue the following command :
If you are on an Centos 7 UI (like the T2B cluster) and want gcc11 compiler (prefer gcc compiler to others on the cluster unless you know what you are doing), issue the following command :
<pre>source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_93c x86_64-slc6-gcc62-opt</pre>
<pre>source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_102 x86_64-centos7-gcc11-opt</pre>


Check the version of gcc :
Check the version of gcc :
<pre>$ gcc -v
<pre>gcc -v
Using built-in specs.
gcc version 11.2.0 (GCC)</pre>
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/cvmfs/sft.cern.ch/lcg/contrib/gcc/6.2.0native/x86_64-slc6/bin/../libexec/gcc/x86_64-pc-linux-gnu/6.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ../gcc-6.2.0/configure --prefix=/build/pmendez-sftnight/install-620local -with-system-zlib --disable-multilib --enable-languages=all
Thread model: posix
gcc version 6.2.0 (GCC)  
</pre>


At this moment your environment has been changed to use all the versions in this specific software area.
At this moment your environment has been changed to use all the versions in this specific software area, just check by issuing '''env'''.


=== Be careful when loading a different version of python ===
=== GFAL issue when loading a different version of python ===
When the software reopistory you want to use contains a version of python that is higher than the one installed on the cluster, you might encounter a problem with the gfal libraries:
When the software repository you want to use contains a version of python that is higher than the one installed on the cluster, you might encounter a problem with the gfal libraries:
<pre>
<pre>
'import site' failed; use -v for traceback
'import site' failed; use -v for traceback
Line 60: Line 67:
</pre>
</pre>


<br><br>
=== CMS specific area ===
=== CMS specific area ===
!! Prefer loading an LCG complete environment (see above), this way you are sure your environment and packages are coherent whith each other !!


The T2 hosts many software that are packaged with specific versions of CMSSW but that can be used in a stand alone mode. A tool was designed to easily find what is available and which versions can be used.<br>
The T2 hosts many software that are packaged with specific versions of CMSSW but that can be used in a stand alone mode. A tool was designed to easily find what is available and which versions can be used.<br>

Latest revision as of 13:35, 17 February 2023

Software available from the T2 UI machines

The T2 has many more software than what is available directly from the M machines. All are provided by cern's centrally managed files system called cvmfs. They are divided in a CMS specific area and a more general area aimed at high energy in general.
The CMS specific area comes with a home brew script that makes it easy for searching specific versions of the software you need. However, the set-up is a lot more complicated than the general purpose area. Therefore we recommend the general purpose area unless you do not find the specific version you need in it.

General purpose area

!!! N.B.: For now, the recommended release is LCG_102 with platform x86_64-centos7-gcc11-opt, with a list of all packages included here !!! Just source the softare area like this:

source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_102 x86_64-centos7-gcc11-opt


The LCG environments, a group of packages neatly built into a coherent environment

Instead of trying to explain in details how the /cvmfs/sft.cern.ch sub-directories are organized, we will limit ourselves in this wiki to give some relevant links and commands about the general purpose area.

A high-level description of the software packages and how they are grouped in LCG Configurations can be found here: lcginfo.cern.ch.
You can search by LCG release, platforms, but more importantly by package name.
Then from the package, you can select the specific version you want, and find which LCG release contains it.
Finally just find a arch-OS-compiler combo that suits you (see below), and voila !

Load a specific software repository

The software packages are grouped in LCG Configurations. In each of these configurations, you will normally find a set of software versions that are compatible with each others. To get the list of LCG Configurations :

/cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh

Now, let's say you want to know more about the "LCG_102" configuration, and get what architecture/OS/compiler are available for it :

/cvmfs/sft.cern.ch/lcg/views/checkSetupViews.sh LCG_102
Available <arch-os-complier> for LCG_102 : 
  aarch64-centos7-gcc11-opt
  arm64-mac12-clang131-opt
  x86_64-centos7-clang12-dbg
  x86_64-centos7-clang12-opt
  x86_64-centos7-gcc11-dbg
  x86_64-centos7-gcc11-opt
  x86_64-centos7-gcc8-dbg
  x86_64-centos7-gcc8-opt
  x86_64-centos8-gcc11-dbg
  x86_64-centos8-gcc11-opt
  x86_64-centos9-gcc11-dbg
  x86_64-centos9-gcc11-opt
  x86_64-mac11-clang120-opt
  x86_64-ubuntu2004-gcc9-opt
  x86_64-ubuntu2204-gcc11-opt

If you are on an Centos 7 UI (like the T2B cluster) and want gcc11 compiler (prefer gcc compiler to others on the cluster unless you know what you are doing), issue the following command :

source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_102 x86_64-centos7-gcc11-opt

Check the version of gcc :

gcc -v
gcc version 11.2.0 (GCC)

At this moment your environment has been changed to use all the versions in this specific software area, just check by issuing env.

GFAL issue when loading a different version of python

When the software repository you want to use contains a version of python that is higher than the one installed on the cluster, you might encounter a problem with the gfal libraries:

'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/usr/bin/gfal-mkdir", line 24, in <module>
    from gfal2_util.shell import Gfal2Shell
ImportError: No module named gfal2_util.shell

This happens because of a mismatch between the software used and the libraries installed. The easiest solution is to use python 3 and a specific version of the UI (user interface) software like in the following example:

    source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_93python3 x86_64-centos7-gcc7-opt
    source /cvmfs/grid.cern.ch/emi3ui-latest/etc/profile.d/setup-ui-example.sh



CMS specific area

!! Prefer loading an LCG complete environment (see above), this way you are sure your environment and packages are coherent whith each other !!


The T2 hosts many software that are packaged with specific versions of CMSSW but that can be used in a stand alone mode. A tool was designed to easily find what is available and which versions can be used.

To list all the available software:

/swmgrs/cmss/soft.pl --list


Most used software are specific version of gcc that are needed, or a root or geant4 version. To find all the available gcc versions:

/swmgrs/cmss/soft.pl --versions gcc


Now, to get a given version of gcc (4.6.2 in this example) :

/swmgrs/cmss/soft.pl --load gcc/4.6.2


This prints the full path of the init.sh that you have to source to get the desired version of gcc :

source /cvmfs/cms.cern.ch/slc6_amd64_gcc462/external/gcc/4.6.2/etc/profile.d/init.sh


Let's now check the version of gfortran :

gfortran --version
GNU Fortran (GCC) 4.6.2


Template:TracNotice