WorkingInCB9

Things that are worth to have in mind when starting to work in CB9

OS flavour

In CB9, we start to work with OS flavour :

object template profile_node19-101.wn.iihe.ac.be;
variable OS_FLAVOUR_ENABLED = true;
...

It means that the OS templates will be taken from os/sl6.x-x86_64 (instead of os/sl630-x86_64 for example). However, you still have to define the precise OS you want in the template site/os_version_db through the nlist OS_VERSION, but this choice will have an effect on the list of OS repositories, that is constructed in the template repository/config/os located in os/sl6.x-x86_64.

If you want to understand in deep the mechanism of OS flavoring, then read the template os/version in standard directory.

Working with repo snapshots

With the new SPMA, the choice of the version of each software is done on-the-fly by the component, in function of the content of the repositories at the time the component is running. If you change the repositories, then it might impact all the machines using them. So, to get a minimum of stability in the software environment of machines, it is necessary to work with repository snapshots.

Repository templates in CB9

The templates of the repositories managed by snapshots are all located under begrid/repository/snapshot. Since with SPMA-YUM, we don't need the software versions anymore, so the content of these templates can be minimalist. Example :

structure template repository/snapshot/ca;

"name" = "ca";
"owner" = "grid_admin@listserv.vub.ac.be";
"protocols" = list(
  nlist("name","http",
        "url","http://yum.iihe.ac.be/cb9/"+REPO_YUM_SNAPSHOT_DATE['ca']+"/egi_ca")
);

In the above example, you see the role played by the nlist REPO_YUM_SNAPSHOT_DATE, that is used to construct the real URL pointing to the desired snapshot of the repository.

Managing the snapshot dates

Let us suppose that, on the 19th of March 2014, you have updated the "ca" repository, and that you have created a new snapshot just after. To force your machine to use the new snapshot, you will have to adapt accordingly the following nlist in site/global_variables :

variable REPO_YUM_SNAPSHOT_DATE = nlist(
    'ca','20140319',
    'sl6_addons','20140313',
    'sl6_epel','20140313',
    'sl630_x86_64','20140313',
    'sl630_x86_64_errata','20140313',
    'emi_3_0_sl6_x86_64_base','20140313',
    'emi_3_0_sl6_x86_64_third_party','20140313',
    'emi_3_0_sl6_x86_64_updates','20140313',
    'wlcg_x86_64','20140313',
);

If you don't adapt this nlist, then the URL of the "ca" repo will still point to the old snapshot with old RPMs !

Howto

How to update repositories and create snapshots ?

Most of local repositories are stored on yum.iihe.ac.be. Update of these repositories should always been done with the command "reposync" (or with the script described below, which is a wrapper above reposync) because they are mirrors of official repositories. There are several possibilies to manage repository snapshots. This can be achieved at a low level by the filesystem (possible with btrfs, LVM, ZFS,...). Or we can simply use hard links, as in the tool created by Michel Jouvin, available in the SCDB repo in Quattor GitHub, in utils/misc (name of the scripts : yum_snapshot and update_yum_repos).

A wrapper to simplify repo management has been written. Here is how to use it :

1. Log in as root on yum.iihe.ac.be

2. Go to the directory /var/www/html/
3. Launch the interactive script manage_repositories.pl and follow the instructions...

Note 1 : on yum, the repositories are defined it /etc/yum.repos.d/cb9.repo.

Note 2 : for packages that don't belong to official repos, please use the sl6_t2b repository (the corresponding repository template is "site").

How to install a package that I can't find in the main repositories ?

A dedicated place has been foreseen for such packages : the repository sl6_t2b. Here is the recipe :

1. Drop your package in yum.iihe.ac.be:/var/www/html/cb9/sl6_t2b

2. Create a snapshot of the repo.
3. Change the snapshot date for the repo sl6_t2b in the REPO_YUM_SNAPSHOT_DATE nlist.

Important remark : as of 13th of August 2014, we decided to use the date in timestamp format, instead of Ymd format, to avoid problems with yum cache when the same repository is modified several times during the same day.