CrabBabysitter

Crab Babysitting tool wiki

PageOutline

Description of the tool

The Crab Babysitting tool is a new tool built on top of the AutoMaticToptreeProduction CrabHandler class. The latter class takes care of the GRID related part in the TopTree Production and also the setting up of the crab config. Except for the creation of the crab config in CrabHandler, this class is generic and could be used for any GRID task using CRAB. This is why the Crab Babysitter was created. It uses all the generic functions of the CrabHandler e.g. create proxy/myproxy credentials, submit jobs, fetch jobs, resubmit crashed jobs, publish to DBS etc.... The only difference to the TopTree production mechanism is that it does not generate your crab.cfg neither any CMSSW setup. The aim is that you create your own CMSSW setup (if needed) and CRAB configuration and then pass it on to the CrabBabysitter tool to process it.

What can I submit with this tool?

Anything from creating DQM files, simulation to analysis. Anything that uses CRAB:-)

How to use it

First you should get the code in your directory on the m-machines (you should NOT put this inside any CMSSW area)

cvs co -d AutoMaticTopTreeProducer UserCode/mmaes/AutoMaticTopTreeProducer
cd AutoMaticTopTreeProducer

Now create a .config file to store your grid-pass. It will be used to generate credentials

touch .config

chmod 600 .config // important

open .config and add: GridPass mypass -> save

Then you should link your CMSSW area inside this folder for convenience. For example:

ln -s /path/to/my/CMSSW/Project MyProject

Use it to submit and handle jobs

In this running-mode, you use the script to submit jobs for a new task. Let's assume you used "MyProject" as symlink-name in the previous command. Further let's assume that the crab config you created is called crab.cfg.

• • First: make shure you have sourced the CMSSW environment (e.g.: you should be able to run scram list CMSSW)
  • Second: make shure you have sourced the relevant CRAB installation!

Now you are ready to run the script

python CrabBabysitter.py -h // if you want to see all options

python CrabBabysitter.py -d MyProject/src/myfolder-where-crab.cfg-is & // change argument to your proper setup

You will find a log-file (timestamp.txt) under logs/ showing the progress of the task

NOTE: If your crab config is not called crab.cfg you should provide --cfgname=filename as extra option!

Use it to handle jobs from a previously submitted task

If you have a crab task where you already sumitted your jobs, the babysitter can take over from you and handle the jobs. Again we assume you linked "MyProject" to your directory containing your crab working directory e.g. crab_0_110517_101636. If the jobs are already sumitted and you want to attach this task to the CrabBabysitter, you can do this the following way:

python CrabBabysitter.py -d MyProject/src/myfolder-where-crab-working-dir-is --attach=crab_0_110517_101636 & // change both arguments to your proper setup

You will find a log-file (timestamp.txt) under logs/ showing the progress of the task

If you don't want to use the symlinks

That's fine, there is no technical reason why to use them except for convenience. If you did not symlink, your -d argument will look like:

-d /path/to/my/CMSSW/Project/src/myfolder // no symlinks

rather than

-d MyProject/src/myfolder // symlink

Blacklisting sites

There are two options to blacklist certain sites on-the-fly (meaning without having to change your crab.cfg)

• • Method 1 (task-per-task blacklist): in the crab working directory, e.g.: crab_0_110517_101636, place a file ".blacklist" containing one line with the following syntax: T2_ES_IFCA,T2_BE_UCL. This will blacklist any number of sites. Remove sites from the file (or remove the file) to undo the blacklist.
  • Method 2 (all-task-blacklist): Instead of applying a blacklist for one babysit task (as method 1 did), we apply a blacklist for all tasks governed by the CrabBabysitter. To do this, you should add the following line to AutoMaticTopTreeProducer/.config

GridBlacklist T2_RU_IHEP,T2_RU_SINP,T2_UK_London_Brunel,T2_FR_GRIF_LLR,T3_CO_Uniandes,T2_RU_RRC_KI