ProdAgentUsers

ProdAgent for IIHE users

• instructions for LCG5 prod team

begin

• login on master3 and make sure you are using bash
• cd to your prodagent workspace
• setup the environment

  source /msa3/prodagent/start.sh

• if your proxy was not valid anymore, generate a new one with the new_proxy script

 source /msa3/prodagent/new_proxy.sh

• • for production user run

 source /msa3/prodagent/new_proxy_prod.sh

• make sure that prodAgentd is running
  • check status with

  prodAgentd --status

• • start it with

  prodAgentd --start

• • stop it with

  prodAgentd --shutdown

first time setup

• make sure you are registered in VO cms
  • you will probably need a production role
• login on master3
• most scripts and commands are in bash, so use bash

  bash

• create a new directory that you will use as working area.
  • current setup needs to be done on a local disk. this will change.

 DIR=/some/dir/<tt>whoami</tt>/prodagent
 mkdir -p $DIR
 cd $DIR

• setup the prodagent environment using the start_new.sh script. The script contains/does the following things:
  • set the version of prodagent and setup the environment according to this verison
  • creates a directory for the gridproxy and changes the environment according to this variable
  • this allows you to be consistent when you switch between VOs and VOMS roles. when doing other work.
  • when the directory is created on nfs-space, you can "take" your proxy from machine to machine
  • will display the validity of the proxy
  • initialise prodagent. this will (for LCG)
  • create some xml files in $PWD/work
  • set the correct working directory for prodagent
  • set the LCG jobcreator
  • set the LCG submitter
  • set test DBS/DLS values. See later instructions for official production values.
  • set some default merge sizes
  • set the phedex dropbox path
  • run

 source /msa3/prodagent/start_new.sh 

• if your proxy was not valid anymore, generate a new one with the new_proxy script. This script will
  • set correct directory for the proxy
  • create new proxy with correct VO and VOMS role
  • set myproxy if needed with

 source /msa3/prodagent/new_proxy.sh

• give your Prodagent config a unique name
  • the name is used for monitoring purposes, and always displayed as 'name'@'host'
  • the name is also used to setup database access
  • this setup uses a central MySQL DB for operation. if you want to install something locally (fully under your own control), please see section Local database setup
  • run

  export MYPANAME=<yourname>
  export MYSQLPASSWD=<yourpassword>
  source /msa3/prodagent/setup_dbaccess.sh

• • you will be prompted 2 times for a root user db password. ask the site admins for this password.

  Please provide the mysql user name (typically "root") for updating the
  database server (leave empty if not needed): root

  Please provide mysql passwd associated to this user name for
  updating the database server:

• • when prompted with the following warning, press Y

 You will be using ports and hosts settings instead of
 sockets. Are you sure you want to use this to connect
 to a database as it is a potential security risk? (Y/n)
  Y

Local database setup

To be used for local testing. You don't need access to the central DB but your workspace must be installed on a local disk.

• create a new directory that you will use as working area.
  • setup needs to be done on a local disk.

 DIR=/scratch/<tt>whoami</tt>/prodagent
 mkdir -p $DIR
 cd $DIR

• setup the prodagent environment using the start_new.sh script.

 source /msa3/prodagent/start_new.sh 

• if your proxy was not valid anymore, generate a new one with the new_proxy script.

 source /msa3/prodagent/new_proxy.sh

• give your Prodagent config a unique name

  prodAgent-edit-config --component=ProdAgent --parameter=ProdAgentName --value=<username>

• install the database part
  • choose a password for the user and set it

  export MYSQLPASSWD=somegoodpassword

• • run

  source /msa3/prodagent/setup_local_mysqldb.sh

• • you will be prompted twice for a root user db password. this is your previous password prefixed with root.

  Please provide the mysql user name (typically "root") for updating the
  database server (leave empty if not needed): root

  Please provide mysql passwd associated to this user name for
  updating the database server:

Example test exercise

You should be able to run this after the initialisation

• start prodagent

  prodAgentd --start

• setup your CMS environment (if not already done)

  export VO_CMS_SW_DIR=/msa3/cmssoft
  source $VO_CMS_SW_DIR/cmsset_default.sh

• • to check wheter or not the environment is already setup, see if the following command returns something useful

  which cmsRun

• go to your project directory do an eval-scram-runtime

 oldpwd=$PWD
 cd /some/path/CMSSW_1_2_0
 eval <tt>scramv1 ru -sh</tt>
 cd $oldpwd

• take an example production config file and convert it in a workflow
  • it will create some other files, one is the actual workflow file test-iihe-fullprod-1-Workflow.xml
  • Note that:
  • when using createPreProdWorkflow.py the PoolOutputModule name in the PSet cfg file should be compliant with the conventions to reflect the data tier the output file contains (i.e. GEN-SIM , GEN-SIM-DIGI ).
  • when using createProductionWorkflow.py the PSet cfg file should have a PSet datset structure with the datatier explicitelly defined.

  mkdir -p first-test
  cd first-test
  wget -O test-iihe-fullprod-1.cfg http://mon.iihe.ac.be/trac/t2b/attachment/wiki/ProdAgentUsers/test-iihe-fullprod-1.cfg?format=raw
  python $PRODAGENT_ROOT/util/createProductionWorkflow.py --cfg $PWD/test-iihe-fullprod-1.cfg --version CMSSW_1_2_0 --fake-hash

• submit the jobs
  • the workflow defined by --workflow= needs to be a full path
  • selecting one specific site can be done by setting --site-pref= to sthe storage element of the site
  • eg --site-pref=maite.iihe.ac.be wil select IIHE

 python $PRODAGENT_ROOT/test/python/IntTests/InjectTestLCG.py --workflow=$PWD/test-iihe-fullprod-1-Workflow.xml --site-pref=maite.iihe.ac.be --run=99999 --nevts=10 --njobs=3

• everything will now run in background
  • check status explicitly

  boss q -c $PRODAGENT_WORKDIR/BOSS/config 

• • see BOSS4MiniHowTo > BOSS4QueryingTaskStatus for the output of the boss query command
  • see the detailed status in $PRODAGENT_WORKDIR/JobTracking/ComponentLog
  • succesful jobs are in $PRODAGENT_WORKDIR/SuccessArchiveDir
  • failed jobs are in $PRODAGENT_WORKDIR/FailureArchiveDir

Barimonitoring

• Full description at ProdAgentBariMonitoring
• set export MYTESTAREA=/msa3/prodagent
• run $PRODAGENT_ROOT/src/php/BariMonitoring/install.sh
• output from the install script

  Installation finished
  Log in your mysql server as root and execute the following command,
  where ProdAgentUser and ProdAgentPassword has to be replaced with your
  settings as defined in fields user and passwd in block ProdAgentDB in
  your Production Agent configuration file.

  mysql> UPDATE mysql.user
         SET Password = OLD_PASSWORD('ProdAgentPassword')
         WHERE Host = 'localhost' AND User = 'ProdAgentUser';
  mysql> FLUSH PRIVILEGES;

  Start the deamon to get component log messages updated regularly :
  (cd /user/stdweird/prod/monitoring/apache/htdocs/PA; nohup sh ComponentLog.sh &)

  Start apache:
  /user/stdweird/prod/monitoring/apache/bin/apachectl start

  Set the password for restricted area:
  /user/stdweird/prod/monitoring/apache/bin/htpasswd -n prodagent >> /user/stdweird/prod/monitoring/apache/htdocs/PA/restricted_folder/.htpasswd

  Point your browser to http://localhost:8080/PA/index.php

  When necessary, stop the apache server:
  /user/stdweird/prod/monitoring/apache/bin/apachectl stop

• in apache/conf/httpd.conf add

  Listen 127.0.0.1:8080

• multiple instances of the monitoring
  • create new adodb cache dir

  mkdir -p /tmp/adodb_cache1

• • modify line in htdocs/PA/common/dbLib-FTS.php

  $ADODB_CACHE_DIR = "/tmp/adodb_cache1";

• • in apache/conf/httpd.conf

  Listen 127.0.0.1:8081
  Port 8081

• • point browser to port 2nd monitoring instance on port 8081

Links

• Full configuration explanation at the prodagent config page
• Original LCG Test Example
• ProdAgentDB instructions