Maintenance script: Created page with " CMSSW Deployment Home == CMSSW Deployment in Detail == ''release-installer.pl'' takes care of creating and submitting installation and removal jobs..."

2015-08-26T12:28:22Z

Created page with " CMSSW Deployment Home == CMSSW Deployment in Detail == ''release-installer.pl'' takes care of creating and submitting installation and removal jobs..."

New page

[[CMSSWDeployment| CMSSW Deployment Home]]

== CMSSW Deployment in Detail ==

''release-installer.pl'' takes care of creating and submitting installation and removal jobs for sites where CMSSW releases are missing or obsolete.
It fetches the output of jobs afterwards and makes sure that the tags are published correctly at the sites.

However, don't use this script directly. Instead use the script ''ri.sh'', which takes care of your grid certificates automatically. You can avoid another additional 5 key strokes by using ''ri'', an alias for ''./ri.sh'', created on sourcing ''CreateProxy.sh''.

----

=== The ri functions ===

All commands are shown with the ''-c CEname'' option. With this option, the command is only executed for the site with the CEname in the argument. To run the command for all sites one after another, just leave away the ''-c CEname'' part

==== Submit deployment jobs ====

<pre>
ri -sc CEname
</pre>
#Check whether the site is available for CMSSW deployment? 
This is the case if
***No other deployment job is running (see further)
***The site does not need manual attention (see further)
***The site is no in maintenance
***There is no tag CMSSW_X_Y_Z_processing install at the site

If these requirements are not fulfilled the script stops
2. Check which releases are missing and which are obsolete 
This is done by answering following questions:
***Which releases are required and which are obsolete?
***Which versions are installed at the site?
***On which operating system does the site run the site? (CMSSW >= 3_4 requires SL5)
***What are the manual requirements for this site? (see further)

A list of missing and obsolete releases is displayed

3. Creating and installation and removal job
***For obsolete releases, ''ri'' sets the tag form CMSSW_X_Y_Z to CMSSW_X_Y_Z_remove_scheduled
***The obsolete release will only be removed on the first ''ri -sc CEname'' 100 hours after the original submission
***For missing releases a tag CMSSW_X_Y_Z_processing_install is added to the site
***A grid job with all installation and removal commands is send to the site
***At the end of the job, CMSSW_X_Y_Z_processing_install tags are replaced by CMSSW_X_Y_Z, the CMSSW_X_Y_Z_remove_scheduled tags are removed
(if the release has been removed at least)
***The site gets a subdirectory named ''CEname'' in the directory ''~/cms/release-installer/running''
As long as this directory exists, no new job can be submitted to the site

==== Checking the site's missing and obsolete releases ====
<pre>
ri -dc CEname
</pre>
executes point 1 and 2 of ''ri -sc CEname'' (see above)

==== Checking the status of running jobs ====
<pre>
ri -l
</pre>

==== Fetching the output of deployment jobs ====
<pre>
ri -f
</pre>
*''~/cms/release-installer/running/CEname'' is moved to ''~/cms/release-installer/running/done/CEname'' where the output is stored
*If the job was not successful (exit code not 0) ''~/cms/release-installer/done/CEname'' is copied to ''~/cms/release-installer/running/done/CEname''
As long as this last directory exists, no jobs can be submitted to this site. On ''ri -sc CEname'' the message "site needs manual intervention" will be displayed

The -f can be combined with the -s option:
<pre>
ri -fs
</pre>
Then, directly after fetching the output of the finished jobs, a new submission round will start.

==== Stealing others' deployment jobs ====
<pre>
-6 otheruser
</pre>
Monitoring or fetching jobs submitted by others is not allowed by default. 
It can be done using the option -6 (666 = EVIL), stealing actually the proxy of the job submitter.

----
=== Monitoring the overall deployment status ===
The most important tools to monitor the deployment status
*''ri -d'' shows which releases are missing/obsolete at all sites(see above)
*''ri -l'' shows at which sites jobs are running (see above)
*the [https://lcg-sam.cern.ch:8443/sam/sam.py sam pages] (Service Availability Monitoring) 
*the [http://lxarda16.cern.ch/dashboard/request.py/latestresultssmry?siteSelect3=T3T2T1&serviceTypeSelect3=vo&sites=T1_DE_KIT&sites=T1_ES_PIC&sites=T1_FR_CCIN2P3&sites=T1_IT_CNAF&sites=T1_TW_ASGC&sites=T1_UK_RAL&sites=T1_US_FNAL&sites=T2_AT_Vienna&sites=T2_BE_IIHE&sites=T2_BE_UCL&sites=T2_BR_SPRACE&sites=T2_BR_UERJ&sites=T2_CH_CSCS&sites=T2_CN_Beijing&sites=T2_DE_DESY&sites=T2_DE_RWTH&sites=T2_EE_Estonia&sites=T2_ES_CIEMAT&sites=T2_ES_IFCA&sites=T2_FI_HIP&sites=T2_FR_CCIN2P3&sites=T2_FR_GRIF_IRFU&sites=T2_FR_GRIF_LLR&sites=T2_FR_IPHC&sites=T2_HU_Budapest&sites=T2_IN_TIFR&sites=T2_IT_Bari&sites=T2_IT_Legnaro&sites=T2_IT_Pisa&sites=T2_IT_Rome&sites=T2_KR_KNU&sites=T2_PK_NCP&sites=T2_PL_Cracow&sites=T2_PL_Warsaw&sites=T2_PT_LIP_Lisbon&sites=T2_PT_NCG_Lisbon&sites=T2_RU_IHEP&sites=T2_RU_INR&sites=T2_RU_ITEP&sites=T2_RU_JINR&sites=T2_RU_PNPI&sites=T2_RU_RRC_KI&sites=T2_RU_SINP&sites=T2_TR_METU&sites=T2_TW_Taiwan&sites=T2_UA_KIPT&sites=T2_UK_London_Brunel&sites=T2_UK_London_IC&sites=T2_UK_SGrid_Bristol&sites=T2_UK_SGrid_RALPP&sites=T2_US_Caltech&sites=T2_US_Florida&sites=T2_US_MIT&sites=T2_US_Nebraska&sites=T2_US_Purdue&sites=T2_US_UCSD&sites=T2_US_Wisconsin&sites=T3_CH_PSI&sites=T3_CN_PKU&sites=T3_CO_Uniandes&sites=T3_DE_Karlsruhe&sites=T3_ES_Oviedo&sites=T3_FR_IPNL&sites=T3_GR_Demokritos&sites=T3_GR_IASA&sites=T3_GR_Ioannina&sites=T3_IT_Napoli&sites=T3_IT_Padova&sites=T3_IT_Perugia&sites=T3_IT_Trieste&sites=T3_TW_NCU&sites=T3_UK_London_QMUL&sites=T3_UK_London_RHUL&sites=T3_UK_London_UCL&sites=T3_UK_ScotGrid_ECDF&sites=T3_UK_ScotGrid_GLA&sites=T3_UK_SGrid_Oxford&sites=T3_US_Colorado&sites=T3_US_Cornell&sites=T3_US_FIT&sites=T3_US_FNALLPC&sites=T3_US_FNALXEN&sites=T3_US_Omaha&sites=T3_US_Rutgers&sites=T3_US_TTU&sites=T3_US_UCD&sites=T3_US_UCLA&sites=T3_US_UCR&sites=T3_US_UMD&sites=T3_US_Vanderbilt&services=CE&tests=1301&tests=133&tests=111&tests=6&tests=1261&tests=76&tests=64&tests=20&exitStatus=all sam dashboard]
a nice overview of the sam pages
*releases.html
<pre>
./publish_to_html.pl ~/.globus_<YourUserName>/cert.p12 releases.html
</pre>
This script will create a very convenient overview of the deployment status, showing for each site which releases are missing, how old the latest sam test is and which operating system is installed. 
You want to have the correct lay out, so download the .css file to the same directory as releases.html
<pre>
wget http://www.desy.de/~wbehrenh/releases.css .
</pre>
*check the published tags at a site
<pre>
lcg-tags --vo cms --ce CEname --list
</pre>
*plot the performance of the CMSSW installation
<pre>
./showtimes.pl CMSSW_X_Y_Z > plots/CMSSW_X_Y_Z.dat
./showtimes.pl CMSSW_A_B_C > plots/CMSSW_A_B_C.dat
cd plots
root -b -q -l 'plottimes.C+("CMSSW_X_Y_Z,CMSSW_A_B_C","lukas.png")'
</pre>

----
=== Editing Tags ===
*add tags for releases CMSSW_A_B_C and CMSSW_D_E_F at site CEname:
<pre>
lcg-tags --vo cms --ce CEname --add --tags VO-cms-CMSSW_A_B_C,VO-cms-CMSSW_D_E_F
</pre>
*remove tags for releases CMSSW_A_B_C and CMSSW_D_E_F site CEname
<pre>
lcg-tags --vo cms --ce CEname --remove --tags VO-cms-CMSSW_A_B_C,VO-cms-CMSSW_D_E_F
</pre>
----
=== Asking site administrators for support ===
Open a [https://savannah.cern.ch savannah] ticket
*login
*go to [https://savannah.cern.ch/my/groups.php my groups]
*on the page for the appropriate group there is a link to open a new ticket
***Category: Facility Operations
***Assigned to: there should be an entry like ''cmscompinfrasup-<sitename>''
***Use GGUS: no
***Site: site
***Summary: drop "central CMSSW installation", the site's name "T1_CO_BLA", and the problem
***Add "cmscompinfrasup-cmsswdeploy" to "Mail Notification Carbon-Copy List"

{{TracNotice|{{PAGENAME}}}}

CMSSWDeploymentInDetail - Revision history

Maintenance script: Created page with " CMSSW Deployment Home == CMSSW Deployment in Detail == ''release-installer.pl'' takes care of creating and submitting installation and removal jobs..."