Maintenance script: Created page with " == Retrieve lost files from datasets == PageOutline ---- === Introduction === Some files in datasets are not copied correctly to the T2's. It is at most a few files p..."

2015-08-26T12:28:31Z

Created page with " == Retrieve lost files from datasets == PageOutline ---- === Introduction === Some files in datasets are not copied correctly to the T2's. It is at most a few files p..."

New page

== Retrieve lost files from datasets ==
[[PageOutline]]
----
=== Introduction ===
Some files in datasets are not copied correctly to the T2's. It is at most a few files per dataset. A few scripts are in place to easily put them where they belong.

=== Identify ===
First, Go to [https://cms-popularity.cern.ch/popdb/popularity/corruptedFiles the corrupted files page] and see if there are any files from our site. If so, click on the "Get source JSON" tab

=== create the necessary files ===

Put the download file in /user/odevroed/Get_Lost_Files<br>
Edit the file and remove all the html tags. This leaves only the json part. For further reference in this document, I called it cms-popularity.cern.ch.json<br>
Run the script a first time (with option 1 :) ):

<pre>
python /user/odevroed/bin/download_missing_dataset_files.py 1 ./cms-popularity.cern.ch.json
</pre>

This will generate the file "files_to_retrieve" which contains only the files from our site that are not present (In case it was just a coincidence).<br>
Go to this file and via [https://cmsweb.cern.ch/das/ DAS], find out at which site they reside. Usually you can aggregate many files from the list, present at one single site.<br>
Put this list together in a new file, let's say "files_per_site" and give this as an argument for the second pass of the script (replace by the appropriate site name):

<pre>
python /user/odevroed/bin/download_missing_dataset_files.py 2 T2_CH_CSCS ./files_per_site
</pre>
The result of the script is the output:

<pre>
fetching from site:
srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/cms/trivcat

The files are ready to be transferred
First, copy the files to us:
srmcp -debug -streams_num=1 -2 -copyjobfile=transfer_to_us
Then, put them on storage:
srmcp -debug -streams_num=1 -2 -copyjobfile=put_on_storage
</pre>
then do what the script told you to do :)

{{TracNotice|{{PAGENAME}}}}

GetLostFiles - Revision history

Maintenance script: Created page with " == Retrieve lost files from datasets == PageOutline ---- === Introduction === Some files in datasets are not copied correctly to the T2's. It is at most a few files p..."