Rucio
Rucio web requests are handled via https://cms-rucio-webui.cern.ch/r2d2/request
Some Rucio vocabulary:
RSE: Rucio Storage Element. In our case this is T2_BE_IIHE (and not the name of the se) DID: Data identyfier; is used to represent any set of file, dataset or container identifier. Data identifiers are unique over time. So they are never reused. rule: a data transfer. This includes DID, RSE and possible end time when the files are deleted again. lock: a tranfereed file scope: in our case the scope is cms.
If you want to make a request (rule in rucio), follow the following tuturial.
This are basic instructions on rucio.
Find the full info on: https://twiki.cern.ch/twiki/bin/view/CMSPublic/Rucio
- You need to be on sl7 for the commands to work
- initialise the env:
source /cvmfs/cms.cern.ch/cmsset_default.sh source /cvmfs/cms.cern.ch/rucio/setup-py3.sh voms-proxy-init -voms cms -rfc -valid 192:00 export RUCIO_ACCOUNT=`whoami`
- in rucio, a transfer is called a rule. In order to create a rule:
rucio add-rule cms:/CMS/DATA/SET/NAME 1 T2_MY_SITE rucio add-rule cms:/CMS/DATA/SET/NAME#BLOCK-NAME 1 T2_MY_SITE
- if there is a lot of data, add
--asynchronous
- users can also request rules, who are then to be approved by the admins. This is done with 2 extra modifiers:
--ask-approval
and
--lifetime (in seconds; for reference, 30 days is 2592000 seconds)
- check your rules:
rucio list-rules --account $RUCIO_ACCOUNT List all the rules you have and their state rucio rule-info [RULE_HASH] monitor the progress of your rule and any transfers it may have initiated
Example:
rucio list-rules --account $RUCIO_ACCOUNT ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC) -------------------------------- --------- ------------------------------------------------------------------------------------------------------------------------------------------------ ---------------------- ---------------- -------- ------------------- ------------------- eafda759f4ae4128b49fede980db5622 odevroed cms:/Neutrino_E-10_gun/RunIISummer17PrePremix-PUAutumn18_102X_upgrade2018_realistic_v15-v1/GEN-SIM-DIGI-RAW#0149acf0-6b06-43c9-b99f-dfc531b6eecb OK[235/0/0] T2_BE_IIHE 1 2021-02-19 15:11:47 2021-01-20 15:11:47 rucio rule-info eafda759f4ae4128b49fede980db5622 Id: eafda759f4ae4128b49fede980db5622 Account: odevroed Scope: cms Name: /Neutrino_E-10_gun/RunIISummer17PrePremix-PUAutumn18_102X_upgrade2018_realistic_v15-v1/GEN-SIM-DIGI-RAW#0149acf0-6b06-43c9-b99f-dfc531b6eecb RSE Expression: T2_BE_IIHE Copies: 1 State: OK Locks OK/REPLICATING/STUCK: 235/0/0 Grouping: DATASET Expires at: 2021-02-19 15:11:47 Locked: False Weight: None Created at: 2021-01-20 15:11:47 Updated at: 2021-01-20 15:36:07 Error: None Subscription Id: None Source replica expression: None Activity: User Subscriptions Comment: None Ignore Quota: True Ignore Availability: False Purge replicas: False Notification: NO End of life: None Child Rule Id: None
More info here
- requests can also be done for users, but I have not tested this.
- users can also be given quota’s
- datasets can be grouped into containers. This can be handy if several datasets are needed for a specific analysis. All of then can then also be removed together if the analysis is finished. This is done via rucio containers. More info here.
At the moment of writing this document, these were the settings for our site:
rucio list-rse-attributes T2_BE_IIHE T2_BE_IIHE: True cms_type: real country: BE ddm_quota: 3355000000000000 fts: https://fts3-cms.cern.ch:8446 lfn2pfn_algorithm: cmstfc pnn: T2_BE_IIHE quota_approvers: odevroed,rougny,srugovac reaper: True region: C rule_approvers: odevroed,rougny,srugovac source_for_total_space: static source_for_used_space: rucio tier: 2