GridStorageAccess: Difference between revisions

From T2B Wiki
Jump to navigation Jump to search
No edit summary
 
(18 intermediate revisions by the same user not shown)
Line 7: Line 7:


== General info ==
== General info ==
As dcache was designed for precious data, the files are immutable. This means that once they are written, they cannot be changed any more. So, if you want to make changes to a file on dcache, you need to first erase it and then write it anew.  
As dCache was designed for precious data, the files are immutable. This means that once they are written, they cannot be changed any more. So, if you want to make changes to a file on dCache, you need to first erase it and then write it anew.  
Our dCache instance is mounted on all the M machines and can be browsed via the /pnfs directory. If you want to find your personal directory, the structure is the following:
Our dCache instance is mounted on all the M machines and can be browsed via the /pnfs directory. If you want to find your personal directory, the structure is the following:
<pre>/pnfs/iihe/<Experiment>/store/user/<Username>  
<pre>/pnfs/iihe/<Experiment>/store/user/<Username>     <-- Replace <Username> and <Experiment> accordingly.</pre>
Fill in you username and the experiment you are affiliated with accordingly.</pre>
On the M machines as well as the whole cluster, /pnfs is mounted read-write via a protocol called 'nfs'. Please be aware that you can now inadvertently remove a large portion of your files. As it is a mass storage system, you can easily delete several TBs of data.  
On the M machines /pnfs is mounted read-only. This to protect to inadvertently remove a large portion of your files. As it is a mass storage system, you can easily delete sevela TBs of data. Therefore, writing and deleting files is done via only via grid enable command (see next section). These commands are mostly done via scripts, so the probability of an error is lessened. <br>
 
However, we have seen the need for a more simple read-write access to /pnfs. Therefore, a dedicated machine was instantiated:
Writing and deleting files can still be done via via grid enable commands (see next section). These should still provide the best read/write speeds, compared to nfs access. These commands are mostly done via scripts, so the probability of an error is lessened. <br>
<pre>rw.iihe.ac.be </pre>
You can log into this machine like any other M-machine, and now use the regular 'cp' and 'rm' commands. However, this machine is not like the other M-machines. Its sole purpose is to access /pnfs via the regular posix commands, so code development and job submission is prohibited.


== Before starting ==
== Before starting ==
In what follows, all the commands will require some type of authentication to access /pnfs. This is because these commands can be executed over WAN and your location is irrelevant. <br>
In what follows, most of the commands will require some type of authentication to access /pnfs. This is because these commands can be executed over WAN and your location is irrelevant. <br>
The way authentication is done on our mass storage instance is via an x509 proxy. This proxy is made through your grid certificate. If you do not have a grid certificate, see [https://t2bwiki.iihe.ac.be/Getting_a_certificate_for_the_T2 this page] on how to get one.<br>
The way authentication is done on our mass storage instance is via an x509 proxy. This proxy is made through your grid certificate. If you do not have a grid certificate, see [https://t2bwiki.iihe.ac.be/Getting_a_certificate_for_the_T2 this page] on how to get one.<br>
The command to make a grid proxy is:
The command to make a grid proxy is:
<pre>voms-proxy-init --voms MYEXPERIMENT</pre>
<pre>voms-proxy-init --voms <MYEXPERIMENT></pre>
 
Where ''<MYEXPERIMENT>'' is one of 'cms, icecube, beapps'
 
== Browser access ==
dCache now exposes files using the WebDav protocol. This means that the files are accessible to browse over https.<br>
For this, you need to have your certificate in your browser (to import your .p12 certificate, google is your friend).<br>
Then just point your browser to:
https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/
 
To be able to see and download your files.
 
 
dCache has an even more powerful web interface. It is called dCache View and can be accessed via:
https://maite.iihe.ac.be:3880/
Work is still in progress to make all actions work (12/2021).


Where MYEXPERIMENT is one of 'cms, icecube, beapps'


== Access via GFAL ==
== Access via GFAL ==
Line 28: Line 40:
<br>
<br>
=== gfal-commands ===
=== gfal-commands ===
If you want more information on the options that can be used, please use '''man gfal-command''' !
If you want more information on the options that can be used, please use the man pages of gfal !


Here are all the commands that can be used:
Here are all the commands that can be used:
Line 37: Line 49:


<br><br>
<br><br>
=== Usage ===
=== Usage ===
There are 2 types of file url:
There are 2 types of file url:
* '''Distant files''': their url is of the type srm://<name_of_server>:<port>/some/path, eg for IIHE:
* '''Distant files''': their url is of the type <protocol>://<name_of_server>:<port>/some/path, eg for IIHE:
srm://maite.iihe.ac.be:8443/pnfs/iihe/
:<pre>davs://maite.iihe.ac.be:2880/pnfs/iihe/</pre>
* '''Local files''': their url is of the type file://path_of_the_file, eg for IIHE:
* '''Local files''': their url is of the type file://path_of_the_file, eg for IIHE:
file:///user/$USER/MyFile.root
:<pre>file:///user/$USER/MyFile.root</pre>


[[File:Exclamation-mark.jpg|left|40x30px|line=1|]] Be careful, the number of '''/''' is very -very- important [[File:Exclamation-mark.jpg|40x30px|line=1|]]
[[File:Exclamation-mark.jpg|left|40x30px|line=1|]] Be careful, the number of '''/''' is very -very- important [[File:Exclamation-mark.jpg|40x30px|line=1|]]


*To get a list of all distant urls for all the Storage Elements, one can do:
*To get a list of all distant urls for all the Storage Elements, one can do:
::<pre> lcg-infosites --vo cms se </pre>
:<pre>lcg-infosites --is grid-bdii.desy.de --vo cms se </pre>
:or read more in the [[CERNGridUrls | CERN EOS urls page]].


=== Protocols ===
* '''https/WebDavs'''  [preferred] [try this one first]
::<pre>gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/</pre>
* '''xrootd'''  [preferred]
::<pre>gfal-ls root://maite.iihe.ac.be:1094/pnfs/iihe/cms/store/user/</pre>
* '''srm'''  [deprecated]
::<pre>gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/</pre>
* '''nfs'''  ''{local-only  |  no-cert}''
::<pre>gfal-ls /pnfs/iihe/cms/store/user/</pre>
* '''dcap'''  ''{local-only  |  no-cert}''
::<pre>gfal-ls dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user</pre>


=== Examples ===
=== Examples ===
*To list the contents of a directory ''/pnfs/iihe/cms'' :
*To list the contents of a directory ''/pnfs/iihe/cms'' :
::<pre> gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms </pre>
::<pre> gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms </pre>


* To create a directory:
* To create a directory:
::<pre> gfal-mkdir srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>
::<pre> gfal-mkdir davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/$USER/NewDir </pre>


*copy file from local disk to remote server  
*copy file from local disk to remote server  
::<pre> gfal-copy file:///user/$USER/MyFile.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/ </pre>
::<pre> gfal-copy file:///user/$USER/MyFile.root davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/$USER/ </pre>


* To copy a file from remote server to our Storage Element:
* To copy a file from remote server to our Storage Element:
::<pre> gfal-copy srm://srm-eoscms.cern.ch:8443/srm/v2/server?SFN=/eos/cms/store/group/comm_trigger/L1TrackTrigger/BE5D_620_SLHC6/singleMu/NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/odevroed/eosTransfer.root </pre>
::<pre> gfal-copy gsiftp://eosuserftp.cern.ch/eos/user/r/rougny/Documents/desktop.ini davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/rougny/ </pre>


* To delete a file on remote server
* To delete a file on remote server
Line 67: Line 97:


* To remove a directory and its entire content on remote server ?!? not working for now ?):   
* To remove a directory and its entire content on remote server ?!? not working for now ?):   
::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>  
::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>


=== Bulk file transfers ===
<br>
There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.
 
== Copy more than 1 file ==
 
==== Copy Directories ====
 
You can easily copy whole directories to/from our site using gfal commands. <br>
It is usually must faster than using scp or rsync commands.
gfal-copy -r [--dry-run] gsiftp://eosuserftp.cern.ch/eos/user/r/rougny/Documents/ davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/rougny/


Syntax:
The magic option is '''-r''' for recursive copying.<br>
<pre> gfal-copy -f --from-file files.txt file://$PWD </pre>
When you are sure you get what you want, remove the --dry-run option.<br>
where files.txt is a file where every line is a source in srm url syntax.


Make some tests with one line in datafile and make sure the srm url is OK for both source and destination before running over several files.
Note that by default, gfal-copy will not overwrite files already present at the destination. This means it is usually safe to run the command several times.<br>
If you want to force the copy over files already there, add the '''-f''' option to your gfal-copy command.


=== Copy directories from and to pnfs within the IIHE ===
<br>
A script to copy full directories to and from pnfs exists on the slc6 UI's:
==== Bulk copy from a list of files ====
<pre>
copyDirectoryPnfs.py
Move all files in a directory to or from pnfs
This script assumes that you copy within the IIHE
The script does not do recursive copying
Make sure you have a valid proxy, made with voms-proxy-init --voms cms:/cms/becms


Mandatory options:
There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.
--in=                : directory to copy from
<pre> gfal-copy -f [--dry-run] --from-file files.txt file://location/to/store/ </pre>
--out=                : directory to copy to
where files.txt is a file where every line is a source like:
Both directories need to be complete (i.e. including the /pnfs or /user part
<pre>davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer-1.root
davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer-2.root
... </pre>


example:
Make some tests with one line in datafile and make sure the url is OK for both source and destination before running over several files.<br>
copyDirectoryPnfs.py --out=/user/odevroed/newfile --in=/pnfs/iihe/cms/store/user/odevroed/newdir
When you are sure you get what you want, remove the --dry-run option.


Optional:
<br>
-h, --help            : print this help message
==== WebFTS web interface ====
</pre>
If you prefer using a web interface to copy files to/from our /pnfs, you can use CERN's [https://webfts.cern.ch/ WebFTS feature].<br>
Not all experiments are allowed to use it, but you can always make a request to have it included.


== Other ways to access the mass storage system ==
== Other ways to access the mass storage system ==
=== Read and copy access ===
=== Read and copy access ===
As stated in the introduction, dCache is an immutable file system, therefore files cannot be changed once they are written. Furthermore, for data protection, files are only available in read mode from /pnfs. <br>
 
Files can be accessed from pnfs in several ways.
As stated in the introduction, dCache is an immutable file system, therefore files cannot be changed once they are written.
* Via the regular 'cp' command
Files can be accessed from pnfs in several ways without the requirement of a grid certificate and grid tools.
<pre>cp /pnfs/iihe/cms/store/user/odevroed/DQMfile_83_1_hF2.root /user/odevroed </pre>
 
* Via the regular 'cp' command (prefer the rsync command below):
::<pre>cp /pnfs/iihe/cms/store/user/odevroed/DQMfile_83_1_hF2.root /user/odevroed </pre>
 
* Via the regular 'rsync' command:
::<pre>rsync -aP /pnfs/iihe/cms/store/user/odevroed/*.root /user/odevroed/</pre>


* Via the dcache copy command (dccp):
* Via the dcache copy command (dccp):
<pre>dccp dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user/odevroed/DQMfile_83_1_hF2.root ./ </pre>
::<pre>dccp dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user/odevroed/DQMfile_83_1_hF2.root ./ </pre>
 
* To open files directly using root, use eg
::<pre>root dcap://maite.iihe.ac.be/pnfs/iihe/some/file.root </pre>
 
::When reading out the root files, if is rather slow or it doesn't work at all, and nothing is wrong with the root file (e.g. in an interactive analysis mX machines) you can increase your dCache readahead buffer. Don't make the buffer larger than 50MB!
::To enlarge the buffer set this in you environment:<br>
::'''For csh'''
:::<pre>setenv DCACHE_RAHEAD 1</pre>
:::<pre>setenv DCACHE_RA_BUFFER 50000000</pre>


* To open files  using root, use eg
::'''For bash'''
<pre>root dcap://maite.iihe.ac.be/pnfs/iihe/some/file.root </pre>
:::<pre>export DCACHE_RAHEAD=true</pre>
When reading out the rootfiles is rather slow or it doesn't work at all and nothing is wrong with the root file (e.g. in an interactive analysis on beo or msa) you can increase your dCache readahead buffer. Don't make the buffer larger than 50MB! To enlarge the buffer set this in you environment :<br />
:::<pre>export DCACHE_RA_BUFFER=50000000</pre>
For csh:
<pre>
setenv DCACHE_RAHEAD 1
setenv DCACHE_RA_BUFFER 50000000
</pre>
For bash:
<pre>
export DCACHE_RAHEAD=true
export DCACHE_RA_BUFFER=50000000
</pre>


* Via the 'curl' command over https
* Via the 'curl' command over https
<pre>Copy from /pnfs:
:'''Copy from /pnfs:'''
curl -L --cert $X509_USER_PROXY --key $X509_USER_PROXY --cacert $X509_USER_PROXY --capath $X509_CERT_DIR  -O https://maite.iihe.ac.be:2880/cms/store/user/testing_transfer
::<pre>curl -L --cert $X509_USER_PROXY --key $X509_USER_PROXY --cacert $X509_USER_PROXY --capath $X509_CERT_DIR  -O https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/testing_transfer</pre>


Copy to /pnfs
:'''Copy to /pnfs:'''
::<pre>curl -L --cert $X509_USER_PROXY --key $X509_USER_PROXY --cacert $X509_USER_PROXY --capath $X509_CERT_DIR  -T testing_transfer  https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/testing_transfer_2</pre>


</pre>
::This is equivalent to issuing the gfal-cp command via the https protocol:
=== WebDav access ===
::<pre>gfal-copy testing_transfer https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/testing_transfer2</pre>
As seen in the last example in the previous section, dCache files are now seen over https. This means that they are also browsable via any web browser that support x509 certificates (like firefox, chrome and safari).<br>
The part of dCache that implements this functionality is called dCache View and can be seen via the following url:

Latest revision as of 08:38, 24 November 2022


This page describes how to handle data stored on our mass storage system.

Introduction

T2B has an ever increasing amount of mass storage available. This system hosts both the centrally produced datasets as well as the user produced data.
The mass storage is managed by a software called dCache. As this software is in full development, new features are added continuously. This page contains an overview of the most used features of the software.

General info

As dCache was designed for precious data, the files are immutable. This means that once they are written, they cannot be changed any more. So, if you want to make changes to a file on dCache, you need to first erase it and then write it anew. Our dCache instance is mounted on all the M machines and can be browsed via the /pnfs directory. If you want to find your personal directory, the structure is the following:

/pnfs/iihe/<Experiment>/store/user/<Username>     <-- Replace <Username> and <Experiment> accordingly.

On the M machines as well as the whole cluster, /pnfs is mounted read-write via a protocol called 'nfs'. Please be aware that you can now inadvertently remove a large portion of your files. As it is a mass storage system, you can easily delete several TBs of data.

Writing and deleting files can still be done via via grid enable commands (see next section). These should still provide the best read/write speeds, compared to nfs access. These commands are mostly done via scripts, so the probability of an error is lessened.

Before starting

In what follows, most of the commands will require some type of authentication to access /pnfs. This is because these commands can be executed over WAN and your location is irrelevant.
The way authentication is done on our mass storage instance is via an x509 proxy. This proxy is made through your grid certificate. If you do not have a grid certificate, see this page on how to get one.
The command to make a grid proxy is:

voms-proxy-init --voms <MYEXPERIMENT>

Where <MYEXPERIMENT> is one of 'cms, icecube, beapps'

Browser access

dCache now exposes files using the WebDav protocol. This means that the files are accessible to browse over https.
For this, you need to have your certificate in your browser (to import your .p12 certificate, google is your friend).
Then just point your browser to:

https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/

To be able to see and download your files.


dCache has an even more powerful web interface. It is called dCache View and can be accessed via:

https://maite.iihe.ac.be:3880/

Work is still in progress to make all actions work (12/2021).


Access via GFAL

GFAL is a wrapper around the latest grid commands. Learning to use it means that whatever middleware requires to be used in the future, you don't need to learn new commands (like srm, lcg, etc)

gfal-commands

If you want more information on the options that can be used, please use the man pages of gfal !

Here are all the commands that can be used:

  • gfal-ls: get information on a file
  • gfal-mkdir: remove a directory
  • gfal-rm: removes a file. To remove an entire directory, use -r
  • gfal-copy: copy files.



Usage

There are 2 types of file url:

  • Distant files: their url is of the type <protocol>://<name_of_server>:<port>/some/path, eg for IIHE:
davs://maite.iihe.ac.be:2880/pnfs/iihe/
  • Local files: their url is of the type file://path_of_the_file, eg for IIHE:
file:///user/$USER/MyFile.root

Be careful, the number of / is very -very- important

  • To get a list of all distant urls for all the Storage Elements, one can do:
lcg-infosites --is grid-bdii.desy.de --vo cms se 
or read more in the CERN EOS urls page.

Protocols

  • https/WebDavs [preferred] [try this one first]
gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/
  • xrootd [preferred]
gfal-ls root://maite.iihe.ac.be:1094/pnfs/iihe/cms/store/user/
  • srm [deprecated]
gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/
  • nfs {local-only | no-cert}
gfal-ls /pnfs/iihe/cms/store/user/
  • dcap {local-only | no-cert}
gfal-ls dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user

Examples

  • To list the contents of a directory /pnfs/iihe/cms :
 gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms 
  • To create a directory:
 gfal-mkdir davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/$USER/NewDir 
  • copy file from local disk to remote server
 gfal-copy file:///user/$USER/MyFile.root davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/$USER/ 
  • To copy a file from remote server to our Storage Element:
 gfal-copy gsiftp://eosuserftp.cern.ch/eos/user/r/rougny/Documents/desktop.ini davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/rougny/ 
  • To delete a file on remote server
 gfal-rm srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/MyFile.root 
  • To remove a directory and its entire content on remote server ?!? not working for now ?):
 gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir 


Copy more than 1 file

Copy Directories

You can easily copy whole directories to/from our site using gfal commands.
It is usually must faster than using scp or rsync commands.

gfal-copy -r [--dry-run] gsiftp://eosuserftp.cern.ch/eos/user/r/rougny/Documents/ davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/rougny/

The magic option is -r for recursive copying.
When you are sure you get what you want, remove the --dry-run option.

Note that by default, gfal-copy will not overwrite files already present at the destination. This means it is usually safe to run the command several times.
If you want to force the copy over files already there, add the -f option to your gfal-copy command.


Bulk copy from a list of files

There is an elegant way to run gfal-copy through several files. This is done using the --from-file option.

 gfal-copy -f [--dry-run] --from-file files.txt file://location/to/store/ 

where files.txt is a file where every line is a source like:

davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer-1.root
davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer-2.root
... 

Make some tests with one line in datafile and make sure the url is OK for both source and destination before running over several files.
When you are sure you get what you want, remove the --dry-run option.


WebFTS web interface

If you prefer using a web interface to copy files to/from our /pnfs, you can use CERN's WebFTS feature.
Not all experiments are allowed to use it, but you can always make a request to have it included.

Other ways to access the mass storage system

Read and copy access

As stated in the introduction, dCache is an immutable file system, therefore files cannot be changed once they are written. Files can be accessed from pnfs in several ways without the requirement of a grid certificate and grid tools.

  • Via the regular 'cp' command (prefer the rsync command below):
cp /pnfs/iihe/cms/store/user/odevroed/DQMfile_83_1_hF2.root /user/odevroed 
  • Via the regular 'rsync' command:
rsync -aP /pnfs/iihe/cms/store/user/odevroed/*.root /user/odevroed/
  • Via the dcache copy command (dccp):
dccp dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user/odevroed/DQMfile_83_1_hF2.root ./ 
  • To open files directly using root, use eg
root dcap://maite.iihe.ac.be/pnfs/iihe/some/file.root 
When reading out the root files, if is rather slow or it doesn't work at all, and nothing is wrong with the root file (e.g. in an interactive analysis mX machines) you can increase your dCache readahead buffer. Don't make the buffer larger than 50MB!
To enlarge the buffer set this in you environment:
For csh
setenv DCACHE_RAHEAD 1
setenv DCACHE_RA_BUFFER 50000000
For bash
export DCACHE_RAHEAD=true
export DCACHE_RA_BUFFER=50000000
  • Via the 'curl' command over https
Copy from /pnfs:
curl -L --cert $X509_USER_PROXY --key $X509_USER_PROXY --cacert $X509_USER_PROXY --capath $X509_CERT_DIR  -O https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/testing_transfer
Copy to /pnfs:
curl -L --cert $X509_USER_PROXY --key $X509_USER_PROXY --cacert $X509_USER_PROXY --capath $X509_CERT_DIR  -T testing_transfer  https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/testing_transfer_2
This is equivalent to issuing the gfal-cp command via the https protocol:
gfal-copy testing_transfer https://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/testing_transfer2