GridStorageAccess - Revision history

Admin: /* Before starting */

2025-03-17T15:18:10Z

Before starting

← Older revision		Revision as of 15:18, 17 March 2025
Line 21:		Line 21:

	Where ''<MYEXPERIMENT>'' is one of 'cms, icecube, beapps'		Where ''<MYEXPERIMENT>'' is one of 'cms, icecube, beapps'

			NOTE: the proxy is created in /tmp, so it is local to the machine. For batch jobs to execute grid commands too, set in your '''.bashrc''' file:
			<pre>
			export X509_USER_PROXY=/user/$USER/x509up_u$UID
			</pre>

	== Browser access ==		== Browser access ==

Admin: /* gfal-commands */

2022-11-24T08:38:01Z

gfal-commands

← Older revision		Revision as of 08:38, 24 November 2022
Line 40:		Line 40:
	<br>		<br>
	=== gfal-commands ===		=== gfal-commands ===
	If you want more information on the options that can be used, please use ~~'''~~man gfal~~-command'''~~ !		If you want more information on the options that can be used, please use the man pages of gfal !

	Here are all the commands that can be used:		Here are all the commands that can be used:
Line 49:		Line 49:

	<br><br>		<br><br>

	=== Usage ===		=== Usage ===
	There are 2 types of file url:		There are 2 types of file url:

Admin: /* General info */

2022-11-24T08:35:11Z

General info

← Older revision		Revision as of 08:35, 24 November 2022
Line 7:		Line 7:

	== General info ==		== General info ==
	As ~~dcache~~ was designed for precious data, the files are immutable. This means that once they are written, they cannot be changed any more. So, if you want to make changes to a file on ~~dcache~~, you need to first erase it and then write it anew.		As dCache was designed for precious data, the files are immutable. This means that once they are written, they cannot be changed any more. So, if you want to make changes to a file on dCache, you need to first erase it and then write it anew.
	Our dCache instance is mounted on all the M machines and can be browsed via the /pnfs directory. If you want to find your personal directory, the structure is the following:		Our dCache instance is mounted on all the M machines and can be browsed via the /pnfs directory. If you want to find your personal directory, the structure is the following:
	<pre>/pnfs/iihe/<Experiment>/store/user/<Username> <-- Replace <Username> and <Experiment> accordingly.</pre>		<pre>/pnfs/iihe/<Experiment>/store/user/<Username> <-- Replace <Username> and <Experiment> accordingly.</pre>
	On the M machines as well as the whole cluster, /pnfs is ~~now~~ mounted read-write ~~(this is new!)~~. Please be aware that you can now inadvertently remove a large portion of your files. As it is a mass storage system, you can easily delete several TBs of data.		On the M machines as well as the whole cluster, /pnfs is mounted read-write via a protocol called 'nfs'. Please be aware that you can now inadvertently remove a large portion of your files. As it is a mass storage system, you can easily delete several TBs of data.

	Writing and deleting files can still be done via via grid enable commands (see next section). These should still provide the best read/write speeds compared to nfs access. These commands are mostly done via scripts, so the probability of an error is lessened. <br>		Writing and deleting files can still be done via via grid enable commands (see next section). These should still provide the best read/write speeds, compared to nfs access. These commands are mostly done via scripts, so the probability of an error is lessened. <br>

	== Before starting ==		== Before starting ==

Admin: /* General info */

2022-11-23T13:46:04Z

General info

← Older revision		Revision as of 13:46, 23 November 2022
Line 13:		Line 13:

	Writing and deleting files can still be done via via grid enable commands (see next section). These should still provide the best read/write speeds compared to nfs access. These commands are mostly done via scripts, so the probability of an error is lessened. <br>		Writing and deleting files can still be done via via grid enable commands (see next section). These should still provide the best read/write speeds compared to nfs access. These commands are mostly done via scripts, so the probability of an error is lessened. <br>


	~~There is a dedicated machine for file copying to /pnfs:~~
	~~<pre>rw.iihe.ac.be </pre>~~
	You can log into this machine like any other M-machine, and use the regular 'cp' and 'rm' commands. However, this machine is not like the other M-machines. Its sole purpose is to access /pnfs via the regular posix commands, so code development and job submission is prohibited.

	== Before starting ==		== Before starting ==

Admin: /* Examples */

2022-11-23T13:44:31Z

Examples

← Older revision		Revision as of 13:44, 23 November 2022
Line 95:		Line 95:

	* To copy a file from remote server to our Storage Element:		* To copy a file from remote server to our Storage Element:
	::<pre> gfal-copy ~~srm~~://~~srm-eoscms~~.cern.ch~~:8443/srm/v2/server?SFN=~~/eos/~~cms/store/group/comm_trigger~~/~~L1TrackTrigger~~/~~BE5D_620_SLHC6~~/~~singleMu~~/~~NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499~~.~~root~~ davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/~~odevroed~~/~~eosTransfer.root~~ </pre>		::<pre> gfal-copy gsiftp://eosuserftp.cern.ch/eos/user/r/rougny/Documents/desktop.ini davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/rougny/ </pre>

	* To delete a file on remote server		* To delete a file on remote server
Line 104:		Line 104:

	<br>		<br>

	== Copy more than 1 file ==		== Copy more than 1 file ==

Admin: /* Usage */

2022-11-23T13:31:43Z

Usage

← Older revision		Revision as of 13:31, 23 November 2022
Line 57:		Line 57:
	There are 2 types of file url:		There are 2 types of file url:
	* '''Distant files''': their url is of the type <protocol>://<name_of_server>:<port>/some/path, eg for IIHE:		* '''Distant files''': their url is of the type <protocol>://<name_of_server>:<port>/some/path, eg for IIHE:
	::<pre>davs://maite.iihe.ac.be:2880/pnfs/iihe/</pre>		:<pre>davs://maite.iihe.ac.be:2880/pnfs/iihe/</pre>
	* '''Local files''': their url is of the type file://path_of_the_file, eg for IIHE:		* '''Local files''': their url is of the type file://path_of_the_file, eg for IIHE:
	::<pre>file:///user/$USER/MyFile.root</pre>		:<pre>file:///user/$USER/MyFile.root</pre>

	[[File:Exclamation-mark.jpg\|left\|40x30px\|line=1\|]] Be careful, the number of '''/''' is very -very- important [[File:Exclamation-mark.jpg\|40x30px\|line=1\|]]		[[File:Exclamation-mark.jpg\|left\|40x30px\|line=1\|]] Be careful, the number of '''/''' is very -very- important [[File:Exclamation-mark.jpg\|40x30px\|line=1\|]]

	*To get a list of all distant urls for all the Storage Elements, one can do:		*To get a list of all distant urls for all the Storage Elements, one can do:
	::<pre>lcg-infosites --is grid-bdii.desy.de --vo cms se </pre>		:<pre>lcg-infosites --is grid-bdii.desy.de --vo cms se </pre>
	::or read more in the [[CERNGridUrls \| CERN urls page]].		:or read more in the [[CERNGridUrls \| CERN EOS urls page]].

	=== Protocols ===		=== Protocols ===

Admin: /* Access via GFAL */

2022-11-23T13:29:56Z

Access via GFAL

← Older revision		Revision as of 13:29, 23 November 2022
Line 64:		Line 64:

	*To get a list of all distant urls for all the Storage Elements, one can do:		*To get a list of all distant urls for all the Storage Elements, one can do:
	::<pre> lcg-infosites --is grid-bdii.desy.de --vo cms se </pre>		::<pre>lcg-infosites --is grid-bdii.desy.de --vo cms se </pre>
	::or read more in the [[CERNGridUrls \| CERN urls page]].		::or read more in the [[CERNGridUrls \| CERN urls page]].

Line 70:		Line 70:

	* '''https/WebDavs''' [preferred] [try this one first]		* '''https/WebDavs''' [preferred] [try this one first]
	gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/		::<pre>gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/</pre>

	* '''xrootd''' [preferred]		* '''xrootd''' [preferred]
	gfal-ls root://maite.iihe.ac.be:1094/pnfs/iihe/cms/store/user/		::<pre>gfal-ls root://maite.iihe.ac.be:1094/pnfs/iihe/cms/store/user/</pre>

	* '''srm''' [deprecated]		* '''srm''' [deprecated]
	gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/		::<pre>gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/</pre>

	* '''nfs''' ''{local-only \| no-cert}''		* '''nfs''' ''{local-only \| no-cert}''
	gfal-ls /pnfs/iihe/cms/store/user/		::<pre>gfal-ls /pnfs/iihe/cms/store/user/</pre>

	* '''dcap''' ''{local-only \| no-cert}''		* '''dcap''' ''{local-only \| no-cert}''
	gfal-ls dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user		::<pre>gfal-ls dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user</pre>

	=== Examples ===		=== Examples ===
Line 104:		Line 104:

	<br>		<br>
	=== Copy more than 1 file ===		== Copy more than 1 file ==

	===== Copy Directories =====		==== Copy Directories ====

	You can easily copy whole directories to/from our site using gfal commands. <br>		You can easily copy whole directories to/from our site using gfal commands. <br>
Line 119:		Line 119:

	<br>		<br>
	=====Bulk copy from a list of files=====		==== Bulk copy from a list of files ====

	There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.		There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.
Line 131:		Line 131:
	When you are sure you get what you want, remove the --dry-run option.		When you are sure you get what you want, remove the --dry-run option.

			<br>
			==== WebFTS web interface ====
			If you prefer using a web interface to copy files to/from our /pnfs, you can use CERN's [https://webfts.cern.ch/ WebFTS feature].<br>
			Not all experiments are allowed to use it, but you can always make a request to have it included.

	== Other ways to access the mass storage system ==		== Other ways to access the mass storage system ==

Admin at 13:22, 23 November 2022

2022-11-23T13:22:11Z

Show changes

Admin: /* Bulk file transfers */

2022-11-23T12:59:41Z

Bulk file transfers

← Older revision		Revision as of 12:59, 23 November 2022
Line 102:		Line 102:
	::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>		::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>

	=== Bulk file transfers ===		=== Bulk file transfers or Whole directory transfers ===

			<br>
			====='''Bulk copy from a list of files:'''=====

	There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.		There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.
			<pre> gfal-copy -f [--dry-run] --from-file files.txt file://location/to/store/ </pre>
			where files.txt is a file where every line is a source like:
			<pre>davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer-1.root
			davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer-2.root
			... </pre>

			Make some tests with one line in datafile and make sure the url is OK for both source and destination before running over several files.<br>
			When you are sure you get what you want, remove the --dry-run option.

			<br>
			====='''Copy Directories:'''=====

			You can easily copy whole directories to/from our site using gfal commands. <br>
			It is usually must faster than using scp or rsync commands.
			gfal-copy -r [--dry-run] gsiftp://eosuserftp.cern.ch/eos/user/r/rougny/Documents/ davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/rougny/

	~~Syntax:~~		The magic option is '''-r''' for recursive copying.<br>
	<~~pre~~> ~~gfal-copy -f~~ --~~from~~-~~file files~~.~~txt file://$PWD~~ <~~/pre~~>		When you are sure you get what you want, remove the --dry-run option.<br>
	~~where files.txt is a file where every line is a source in srm url syntax.~~

	~~Make some tests with one line in datafile and make sure~~ the ~~srm url~~ is ~~OK for both source and destination before running~~ over ~~several~~ files.		Note that by default, gfal-copy will not overwrite files already present at the destination. This means it is usually safe to run the command several times.<br>
			If you want to force the copy over files already there, add the '''-f''' option to your gfal-copy command.

	=== Copy directories from and to pnfs within the IIHE ===		=== Copy directories from and to pnfs within the IIHE ===

Admin: /* Examples */

2022-11-23T12:43:17Z

Examples

← Older revision		Revision as of 12:43, 23 November 2022
Line 85:		Line 85:
	=== Examples ===		=== Examples ===
	*To list the contents of a directory ''/pnfs/iihe/cms'' :		*To list the contents of a directory ''/pnfs/iihe/cms'' :
	::<pre> gfal-ls ~~srm~~://maite.iihe.ac.be:~~8443~~/pnfs/iihe/cms </pre>		::<pre> gfal-ls davs://maite.iihe.ac.be:2880/pnfs/iihe/cms </pre>

	* To create a directory:		* To create a directory:
	::<pre> gfal-mkdir ~~srm~~://maite.iihe.ac.be:~~8443~~/pnfs/iihe/cms/store/user/$USER/NewDir </pre>		::<pre> gfal-mkdir davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/$USER/NewDir </pre>

	*copy file from local disk to remote server		*copy file from local disk to remote server
	::<pre> gfal-copy file:///user/$USER/MyFile.root ~~srm~~://maite.iihe.ac.be:~~8443~~/pnfs/iihe/cms/store/user/$USER/ </pre>		::<pre> gfal-copy file:///user/$USER/MyFile.root davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/$USER/ </pre>

	* To copy a file from remote server to our Storage Element:		* To copy a file from remote server to our Storage Element:
	::<pre> gfal-copy srm://srm-eoscms.cern.ch:8443/srm/v2/server?SFN=/eos/cms/store/group/comm_trigger/L1TrackTrigger/BE5D_620_SLHC6/singleMu/NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499.root ~~srm~~://maite.iihe.ac.be:~~8443~~/pnfs/iihe/cms/store/user/odevroed/eosTransfer.root </pre>		::<pre> gfal-copy srm://srm-eoscms.cern.ch:8443/srm/v2/server?SFN=/eos/cms/store/group/comm_trigger/L1TrackTrigger/BE5D_620_SLHC6/singleMu/NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499.root davs://maite.iihe.ac.be:2880/pnfs/iihe/cms/store/user/odevroed/eosTransfer.root </pre>

	* To delete a file on remote server		* To delete a file on remote server
Line 100:		Line 100:

	* To remove a directory and its entire content on remote server ?!? not working for now ?):		* To remove a directory and its entire content on remote server ?!? not working for now ?):
	::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>		::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>


	=== Bulk file transfers ===		=== Bulk file transfers ===