T2B Wiki - User contributions [en-gb]

Cluster Overview

2016-11-04T15:58:45Z

Romain Rougny: /* Job Deletion */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : mshort [ m0 , m1 , m2 , m3 ] , mlong [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@mshort.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>mshort.iihe.ac.be : m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be </pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>mlong.iihe.ac.be : m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 

== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | highbw
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Subgroup of localgrid, subgroup of highmem, with WNs having 10Gb/s bandwidth access to storage. 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3512 
| nowrap="nowrap" align="center" | 1984 
| nowrap="nowrap" align="center" | 448 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="3" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="3"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="3"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

=== Job Deletion ===
Use the following command:
qdel <JOBID>
To delete all your jobs, be patient while using the following line:
for j in $(qselect -u $USER);do timeout 3 qdel -a $j;done

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Getting a certificate for the T2

2016-11-04T12:46:28Z

Romain Rougny: /* First time certificate */

If you need grid access on the T2, please follow all the steps below: 

=== First time certificate ===
# [[Obtaining_a_certificate | Get a certificate (new)]]. ''If you already have a CERN grid certificate, you can temporarily use it. But for accounting reasons, we need you to get a Belgian certificate ''
# Extract your certificate from your browser: '''[[Chrome]]''' | '''[[Firefox]]'''
# [[certificate_to_UI | Put your certificate on the UIs]]
# Register to the VO
#:* '''CMS'''
#:*# [[Register_to_the_CMS_VO|Register to the CMS VO]]
#:*# [[SiteDB | Check if your certificate is ok on SiteDB]]. Note the DN.
#:*# [[CERN_certificate_management | Check that your certificate is the only one registered on the CERN website.]]
#:* '''IceCube'''
#:** [[Register_to_the_IceCube_VO|Register to the IceCube VO]]
#:* '''Solid''', '''Others'''
#:** [[Register_to_the_Beapps_VO|Register to the Belgian VO (beapps) ]]
# Send a mail to the T2B support (grid_adminATlistserv.vub.ac.be) with your DN in order to have write access on the T2.
# [[Check_Certificate_UIs | Check if everything works fine on the mX machines]]

=== Certificate renewal ===
As we switched the company who provides us with certificates, you need to follow the 'First time certificate' to have a new one, this until December 2016 at least.

Register to the Beapps VO

2016-11-04T12:45:25Z

Romain Rougny: Created page with "* Register with your certificate here : https://voms01.begrid.be:8443/voms/beapps/user/home.action * Test on m machines : <pre> voms-proxy-init --voms beapps </pre>"

* Register with your certificate here :

https://voms01.begrid.be:8443/voms/beapps/user/home.action

* Test on m machines :

<pre>
voms-proxy-init --voms beapps
</pre>

Getting a certificate for the T2

2016-11-04T12:42:05Z

Romain Rougny: /* First time certificate */

If you need grid access on the T2, please follow all the steps below: 

=== First time certificate ===
# [[Obtaining_a_certificate | Get a certificate (new)]]. ''If you already have a CERN grid certificate, you can temporarily use it. But for accounting reasons, we need you to get a Belgian certificate ''
# Extract your certificate from your browser: '''[[Chrome]]''' | '''[[Firefox]]'''
# [[certificate_to_UI | Put your certificate on the UIs]]
# Register to the VO
#:* CMS
#:*# [[Register_to_the_CMS_VO|Register to the CMS VO]]
#:*# [[SiteDB | Check if your certificate is ok on SiteDB]]. Note the DN.
#:*# [[CERN_certificate_management | Check that your certificate is the only one registered on the CERN website.]]
#:* IceCube
#:** [[Register_to_the_IceCube_VO|Register to the IceCube VO]]
#:* Solid, Others
#:** [[Register_to_the_Beapps_VO|Register to the Belgian VO (beapps) ]]
# Send a mail to the T2B support (grid_adminATlistserv.vub.ac.be) with your DN in order to have write access on the T2.
# [[Check_Certificate_UIs | Check if everything works fine on the mX machines]]

=== Certificate renewal ===
As we switched the company who provides us with certificates, you need to follow the 'First time certificate' to have a new one, this until December 2016 at least.

Cluster Overview

2016-11-03T15:04:21Z

Romain Rougny: /* Job Deletion */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : mshort [ m0 , m1 , m2 , m3 ] , mlong [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@mshort.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>mshort.iihe.ac.be : m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be </pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>mlong.iihe.ac.be : m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 

== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | highbw
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Subgroup of localgrid, subgroup of highmem, with WNs having 10Gb/s bandwidth access to storage. 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3512 
| nowrap="nowrap" align="center" | 1984 
| nowrap="nowrap" align="center" | 448 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="3" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="3"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="3"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

=== Job Deletion ===
Use the following command:
qdel <JOBID>
To delete all your jobs, be patient while using the following line:
qdel $(qselect -u $USER -s EHQRTW)

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

CernLxplus

2016-10-27T10:29:47Z

Romain Rougny: /* How to connect without password to lxplus */

=== How to connect without password to lxplus ===
* The CERN wiki page is [http://linux.web.cern.ch/linux/docs/kerberos-access.shtml here]
* On '''ubuntu''', you need to:
** install kerberos
sudo apt-get install krb5-user
with domain: CERN.CH
:* Copy to '''/etc/krb5.conf''' the config file taken from CERN [http://linux.web.cern.ch/linux/docs/krb5.conf here].
:* Do a '''kinit''' . It should ask you for you CERN password, specifying yourlogin@CERN.CH. If this is wrong, specify it yourself with '''kinit yourlogin@CERN.CH'''
::: This should give you a valid kerberos ticket lasting 25h, meaning you will be able to connect passwordlessly to lxplus as long as the kerberos ticket is valid.
::: To check if this worked, do a '''klist -f'''. Check the ''Flags'', there should be an '''F''', meaning you will have access to your /afs space.
:* Connect to lxplus using the following ssh options (replace ''MYLOGIN'' with the CERN login):
ssh -2 -oGSSAPIAuthentication=yes -oGSSAPIDelegateCredentials=yes -oGSSAPITrustDNS=yes -l MYLOGIN lxplus.cern.ch

 
* From the UIs, you should simply do:
<pre>
kinit
ssh -2 -oGSSAPIAuthentication=yes -oGSSAPIDelegateCredentials=yes -oGSSAPITrustDNS=yes -oStrictHostKeyChecking=no -l $USER $(nslookup lxplus.cern.ch|grep Address|sed -n 2,2p|awk -F':' '{print $2}')
</pre>

=== How to copy data from disk (eg. beo5) to you space at lxplus? ===
Use secure copy: scp
<pre>
scp 'uname'@lxpub3.iihe.ac.be:/beo5/somefile 'uname'@lxplus.cern.ch:/afs/cern.ch/user/m/me
</pre>

=== How to print from lxplus when you're in building 40 ===
Command to print
<pre>
lpr -P 40-4B-COR ....
</pre>
*[https://winservices.web.cern.ch/winservices/Services/Printers/PrinterList.aspx List] of available printers at CERN.
*[http://linux.web.cern.ch/linux/documentation/printing.shtml More] information.
*[http://cerncourier.com/cws/article/cnl/26307 Article] on printing

Cluster Overview

2016-10-26T14:08:46Z

Romain Rougny: /* How to Connect */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : mshort [ m0 , m1 , m2 , m3 ] , mlong [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@mshort.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>mshort.iihe.ac.be : m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be </pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>mlong.iihe.ac.be : m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 

== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | highbw
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Subgroup of localgrid, subgroup of highmem, with WNs having 10Gb/s bandwidth access to storage. 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3512 
| nowrap="nowrap" align="center" | 1984 
| nowrap="nowrap" align="center" | 448 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="3" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="3"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="3"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

=== Job Deletion ===
Use the following command:
qdel <JOBID>
To delete all your jobs, be patient while using the following line:
qstat localgrid@cream02 |grep $USER|awk '{print "qdel "$1}'|xargs -0 bash -c

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-10-26T14:05:15Z

Romain Rougny: /* Overview */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : mshort [ m0 , m1 , m2 , m3 ] , mlong [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | highbw
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Subgroup of localgrid, subgroup of highmem, with WNs having 10Gb/s bandwidth access to storage. 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3512 
| nowrap="nowrap" align="center" | 1984 
| nowrap="nowrap" align="center" | 448 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="3" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="3"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="3"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

=== Job Deletion ===
Use the following command:
qdel <JOBID>
To delete all your jobs, be patient while using the following line:
qstat localgrid@cream02 |grep $USER|awk '{print "qdel "$1}'|xargs -0 bash -c

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-09-20T14:46:01Z

Romain Rougny: /* Queues */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | highbw
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Subgroup of localgrid, subgroup of highmem, with WNs having 10Gb/s bandwidth access to storage. 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3512 
| nowrap="nowrap" align="center" | 1984 
| nowrap="nowrap" align="center" | 448 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="3" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="3"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="3"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

=== Job Deletion ===
Use the following command:
qdel <JOBID>
To delete all your jobs, be patient while using the following line:
qstat localgrid@cream02 |grep $USER|awk '{print "qdel "$1}'|xargs -0 bash -c

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

BigSubmission

2016-09-12T09:25:50Z

Romain Rougny: /* Submitting more than 2500 jobs to the cluster */

== Submitting more than 2500 jobs to the cluster ==

Use the script '''big-submission''' on the UIs. Here's the help:
<pre>
> big-submission -h

This script will execute qsub lines taken from a file, making sure there's
at least 1500 jobs present every 900 seconds for the user on the cluster.

USAGE:
big-submission myqsubcommands.txt

with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh

big-submission -h : will print this help.

DETAILS:

* The script works in the background, so you can deconnect from your session
while it continues sending jobs.
* Commented lines starting with '#' as well as empty lines will be omitted.
* Each time a job is sent, it is deleted from myqsubcommands.txt,
ie the file always contains jobs not sent. This way, if you kill
big-submission, you know what's left to send.
* The lines that failed will be tried once more at the end.
* The script is meant to run once per user. If you want to add
some commands to send, [stop big-submission], add them to
myqsubcommands.txt, [restart big-submission].
Editing the command list live is possible although not recommended,
and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
* To read the logs, one suggests using `less -r logfile`, this way colors will be
printed nicely.

</pre>

Cluster Overview

2016-09-08T15:25:56Z

Romain Rougny: /* Job Statistics */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

=== Job Deletion ===
Use the following command:
qdel <JOBID>
To delete all your jobs, be patient while using the following line:
qstat localgrid@cream02 |grep $USER|awk '{print "qdel "$1}'|xargs -0 bash -c

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

BigSubmission

2016-09-08T08:20:35Z

Romain Rougny: /* Submitting more than 2500 jobs to the cluster */

== Submitting more than 2500 jobs to the cluster ==

Use the script '''big-submission''' on the UIs. Here's the help:
<pre>
> big-submission -h

This script will execute qsub lines taken from a file, making sure there's
at least 1500 jobs present every 900 seconds for the user on the cluster.

USAGE:
big-submission myqsubcommands.txt

with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh

big-submission -h : will print this help.

DETAILS:

* Commented lines starting with '#' as well as empty lines will be omitted.
* Each time a job is sent, it is deleted from myqsubcommands.txt,
ie the file always contains jobs not sent. This way, if you kill
big-submission, you know what's left to send.
* The lines that failed will be tried once more at the end.
* The script is meant to run once per user. If you want to add
some commands to send, [stop big-submission], add them to
myqsubcommands.txt, [restart big-submission].
Editing the command list live is possible although not recommended,
and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
* To read the logs, I suggest using `less -r logfile`, this way colors will be
printed nicely.

</pre>

BigSubmission

2016-08-25T16:02:20Z

Romain Rougny: /* Submitting more than 2500 jobs to the cluster */

== Submitting more than 2500 jobs to the cluster ==

Use the script '''big-submission''' on the UIs. Here's the help:
<pre>
> big-submission -h

This script will execute qsub lines taken from a file, making sure there's
at least 1500 jobs present every 900 seconds for the user on the cluster.

USAGE:
big-submission myqsubcommands.txt

with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh

big-submission -h : will print this help.

DETAILS:

* Commented lines starting with '#' will be omitted.
* Each time a job is sent, it is deleted from myqsubcommands.txt,
ie the file always contains jobs not sent. This way, if you kill
big-submission, you know what's left to send.
* The script is meant to run once per user. If you want to add
some commands to send, [stop big-submission], add them to
myqsubcommands.txt, [restart big-submission].
Editing the command list live is possible although not recommended,
and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`

</pre>

Main Page

2016-08-11T13:08:45Z

Romain Rougny: /* Using the Tier2 computing resources */

== Welcome to the CMS Belgian T2 Wiki ==
<center> [[first_access_to_t2b|=> FIRST ACCESS TO T2B <=]] </center>

=== General information for users ===
*[[Faq_t2b | FAQ]]
*[[ Cluster_Presentation | Cluster Presentation ]]
==== Information for new users ====
*[[T2bSupport| T2B Support guidelines]]
*[[First_access_to_t2b|Getting access to T2B]]
*[[Getting_a_certificate_for_the_T2|Certificates and VOs]]

==== Using the Tier2 computing resources ====
*[[policies| Policies concerning the usage of computing resources.]]
*[[CurrentStatus| Current status of the Tier2]]
*[[Getting_started_with_the_CMSSW_software| Getting started with the CMSSW software]]
*[[Using_Git| Using Git]]
*[[Getting_started_with_the_MadGraph_software| Getting started with the MadGraph software]]
*Submitting jobs with CRAB
**[[gridSubmission_withCrab| To the worldwide grid]]
*Submitting jobs without CRAB
**[[localSubmission| To the local resources]]
::* [[bigSubmission|Sending unlimited number of Jobs]]
:*[[gridSubmission| To the worldwide grid]]
*[[GridStorageAccess| How to handle data on Grid storage]]
*[[FAQ_CMSSW_on_the_Grid| FAQ CMSSW on the Grid on proxy and more!]]
*[[OtherSoftware| Other software available at the T2]]

==== Back-up procedures ====
*[[BackupDocsLinuxLaptop| Procedure to automate backups of personal documents (Linux laptops only)]]
*[[Backup| Backups of /user , /group , /data , /ice3]]
*[[AccidentalDeleteOfFiles| What to do if I have accidentally deleted some files on my personal computer ?]]

==== Other topics ====
*[[User_Meetings]] list of user meetings with added transparencies
*[[Basic_computing_skills| Basic computing skills]]
*[[CernLxplus| Useful info on use of lxplus.cern.ch]]

----

=== Dedicated twiki pages maintained by several research groups ===
*[[TtBar_Analysis_Framework| TtBar Analysis Framework (old)]]
*[[TopQuarkGroup| Top Quark Group wiki]]
*[[HEEP_Analysis_Framework| HEEP Analysis Framework]]
*[[V0_Analysis_wiki| V0 Analysis wiki]]
*[[Info_exchange| Higgs analysis]]

----

=== Service work for CMS ===
*[[DDT]]
*[[CMSSWDeployment]]
*[[ProdAgentAllUsers| Prodagent for users]]
*[[TestStoreTemp| Writing tests in /store/temp/user on T2 SE's]]
*[[CrabServerInstall| CRAB Server installation]]
*[[CrabValidation| Basic validation of CRAB releases]]

----

=== Obsolete twiki pages ===
*[[DIY-UI]]
*[[CrabIIHETransitionSLC4ToSLC5| Using CRAB at IIHE during the transition from SLC4 to SLC5]]
*[[DataSamplesRequests2007| Samples Requests for 2007]]
*[[DataSamplesRequests2008| Samples Requests for 2008]]
*[[CrabNewIIHE| Crab at IIHE]]
*[[OldMainPage| The old main page is kept here for reference]]

== Admin section ==
*[[AdminPage| Pages for administrators]]

== Getting started with MediaWiki ==
*[//meta.wikimedia.org/wiki/Help:Contents User's Guide]
*[//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]
*[//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]
*[https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

Cluster Overview

2016-08-11T12:33:56Z

Romain Rougny: /* Job submission */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

[[File:Exclamation-mark.jpg|left|25px|line=1|]] If you want to send more than 2500 jobs to the cluster, write all qsub commands in a text file, and use the script '''big-submission''' (more info [[bigSubmission|here]]).
 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

BigSubmission

2016-08-11T12:29:00Z

Romain Rougny: Created page with "== Submitting more than 2500 jobs to the cluster == Use the script '''big-submission''' on the UIs. Here's the help: <pre> > big-submission -h This script will execute qs..."

Cluster Overview

2016-08-11T12:24:50Z

Romain Rougny: /* Queues */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs     (see [[bigSubmission|here]] if you want to send more) 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Backup

2016-07-15T14:16:15Z

Romain Rougny:

== Backups ==

All backups can be found in the '''/backup''' directory on the UIs and WNs.

=== /user & /group ===

Our zfs server allows us to do regular snapshots:
* There is one done every 24 hours at midnight, we keep the last seven of those (so a complete week), eg:
:: <pre>auto:hourly-storage-2016-07-09-00:00</pre>

* We also have one snapshot done every 7 days at 3am, and keep the last 4 of them (so a month), eg:
:: <pre>auto:daily-storage-2016-06-29-03:00</pre>

Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.

Cluster Overview

2016-07-15T13:13:51Z

Romain Rougny: /* Backup */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

== Backup ==
There are several areas that we regularly back up: '''/user''' , '''/group''' , '''/data''' , '''/ice3'''. 
You can find more information on the backup frequency and how to access them [[Backup|here]].

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Main Page

2016-07-15T13:08:48Z

Romain Rougny: /* Back-up procedures */

== Welcome to the CMS Belgian T2 Wiki ==
<center> [[first_access_to_t2b|=> FIRST ACCESS TO T2B <=]] </center>

=== General information for users ===
*[[Faq_t2b | FAQ]]
*[[ Cluster_Presentation | Cluster Presentation ]]
==== Information for new users ====
*[[T2bSupport| T2B Support guidelines]]
*[[First_access_to_t2b|Getting access to T2B]]
*[[Getting_a_certificate_for_the_T2|Certificates and VOs]]

==== Using the Tier2 computing resources ====
*[[policies| Policies concerning the usage of computing resources.]]
*[[CurrentStatus| Current status of the Tier2]]
*[[Getting_started_with_the_CMSSW_software| Getting started with the CMSSW software]]
*[[Using_Git| Using Git]]
*[[Getting_started_with_the_MadGraph_software| Getting started with the MadGraph software]]
*Submitting jobs with CRAB
**[[gridSubmission_withCrab| To the worldwide grid]]
*Submitting jobs without CRAB
**[[localSubmission| To the local resources]]
**[[gridSubmission| To the worldwide grid]]
*[[GridStorageAccess| How to handle data on Grid storage]]
*[[FAQ_CMSSW_on_the_Grid| FAQ CMSSW on the Grid on proxy and more!]]
*[[OtherSoftware| Other software available at the T2]]

==== Back-up procedures ====
*[[BackupDocsLinuxLaptop| Procedure to automate backups of personal documents (Linux laptops only)]]
*[[Backup| Backups of /user , /group , /data , /ice3]]
*[[AccidentalDeleteOfFiles| What to do if I have accidentally deleted some files on my personal computer ?]]

==== Other topics ====
*[[User_Meetings]] list of user meetings with added transparencies
*[[Basic_computing_skills| Basic computing skills]]
*[[CernLxplus| Useful info on use of lxplus.cern.ch]]

----

=== Dedicated twiki pages maintained by several research groups ===
*[[TtBar_Analysis_Framework| TtBar Analysis Framework (old)]]
*[[TopQuarkGroup| Top Quark Group wiki]]
*[[HEEP_Analysis_Framework| HEEP Analysis Framework]]
*[[V0_Analysis_wiki| V0 Analysis wiki]]
*[[Info_exchange| Higgs analysis]]

----

=== Service work for CMS ===
*[[DDT]]
*[[CMSSWDeployment]]
*[[ProdAgentAllUsers| Prodagent for users]]
*[[TestStoreTemp| Writing tests in /store/temp/user on T2 SE's]]
*[[CrabServerInstall| CRAB Server installation]]
*[[CrabValidation| Basic validation of CRAB releases]]

----

=== Obsolete twiki pages ===
*[[DIY-UI]]
*[[CrabIIHETransitionSLC4ToSLC5| Using CRAB at IIHE during the transition from SLC4 to SLC5]]
*[[DataSamplesRequests2007| Samples Requests for 2007]]
*[[DataSamplesRequests2008| Samples Requests for 2008]]
*[[CrabNewIIHE| Crab at IIHE]]
*[[OldMainPage| The old main page is kept here for reference]]

== Admin section ==
*[[AdminPage| Pages for administrators]]

== Getting started with MediaWiki ==
*[//meta.wikimedia.org/wiki/Help:Contents User's Guide]
*[//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]
*[//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]
*[https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

GridStorageAccess

2016-07-13T15:23:56Z

Romain Rougny: /* GFAL */

This page describes how to handle data stored on grid storage.

== Before starting ==
[[File:Exclamation-mark.jpg|left|40x30px|line=1|]] Before being able to run these commands you first need to make a valid proxy [[File:Exclamation-mark.jpg|40x30px|line=1|]] 
::: <pre>voms-proxy-init --voms cms</pre>

 
== GFAL ==
GFAL is a wrapper around the latest grid commands. Learning to use it means that whatever middleware requires to be used in the future, you don't need to learn new commands (like srm, lcg, etc)

 
=== gfal-commands ===
If you want more information on the options that can be used, please use '''man gfal-command''' !

Here are all the commands that can be used:
*''gfal-ls'': get information on a file
*''gfal-mkdir'': remove a directory
*''gfal-rm'': removes a file. To remove an entire directory, use -r
*''gfal-copy'': copy files.

 
=== Usage ===
There are 2 types of file url:
* '''Distant files''': their url is of the type srm://<name_of_server>:<port>/some/path, eg for IIHE:
srm://maite.iihe.ac.be:8443/pnfs/iihe/
* '''Local files''': their url is of the type file://path_of_the_file, eg for IIHE:
file:///user/$USER/MyFile.root

[[File:Exclamation-mark.jpg|left|40x30px|line=1|]] Be careful, the number of '''/''' is very -very- important [[File:Exclamation-mark.jpg|40x30px|line=1|]]

*To get a list of all distant urls for all the Storage Elements, one can do:
::<pre> lcg-infosites --vo cms se </pre>

=== Examples ===
*To list the contents of a directory ''/pnfs/iihe/cms'' :
::<pre> gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms </pre>

* To create a directory:
::<pre> gfal-mkdir srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>

*copy file from local disk to remote server
::<pre> gfal-copy file:///user/$USER/MyFile.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/ </pre>

* To copy a file from remote server to our Storage Element:
::<pre> gfal-copy srm://srm-eoscms.cern.ch:8443/srm/v2/server?SFN=/eos/cms/store/group/comm_trigger/L1TrackTrigger/BE5D_620_SLHC6/singleMu/NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/odevroed/eosTransfer.root </pre>

* To delete a file on remote server
::<pre> gfal-rm srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/MyFile.root </pre>

* To remove a directory and its entire content on remote server ?!? not working for now ?):
::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>

=== Bulk file transfers ===

There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.

Syntax:
<pre> gfal-copy -f --from-file files.txt file://$PWD </pre>
where files.txt is a file where every line is a source in srm url syntax.

Make some tests with one line in datafile and make sure the srm url is OK for both source and destination before running over several files.

=== Copy directories from and to pnfs within the IIHE ===
A script to copy full directories to and from pnfs exists on the slc6 UI's:

<pre>
copyDirectoryPnfs.py
Move all files in a directory to or from pnfs
This script assumes that you copy within the IIHE
The script does not do recursive copying
Make sure you have a valid proxy, made with voms-proxy-init --voms cms:/cms/becms

Mandatory options:
--in= : directory to copy from
--out= : directory to copy to
Both directories need to be complete (i.e. including the /pnfs or /user part

example:
copyDirectoryPnfs.py --out=/user/odevroed/newfile --in=/pnfs/iihe/cms/store/user/odevroed/newdir

Optional:
-h, --help : print this help message
</pre>

== dCache ==
Direct dcache access to files is only possible if the software supports it.
PNFS (the directory structure seen under /pnfs/) is NOT a real filesystem: it is an '''immutable''' filesystem, and is mounted read-only.

*Therefore, here is a list of the commands that work:
'''ls'''
* And replacement commands:
** '''dccp''': to copy files from /pnfs to local disk. Example:
dccp dcap://maite.iihe.ac.be/pnfs/iihe

* To open files using root, use eg
<pre> root dcap://maite.iihe.ac.be/pnfs/iihe/some/file.root </pre>

When reading out the rootfiles is rather slow or it doesn't work at all and nothing is wrong with the root file (e.g. in an interactive analysis on beo or msa) you can increase your dCache readahead buffer. Don't make the buffer larger than 50MB! To enlarge the buffer set this in you environment for csh:
<pre>
setenv DCACHE_RAHEAD 1
setenv DCACHE_RA_BUFFER 50000000
</pre>
For bash:
<pre>
export DCACHE_RAHEAD=true
export DCACHE_RA_BUFFER=50000000
</pre>
See the dChache [http://www.dcache.org/ fanpage] for further reading.

GridStorageAccess

2016-07-13T15:22:10Z

Romain Rougny:

This page describes how to handle data stored on grid storage.

== Before starting ==
[[File:Exclamation-mark.jpg|left|40x30px|line=1|]] Before being able to run these commands you first need to make a valid proxy [[File:Exclamation-mark.jpg|40x30px|line=1|]] 
::: <pre>voms-proxy-init --voms cms</pre>

== GFAL ==
GFAL is a wrapper around the latest grid commands. Learning to use it means that whatever middleware requires to be used in the future, you don't need to learn new commands (like srm, lcg, etc)

 
=== gfal-commands ===
If you want more information on the options that can be used, please use '''man gfal-command''' !

Here are all the commands that can be used:
*''gfal-ls'': get information on a file
*''gfal-mkdir'': remove a directory
*''gfal-rm'': removes a file. To remove an entire directory, use -r
*''gfal-copy'': copy files.

 
=== Usage ===
There are 2 types of file url:
* '''Distant files''': their url is of the type srm://<name_of_server>:<port>/some/path, eg for IIHE:
srm://maite.iihe.ac.be:8443/pnfs/iihe/
* '''Local files''': their url is of the type file://path_of_the_file, eg for IIHE:
file:///user/$USER/MyFile.root

[[File:Exclamation-mark.jpg|left|40x30px|line=1|]] Be careful, the number of '''/''' is very -very- important [[File:Exclamation-mark.jpg|40x30px|line=1|]]

*To get a list of all distant urls for all the Storage Elements, one can do:
::<pre> lcg-infosites --vo cms se </pre>

=== Examples ===
*To list the contents of a directory ''/pnfs/iihe/cms'' :
::<pre> gfal-ls srm://maite.iihe.ac.be:8443/pnfs/iihe/cms </pre>

* To create a directory:
::<pre> gfal-mkdir srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>

*copy file from local disk to remote server
::<pre> gfal-copy file:///user/$USER/MyFile.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/ </pre>

* To copy a file from remote server to our Storage Element:
::<pre> gfal-copy srm://srm-eoscms.cern.ch:8443/srm/v2/server?SFN=/eos/cms/store/group/comm_trigger/L1TrackTrigger/BE5D_620_SLHC6/singleMu/NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/odevroed/eosTransfer.root </pre>

* To delete a file on remote server
::<pre> gfal-rm srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/MyFile.root </pre>

* To remove a directory and its entire content on remote server ?!? not working for now ?):
::<pre> gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir </pre>

=== Bulk file transfers ===

There is an elegant way to run gfal-copy through several files. This is done using the '''--from-file''' option.

Syntax:
<pre> gfal-copy -f --from-file files.txt file://$PWD </pre>
where files.txt is a file where every line is a source in srm url syntax.

Make some tests with one line in datafile and make sure the srm url is OK for both source and destination before running over several files.

=== Copy directories from and to pnfs within the IIHE ===
A script to copy full directories to and from pnfs exists on the slc6 UI's:

<pre>
copyDirectoryPnfs.py
Move all files in a directory to or from pnfs
This script assumes that you copy within the IIHE
The script does not do recursive copying
Make sure you have a valid proxy, made with voms-proxy-init --voms cms:/cms/becms

Mandatory options:
--in= : directory to copy from
--out= : directory to copy to
Both directories need to be complete (i.e. including the /pnfs or /user part

example:
copyDirectoryPnfs.py --out=/user/odevroed/newfile --in=/pnfs/iihe/cms/store/user/odevroed/newdir

Optional:
-h, --help : print this help message
</pre>

== dCache ==
Direct dcache access to files is only possible if the software supports it.
PNFS (the directory structure seen under /pnfs/) is NOT a real filesystem: it is an '''immutable''' filesystem, and is mounted read-only.

*Therefore, here is a list of the commands that work:
'''ls'''
* And replacement commands:
** '''dccp''': to copy files from /pnfs to local disk. Example:
dccp dcap://maite.iihe.ac.be/pnfs/iihe

* To open files using root, use eg
<pre> root dcap://maite.iihe.ac.be/pnfs/iihe/some/file.root </pre>

When reading out the rootfiles is rather slow or it doesn't work at all and nothing is wrong with the root file (e.g. in an interactive analysis on beo or msa) you can increase your dCache readahead buffer. Don't make the buffer larger than 50MB! To enlarge the buffer set this in you environment for csh:
<pre>
setenv DCACHE_RAHEAD 1
setenv DCACHE_RA_BUFFER 50000000
</pre>
For bash:
<pre>
export DCACHE_RAHEAD=true
export DCACHE_RA_BUFFER=50000000
</pre>
See the dChache [http://www.dcache.org/ fanpage] for further reading.

File:Exclamation-mark.jpg

2016-07-13T13:14:00Z

Romain Rougny:

Getting a certificate for the T2

2016-06-22T15:53:07Z

Romain Rougny: /* First time certificate */

If you need grid access on the T2, please follow all the steps below: 

=== First time certificate ===
# [[Obtaining_a_certificate | Get a certificate (new)]]. ''If you already have a CERN grid certificate, you can temporarily use it. But for accounting reasons, we need you to get a Belgian certificate ''
# Extract your certificate from your browser: '''[[Chrome]]''' | '''[[Firefox]]'''
# [[certificate_to_UI | Put your certificate on the UIs]]
# Register to the VO
#:* CMS
#:*# [[Register_to_the_CMS_VO|Register to the CMS VO]]
#:*# [[SiteDB | Check if your certificate is ok on SiteDB]]. Note the DN.
#:*# [[CERN_certificate_management | Check that your certificate is the only one registered on the CERN website.]]
#:* IceCube
#:** [[Register_to_the_IceCube_VO|Register to the IceCube VO]]
# Send a mail to the T2B support (grid_adminATlistserv.vub.ac.be) with your DN in order to have write access on the T2.
# [[Check_Certificate_UIs | Check if everything works fine on the mX machines]]

=== Certificate renewal ===
As we switched the company who provides us with certificates, you need to follow the 'First time certificate' to have a new one, this until December 2016 at least.

Cluster Overview

2016-06-10T08:48:42Z

Romain Rougny: /* Queues */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |30 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Monitoring

2016-05-20T18:11:53Z

Romain Rougny:

{| class="wikitable"
| '''Nagios old style'''
| '''Nagios Check_mk style'''
| '''Nagios History'''
| '''JSON'''
| '''Python'''
|-
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/status.cgi?host=cream02.iihe.ac.be][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=cream02.iihe.ac.be][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/notifications.cgi?host=cream02.iihe.ac.be][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=cream02.iihe.ac.be&output_format=json][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=cream02.iihe.ac.be&output_format=python][cream02]]
|-
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/status.cgi?host=maite.iihe.ac.be][maite]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=maite.iihe.ac.be][maite]]
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/notifications.cgi?host=maite.iihe.ac.be][maite]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=maite.iihe.ac.be&output_format=json][maite]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=maite.iihe.ac.be&output_format=python][maite]]
|}

----
Links:

*Pilot Monitoring (CMS):
** '''Pilot factory''' http://submit-3.t2.ucsd.edu/CSstoragePath/aperez/T2s/multicore_occupancy_t2s_168h.html
** '''Current Pilot info''' http://submit-3.t2.ucsd.edu/CSstoragePath/aperez/HTML/site_status/status_mcore_T2_BE_IIHE.html
** '''Factory+Pressure summary''' http://cms-gwmsmon.cern.ch/totalview/T2_BE_IIHE
** '''Schedds + Pressure + Exclusive Pressure''' http://submit-3.t2.ucsd.edu/CSstoragePath/Monitor/latest-new.txt

*VUB-ULB GRID:
**Ganglia http://mon.iihe.ac.be/ganglia

*BEgrid:
**Ganglia https://ganglia.begrid.be/ganglia/
**GridIce https://gridice.begrid.be/gridice

*LCG
**GocDB/Gstat http://goc.grid.sinica.edu.tw/gstat/
**GridIce http://gridice2.cnaf.infn.it:50080/gridice/site/site.php
**Admin SFTs https://monitoring.egee.man.poznan.pl/admin2
**GGUS Ticket Service https://gus.fzk.de/pages/home.php
**Accounting Info http://www2.egee.cesga.es/gridsite/accounting/CESGA/tree_egee.php?Path=1.7
**Availabilty/Reliabilty http://gridview.cern.ch/GRIDVIEW/same_index.php
**Modded version (only displays UCL and ULB-VUB): http://mon.iihe.ac.be/modded_gridview/same_index.php
**Basic plots
***Select <tt>Tier-2 Site Availability</tt>
***Select both sites
***Select eg daily plots for one month

*Belnet Network Monitor http://monitor.belnet.be/graph/grapherrd.php: (This page only works from whitin BELNET client network.)
**the Inbound/Outbound are from BELNET view (ie Outbound BELNET = Incoming university traffic)
**the "Free traffic" is Research traffic, the red line is a limit for commercial traffic.
**UCL (whole campus) http://monitor.belnet.be/graph/grapherrd.php?page=lln.cfg&target=ucl
**ULB-VUB (whole campus) http://monitor.belnet.be/graph/grapherrd.php?page=brussels.cfg&target=ulb-vub

*CMS
**Dashboard http://arda-dashboard.cern.ch/cms/
**Monalisa http://monalisa.cacr.caltech.edu:9090/
**CRAB http://cmsgridweb.pg.infn.it/crab/crabmon.php
**Phedex Heartbeat http://cmsdoc.cern.ch/cms/aprom/TransferHeartbeat/browser
**Phedex on Dashboard http://pcardabg.cern.ch:8080/dashboard/phedex/
**Installed software https://twiki.cern.ch/twiki/bin/view/CMS/CMSSWInstStatus
**SC4
**SC4 Site Status https://twiki.cern.ch/twiki/bin/view/CMS/SWIntSC4SiteStatus
**CSA06
**Wiki: https://uimon.cern.ch/twiki/bin/view/CMS/CSA06

{{TracNotice|{{PAGENAME}}}}

Register to the IceCube VO

2016-05-20T17:25:20Z

Romain Rougny:

* Register with your certificate here :

https://grid-voms.desy.de:8443/voms/icecube

* Test on m machines :

<pre>
voms-proxy-init --voms icecube
</pre>

Monitoring

2016-05-18T17:24:18Z

Romain Rougny:

{| class="wikitable"
| '''Nagios old style'''
| '''Nagios Check_mk style'''
| '''Nagios History'''
| '''JSON'''
| '''Python'''
|-
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/status.cgi?host=cream02.iihe.ac.be][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=cream02.iihe.ac.be][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/notifications.cgi?host=cream02.iihe.ac.be][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=cream02.iihe.ac.be&output_format=json][cream02]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=cream02.iihe.ac.be&output_format=python][cream02]]
|-
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/status.cgi?host=maite.iihe.ac.be][maite]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=maite.iihe.ac.be][maite]]
| [[https://etf-cms-prod.cern.ch/etf/nagios/cgi-bin/notifications.cgi?host=maite.iihe.ac.be][maite]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=maite.iihe.ac.be&output_format=json][maite]]
| [[https://etf-cms-prod.cern.ch/etf/check_mk/view.py?view_name=host&host=maite.iihe.ac.be&output_format=python][maite]]
|}

----
Links:

*Pilot Monitoring (CMS):
** http://submit-3.t2.ucsd.edu/CSstoragePath/aperez/T2s/multicore_occupancy_t2s_168h.html
** http://submit-3.t2.ucsd.edu/CSstoragePath/aperez/HTML/site_status/status_mcore_T2_BE_IIHE.html
** http://cms-gwmsmon.cern.ch/totalview/T2_BE_IIHE

*VUB-ULB GRID:
**Ganglia http://mon.iihe.ac.be/ganglia

*BEgrid:
**Ganglia https://ganglia.begrid.be/ganglia/
**GridIce https://gridice.begrid.be/gridice

*LCG
**GocDB/Gstat http://goc.grid.sinica.edu.tw/gstat/
**GridIce http://gridice2.cnaf.infn.it:50080/gridice/site/site.php
**Admin SFTs https://monitoring.egee.man.poznan.pl/admin2
**GGUS Ticket Service https://gus.fzk.de/pages/home.php
**Accounting Info http://www2.egee.cesga.es/gridsite/accounting/CESGA/tree_egee.php?Path=1.7
**Availabilty/Reliabilty http://gridview.cern.ch/GRIDVIEW/same_index.php
**Modded version (only displays UCL and ULB-VUB): http://mon.iihe.ac.be/modded_gridview/same_index.php
**Basic plots
***Select <tt>Tier-2 Site Availability</tt>
***Select both sites
***Select eg daily plots for one month

*Belnet Network Monitor http://monitor.belnet.be/graph/grapherrd.php: (This page only works from whitin BELNET client network.)
**the Inbound/Outbound are from BELNET view (ie Outbound BELNET = Incoming university traffic)
**the "Free traffic" is Research traffic, the red line is a limit for commercial traffic.
**UCL (whole campus) http://monitor.belnet.be/graph/grapherrd.php?page=lln.cfg&target=ucl
**ULB-VUB (whole campus) http://monitor.belnet.be/graph/grapherrd.php?page=brussels.cfg&target=ulb-vub

*CMS
**Dashboard http://arda-dashboard.cern.ch/cms/
**Monalisa http://monalisa.cacr.caltech.edu:9090/
**CRAB http://cmsgridweb.pg.infn.it/crab/crabmon.php
**Phedex Heartbeat http://cmsdoc.cern.ch/cms/aprom/TransferHeartbeat/browser
**Phedex on Dashboard http://pcardabg.cern.ch:8080/dashboard/phedex/
**Installed software https://twiki.cern.ch/twiki/bin/view/CMS/CMSSWInstStatus
**SC4
**SC4 Site Status https://twiki.cern.ch/twiki/bin/view/CMS/SWIntSC4SiteStatus
**CSA06
**Wiki: https://uimon.cern.ch/twiki/bin/view/CMS/CSA06

{{TracNotice|{{PAGENAME}}}}

Network bond and tag

2016-04-21T12:38:30Z

Romain Rougny: /* On the switch */

=== On the host ===

In '''/etc/sysconfig/network-scripts''':
:* ifcfg-bond0
<pre>DEVICE="bond0"
BOOTPROTO=none
IPADDR=192.168.10.135
BROADCAST=192.168.255.255
NETMASK=255.255.0.0
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
BONDING_MASTER=yes
TYPE=Bond
BONDING_OPTS="mode=4 miimon=100 lacp_rate=0"
</pre>

To get second bond interface tagged to vlan 2 (PUB)
vconfig add bond0 2

:* ifcfg-bond0.2
<pre>
DEVICE="bond0.2"
BOOTPROTO=none
ONPARENT=yes
IPADDR=193.58.172.92
BROADCAST="193.58.172.127"
NETMASK=255.255.255.128
GATEWAY=193.58.172.2
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
VLAN=yes
</pre>

* Aggregate interfaces eth0 and eth1 to the bond:
** ifcfg-eth0
<pre>
DEVICE="eth0"
BOOTPROTO=none
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
MASTER=bond0
SLAVE=yes
</pre>

:* ifcfg-eth1
<pre>
DEVICE="eth1"
BOOTPROTO=none
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
MASTER=bond0
SLAVE=yes
</pre>

=== On the switch ===
We are configuring everything in LACP [<=> mode=4 in MODPROBE_OPTS for bond0]
* Adding the PoX interface on the 10G switch
** Using the interface, create the lacp port first in tab '''ETH Mgmt''', click on '''Link Aggregation''', Provide in the box '''LAG Interface Number''' PoX (X a number).
** On tab '''Ports''', click on the port you want to add, select '''LAG''': PoX, '''LAG mode''': active ('''LACP rate''' is auto on slow [<=> lacp_rate=0 in MODPROBE_OPTS for bond0] )
::: ''If is does not work and complains about vlans, follow the section '''Adding a new port to PoX''' below''

* on the switch in CLI:
** the PoX port is going to be in hybrid mode, to have untagged[=access] vlan1 and tagged[=hybrid allowed-vlan] vlan 2 : '''[ vlan1 cannot be tagged on the switch ]'''
<pre>
cd somewhere ?
switchport mode hybrid
switchport access vlan 1
switchport hybrid allowed-vlan 2
</pre>

:* Adding a new port to PoX
<pre>
cd port
switchport mode hybrid
switchport access vlan 1
switchport hybrid allowed-vlan 2
</pre>
then on the interface, add the port to PoX following the previous section.

Network bond and tag

2016-04-21T12:37:24Z

Romain Rougny: /* On the switch */

=== On the host ===

In '''/etc/sysconfig/network-scripts''':
:* ifcfg-bond0
<pre>DEVICE="bond0"
BOOTPROTO=none
IPADDR=192.168.10.135
BROADCAST=192.168.255.255
NETMASK=255.255.0.0
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
BONDING_MASTER=yes
TYPE=Bond
BONDING_OPTS="mode=4 miimon=100 lacp_rate=0"
</pre>

To get second bond interface tagged to vlan 2 (PUB)
vconfig add bond0 2

:* ifcfg-bond0.2
<pre>
DEVICE="bond0.2"
BOOTPROTO=none
ONPARENT=yes
IPADDR=193.58.172.92
BROADCAST="193.58.172.127"
NETMASK=255.255.255.128
GATEWAY=193.58.172.2
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
VLAN=yes
</pre>

* Aggregate interfaces eth0 and eth1 to the bond:
** ifcfg-eth0
<pre>
DEVICE="eth0"
BOOTPROTO=none
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
MASTER=bond0
SLAVE=yes
</pre>

:* ifcfg-eth1
<pre>
DEVICE="eth1"
BOOTPROTO=none
NM_CONTROLLED="no"
ONBOOT="yes"
USERCTL=no
MASTER=bond0
SLAVE=yes
</pre>

=== On the switch ===
We are configuring everything in LACP [<=> mode=4 in MODPROBE_OPTS for bond0]
* Adding the PoX interface on the 10G switch
** Using the interface, create the lacp port first in tab '''ETH Mgmt''', click on '''Link Aggregation''', Provide in the box '''LAG Interface Number''' PoX (X a number).
** On tab '''Ports''', click on the port you want to add, select '''LAG''': PoX, '''LAG mode''': active ('''LACP rate''' is auto on slow [<=> lacp_rate=0 in MODPROBE_OPTS for bond0] )
::: ''If is does not work and complains about vlans, follow the section '''Adding a new port to PoX''' below''

* on the switch in CLI:
** the PoX port is going to be in hybrid mode, to have untagged[=access] vlan1 and tagged[=hybrid allowed-vlan] vlan 2 : '''[ vlan1 cannot be tagged on the switch ]'''
<pre>
cd somewhere ?
switchport mode hybrid
switchport access vlan 1
switchport hybrid allowed-vlan 2
</pre>

:* Adding a new port to PoX
cd port
switchport mode hybrid
switchport access vlan 1
switchport hybrid allowed-vlan 2
</pre>
then on the interface, add the port to PoX following the previous section.

Network bond and tag

2016-04-21T12:18:15Z

Romain Rougny: Created page with "=== On the host === In '''/etc/sysconfig/network-scripts''': :* ifcfg-bond0 <pre>DEVICE="bond0" BOOTPROTO=none IPADDR=192.168.10.135 BROADCAST=192.168.255.255 NETMASK=255.255..."

AdminPage

2016-04-21T12:10:21Z

Romain Rougny:

{{DISPLAYTITLE:Page for Administrators}}
==== Management of the whole cluster ====
*[[elog]]
*[[ShutDownCluster| How to properly switch off the cluster]]
*[[PutClusterOn| How to properly put the cluster on]]
==== CMS Services ====
*[[Phedex]]
*[[Heartbeat]]
*[[LoadTest]]
*[[FroNTier]]
*[[ProdAgent]]
*[[GitForSiteConf| instructions to commit siteconf to git]]
==== Grid Configuration Issues ====
*[[UpdateCertificates| Update the certificates of all our machines]]
*[[CreamIssues| Issues with cream and how to solve them]]
*[[PBS_TMPDIR| PBS TMPDIR]]
*[[APEL| <strike>APEL</strike>(OBSOLETE)]]
*[[BDII]]
*[[FTS]]
*[[SL4_x86_64_WNs| <strike>SL4 x86_64 WNs</strike>(OBSOLETE)]]
*[[CE_oveloaded| CE overloaded]]
*[[RB]]
*[[IPMI]]
*[[CA_certificates| <strike>Upgrade CA certificates</strike> (OBSOLETE)]]
*[[Shutdown| Shutting down the cluster]]
*[[Software_Area_Switch| Software Area Switch]]
*[[KernelUpdate| Kernel mandatory updates for critical vulnerabilities]]
*[[Argus| Argus server and glexec on the workernodes]]
*[[ApelGapPublishing| Apel gap publishing]]
*[[UpdateCACertificates| Update IGTF CA certificates]]

==== Files section ====
*[[DCache| dCache]]
**[[DeleteObsoleteFiles| Find obsolete files]]
**[[DCacheAdminMode| dCache admin mode]]
**[[FindSizeOnDcache| find size of a directory on dcache]]
**[[DcachePoolConfig1912| dCache Pool Postinstallation steps]]
**[[DCacheMaxMovers| Adapt max mover]]
**[[pnfsScripts| scripts to see on what pools files in a directory reside and to move them to other pools]]
*[[OlPnfsFiles| Procedure for removal of old user files on pnfs]]
*[[GetLostFiles| Retrieve lost files from datasets]]
*[[StorageConsistency| Storage Consistency]]

==== Status and Monitoring ====
*[[ReservedWNs| List of reserved WNs]]
*[[Todo| Todo-list]]
*[[Monitoring]]
*[[Plans-Schedule| Plans/Schedule]]
*[[Grid_Troubleshooting_link| Grid Troubleshooting link]]
*[[Incident_reports| Incident Reports]]
*[[Dissapeared_software| How to put the software back]]
*[[Bad_WN| What to do when a WN sends a "bad_wn.pl" email to grid_admin ?]]
*[[Nagios_installation| Nagios Installation at IIHE]]
*[[Restart_DCache| How to restart DCache ]]

==== Info ====
*[[General_info| General info]]
*[[Installing_CMSSW| Installing CMSSW]]
*[[Installing_CRAB| Installing CRAB]]
*[[System_Benchmarks| System Benchmarks]]
*[[T2BTrac| T2B Trac config info]]
*[[HardWare| Hardware information]]
*[[NetworkSetup| Network Setup]]
*[[SetupMonitoringControlerSunfireV20z| Setup Monitoring of LSI Disk Controler on Sunfire V20z Server]]
*[[LDAP_UCL_IIHE| <strike>LDAP authentication system for the replication between UCL and IIHE sites</strike> (OBSOLETE)]]
*[[GridAdminSurvivalGuide| IIHE Grid-admin survival guide]]
*[[Solaris| Solaris 10]]
*[[SolarisSSD| Adding an SSD card and configuring RAID, zpools, filesystems and shares on the new Solaris fileserver]]
*[[LinuxAdminTricks| Linux tricks for admins]]
*[[CrabLocalPbsSubmission| <strike>How to implement local PBS submission with CRAB ?</strike>(OBSOLETE)]]
*[[AddNewUserFromUCLToLDAP| <strike>How to create an account for a CMS user from UCL ?</strike>(OBSOLETE)]]
*[[OSErrata| <strike>Deploying OS errata</strike>(OBSOLETE)]]
*[[BenchmarkHEPSPEC06| Howto benchmark a node with HEPSPEC06]]
*[[Installing_dcache_pool| Install a new dCache pool]]
*[[BackupUsersHomeDirs| Backup of the users home dirs on Jefke]]
*[[MonWebServicesMigration| Migration of mon and its Web services]]
*[[HOWTORestartNagiosTest| HOWTO restart a nagios test manually]]
*[[CompileAndInstallRoot| Compile and install ROOT]]
*[[CleanCreamdb| Clean creamdb]]
*Reboot campaign for the workernodes :
**[[KernelUpdate| Reboot after a kernel update]]
**[[UpgradeWNstoSL5.5| Reboot after an OS upgrade]]
*[[ManageAllAdminScriptsWithGit| Central management of all the admin scripts with Git]]
*[[ConfigProxyCvmfs| Configuration of a proxy for CVMFS]]
**[[RecoverCvmfs| How to recover CVMFS]]
*[[TestNFSPerformance| How to test NFS Performance]]
*[[TetexNotAvailableInSL6| Alternatives to Tetex]]
*[[NewMethodUpdateKernelWorkernodes| A new easy method to update kernel on the workernodes]]
*[[AutomaticMailSendingFromCluster| About automatic mail sending from the cluster]]
*[[T2BTracAccess| T2B Trac access configuration]]
*[[WorkingWithRHEL7| Surviving to RHEL7]]
*[[CCMWithKerberos| Experimental : Securing profiles with Kerberos]]
*[[MigrateToMediaWiki| Migration of T2B Wiki from Trac to MediaWiki]]
*[[motd|Message Of The Day (motd)]]

==== Quattor ====
*[http://quattor.begrid.be/trac/centralised-begrid-v5/wiki/BEgridAndQuattor <strike>BEgrid wiki</strike>(OBSOLETE)]
*[[Test_things| <strike>Test things</strike>(OBSOLETE)]]
*[[Lemon_installation| <strike>Lemon installation</strike>(OBSOLETE)]]
*[[QuattorPointers| <strike>Pointers</strike>]]<strike> to more in-depth information on quattor</strike>(OBSOLETE)
*[[AddingMachineToCluster| <strike>Adding</strike>]]<strike> a new machine to the cluster</strike>(OBSOLETE)
*[[AutomaticMachineTemplateGeneration| Automatic generation of hardware and profile templates for new workernodes]]
*[[InstallationBEgridClient0| Installation of a Quattor deployment server release 13.1]]
*[[InstallFilesNewOS| How to add a new OS to the Quattor Repository]]
*[[GenerateRPMFromATagInGithub| How to build an RPM from a tag in Github]]
*[[HowtoMigrateWNToCB9| How to migrate workernodes from CB8 to CB9]]
*[[WorkingInCB9| Working in CB9 (Quattor release >= 14.2)]]
*[[AideMemoire| FAQ - Aide-mémoire - Howtos]]
*[[BuildANewPysvnOnAiiServer| Howto build a new pysvn on a SL63 AII server]]
*[[QuattorFreeIPA| Quattor and FreeIPA]]
*[[NewRuncheck| Rewrite of the runcheck script in Perl]]
*[[HardDisksManagement| Hard disks management]]
*[[Aquilon| Aquilon]]

==== KVM virtualization ====
*[[VirtWithKVM| Virtualization of the new CREAM-CE on dom02 with KVM]]
*[[VirtWithKVM1| Installation of the new virtualization server dom04]]
*[[CreateVM| Easy creation of virtual machines]]
*[[MonitoringvHostswithGanglia| Monitoring the KVM vHosts with Ganglia]]

==== T2B Cloud ====
*[[MigrationToOpenNebula| Transforming the KVM hypervisors farm into an OpenNebula cloud]]
*[[WorkingInT2BCloud| Working in the T2B cloud]]

==== gUSE/WS-PGRADE portal ====
*[[PortalInstall| Portal installation]]
*[[PortalConfig| Portal configuration]]
*[[PortalOperations| Portal operations]]

==== Migration to EMI-3 ====
*[[MigrateBEgridToEMI3_part1| BEgrid facilities - Part 1]]
*[[MigrateBEgridToEMI3_part2| BEgrid facilities - Part 2]]

==== XEN ====
*[[Manage_XEN| Manage XEN]]
*[[XenQuattor| Xen and Quattor]]

==== CEPH ====
*[[UnderstandingCeph| Understanding Ceph]]
*[[InstallCephWithQuattor| Installing Ceph with Quattor]]
*[[ExperimentsWithCeph| Experiments with Ceph]]
*[[CephBasics| Operating a Ceph cluster]]

==== Logstash / Elasticsearch / Kibana (ELK) ====
machine: log10 | [http://log10.iihe.ac.be/index.html interface] | [http://log10.iihe.ac.be/HQ index manager]
* [[log_forwarding_with_quattor|Forwarding a log with rsyslog to logstash using quattor]]
* [[log_parsing_with_logstash|Parsing the logs with logstash]]

==== Network ====
* [[network_bond_and_tag|Bonding of 2 interfaces + tagging of 2 vlans on the bond (PRIV+PUB)|]]

Main Page

2016-04-06T15:19:27Z

Romain Rougny: /* General information for users */

== Welcome to the CMS Belgian T2 Wiki ==
<center> [[first_access_to_t2b|=> FIRST ACCESS TO T2B <=]] </center>

=== General information for users ===
*[[Faq_t2b | FAQ]]
*[[ Cluster_Presentation | Cluster Presentation ]]
==== Information for new users ====
*[[T2bSupport| T2B Support guidelines]]
*[[First_access_to_t2b|Getting access to T2B]]
*[[Getting_a_certificate_for_the_T2|Certificates and VOs]]

==== Using the Tier2 computing resources ====
*[[policies| Policies concerning the usage of computing resources.]]
*[[CurrentStatus| Current status of the Tier2]]
*[[Getting_started_with_the_CMSSW_software| Getting started with the CMSSW software]]
*[[Using_Git| Using Git]]
*[[Getting_started_with_the_MadGraph_software| Getting started with the MadGraph software]]
*Submitting jobs with CRAB
**[[gridSubmission_withCrab| To the worldwide grid]]
*Submitting jobs without CRAB
**[[localSubmission| To the local resources]]
**[[gridSubmission| To the worldwide grid]]
*[[GridStorageAccess| How to handle data on Grid storage]]
*[[FAQ_CMSSW_on_the_Grid| FAQ CMSSW on the Grid on proxy and more!]]
*[[OtherSoftware| Other software available at the T2]]

==== Back-up procedures ====
*[[BackupDocsLinuxLaptop| Procedure to automate backups of personal documents (Linux laptops only)]]
*[[Backup| Backup scheme for the user disks]]
*[[AccidentalDeleteOfFiles| What to do if I have accidentally deleted some files on my personal computer ?]]

==== Other topics ====
*[[User_Meetings]] list of user meetings with added transparencies
*[[Basic_computing_skills| Basic computing skills]]
*[[CernLxplus| Useful info on use of lxplus.cern.ch]]

----

=== Dedicated twiki pages maintained by several research groups ===
*[[TtBar_Analysis_Framework| TtBar Analysis Framework (old)]]
*[[TopQuarkGroup| Top Quark Group wiki]]
*[[HEEP_Analysis_Framework| HEEP Analysis Framework]]
*[[V0_Analysis_wiki| V0 Analysis wiki]]
*[[Info_exchange| Higgs analysis]]

----

=== Service work for CMS ===
*[[DDT]]
*[[CMSSWDeployment]]
*[[ProdAgentAllUsers| Prodagent for users]]
*[[TestStoreTemp| Writing tests in /store/temp/user on T2 SE's]]
*[[CrabServerInstall| CRAB Server installation]]
*[[CrabValidation| Basic validation of CRAB releases]]

----

=== Obsolete twiki pages ===
*[[DIY-UI]]
*[[CrabIIHETransitionSLC4ToSLC5| Using CRAB at IIHE during the transition from SLC4 to SLC5]]
*[[DataSamplesRequests2007| Samples Requests for 2007]]
*[[DataSamplesRequests2008| Samples Requests for 2008]]
*[[CrabNewIIHE| Crab at IIHE]]
*[[OldMainPage| The old main page is kept here for reference]]

== Admin section ==
*[[AdminPage| Pages for administrators]]

== Getting started with MediaWiki ==
*[//meta.wikimedia.org/wiki/Help:Contents User's Guide]
*[//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]
*[//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]
*[https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

LocalSubmission

2016-04-06T12:20:26Z

Romain Rougny: /* Procedure */

== Direct submission to local queue on the T2_BE_IIHE cluster ==

=== Aim ===

*The aim of this page is to provide a brief introduction how to submit to the localqueue.
*The localqueue allows to send executable code to the Tier2 cluster.
*This procedure can be used to run non-CMSSW code that need access to files on the Storage Element (SE) maite.iihe.ac.be.
*It is useful to use this procedure to not overload the User Interfaces (UIs) known as the mX machines.

=== Procedure ===

*Log in to a UI mX.iihe.ac.be; replace X with a number of choice. See [[policies]] about the policies on the UIs.
*Make a directory and prepare an executable.
<pre>
mkdir directsubmissiontest
cd directsubmissiontest/
emacs script.sh&
</pre>
*Paste following code into script.sh. (see below)
*Execute the following command to submit the script to the local queue
<pre>
qsub -q localgrid@cream02 -o script.stdout -e script.stderr script.sh
</pre>
*Follow the progress of your job on the UI
<pre>
qstat -u $USER localgrid@cream02
</pre>

*Your job are finished if you don't see it anymore with qstat. You should now be able to find your output files in the directory you've create:
<pre>
/user/$USER/directsubmissiontest/script.stdout
/user/$USER/directsubmissiontest/script.stderr
/user/$USER/directsubmissiontest/
</pre>

=== Comments and FAQ ===
*In case you would like to access a root file you should copy it to the /scratch space on the workernode.
**/scratch is the native disk of the workernode and is several 100 GBs big.
**Each job is allotted a working directory that is cleaned automatically at the end of the job. This directory is store in the variable $TMPDIR
**Your procedure should look like this:
**copy the necessary root from /localgrid (if you have any) to $TMPDIR
**Make sure the output of the job is also written to $TMPDIR
**Copy your output files back to /localgrid
**Do not read root files from /localgrid. This directory is not physically located on the workernode, it is mounted from the fileserver. Doing this will put a big load on the fileserver potentially causing the UIs to be slow.

'''****** IMPORTANT *******''' 
If you use the local submission, please notice that you potentially can slow down our site. So please, copy all the files you will use during the job to /scratch to avoid this. 
Many thanks, 
The Admin Team

*How to set CMSSW environment in a batch job?

Add the following lines to your script :
<pre>
pwd=$PWD
source $VO_CMS_SW_DIR/cmsset_default.sh # make scram available
cd /localgrid/<USER NAME>/path/to/CMSSW_4_1_4/src/ # your local CMSSW release
eval `scram runtime -sh` # don't use cmsenv, won't work on batch
cd $pwd
</pre>

*How to make your proxy available during batch jobs?

Make sure you have a valid proxy and copy it to some place on /localgrid :
<pre>
cp $X509_USER_PROXY /localgrid/<USER NAME>/some/place
</pre>

Add the following line to your script :
<pre>
export X509_USER_PROXY=/localgrid/<USER NAME>/some/place
</pre>

*How to avoid my short jobs from being blocked in the waiting queue when the site is full ?

If you intend to submit short jobs, then it is wise to specify explicitly to the batch system their estimated maximum walltime. You can do this by adding an option to the qsub command :
<pre>
qsub -q localgrid@cream02 -o script.stdout -e script.stderr -l walltime=<HH:MM:SS> script.sh
</pre>
or by adding the following line at the beginning of your job script :
<pre>
#PBS -l walltime=<HH:MM:SS>
</pre>
Proceeding this way, your jobs priority will grow faster as time goes by, increasing the chances of being executed first. (The shorter they are, the faster their priority will increase over the time.)

But be aware that if your jobs are running longer then the specified maximum walltime, they will be killed by the batch system. So, don't hesitate to overestimate a bit this maximum walltime.

=== Stop your jobs ===

If for some reason, you want to stop your jobs on the server, you can use this procedure:

<pre>
qstat @cream02 | grep <your user name>
</pre>
This will give you a list of jobs running with thier ID's. f.i.

<pre>
394402.cream02 submit.sh odevroed 0 R localgrid
</pre>
Now, use the ID to kill the job with the qdel command:

<pre>
qdel 394402.cream02
</pre>

Your job will now be removed.

=== Attachments ===

*script.sh
<pre>
#!/bin/bash

##Some general shell commands
STR="Hello World!"
echo $STR
echo ">> script.sh is checking where it is"
pwd
echo ">> script.sh is checking how much disk space is still available"
df -h
echo ">> script.sh is listing files and directories in the current location"
ls -l
echo ">> script.sh is listing files and directories in userdir on storage element"
ls -l /pnfs/iihe/cms/store/user/$USER

##When accessing files on the storage element it is important to execute your code on the /scratch partition of the workernode you are running on. Therefore you need to copy your executable which is accessing/writing root files onto the /scratch partition and execute it there. This is illustrated below.

echo ">> go to TMPDIR"
cd $TMPDIR
echo ">> ls of TMPDIR partition"
ls -l

##Create a small root macro

echo "{
//TFile *MyFile = new TFile(\"testfile.root\",\"RECREATE\");
//MyFile->ls();
//MyFile->Close(),
TFile* f=TFile::Open(\"dcap://maite.iihe.ac.be/pnfs/iihe/cms/store/user/$USER/testfile.root\");
f->ls();
f->Close();
}
" > rootScript.C

cat rootScript.C

echo ">> set root"
##Copied a root version from /user/cmssoft into /localgrid
export ROOTSYS=/localgrid/$USER/cmssoft/root_5.26.00e_iihe_default_dcap/root
export PATH=$PATH:$ROOTSYS/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOTSYS/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/lib

echo ">> execute root macro"
root -q -l -b -n rootScript.C

echo ">> ls of TMPDIR"
ls -l

echo "copy the file back to the /localgrid sandbox"
#cp testfile.root /localgrid/jmmaes/directsubmissiontest
</pre>

{{TracNotice|{{PAGENAME}}}}

Cluster Overview

2016-04-06T11:58:24Z

Romain Rougny: /* Batch System */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue you want [mandatory]

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options
:: For instance, if you want to use 2 cores: -lnodes=1:ppn=2

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-04-06T11:52:35Z

Romain Rougny: /* Job management */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

=== Job Statistics ===
All the log files from the batch system are synced every 30 minutes in:
/group/log/torque/

A simple script to analyze the logs and provide some statistics for the user is provided:
/group/log/torque/torque-user-info.py

Just execute it as is. It will print information like the following:
<pre>
ID: 6077555 ExCode: 0 Mem: 0M cpuT: 0s wallT: 3s eff: 0.0% STDIN
ID: 6077602 ExCode: 0 Mem: 50M cpuT: 0s wallT: 2s eff: 0.0% STDIN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
user[G] # Jobs <MEM> +- RMS #HiMem MAX Mem <CPU time> <walltime> <Eff> % WT/WT_TOT # Jobs with Error code (% of user job)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
rougny[l] 12 13 +- 22 MB 0 52 MB | 00:00:00:00 00:00:00:24 ( 0.0%) (-1.00% of tot) | # EC:
</pre>

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-04-06T09:41:40Z

Romain Rougny: /* Queues */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-02-26T13:05:24Z

Romain Rougny: /* Queues */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 01/2016 : Not enabled yet
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 3064 
| nowrap="nowrap" align="center" | 1536 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

First access to t2b

2016-02-03T15:12:03Z

Romain Rougny: /* Next Steps */

There are a few steps to gain access to the T2:

* '''First thing to know:''' our contact mail is '''''grid_admin AT listserv.vub.ac.be''''' . Please always send mails to this list and not to personnal emails, this way everyone here can respond and keep up-to-date with problems.

 
=== Getting access ===
* Now, to get access to our T2, send us a mail with the following :
*# To know you: add your promotor/boss in cc, and present yourself shortly (name, university, Physics group/experiment). Please do so preferrably from your preferred contact email, as it is the one we will keep to contact you.
*# Add to this email the login you want to use. If you have a CERN account, it needs to be the same (even if said account is not enabled anymore).
*# Finally, copy-paste the content of your '''PUBLIC''' ssh key in the mail, in order to have access to our User Interfaces (UIs, or often called the mX machines). To create a pair of ssh keys if you don't have one already, do the following:
<pre>
in a shell type:
> ssh-keygen
and follow the instructions. The defaults suggested are fine. Just choose a password.

this will create 2 files in the following directory: $HOME/.ssh
The files are:
id_rsa
id_rsa.pub

You need to send us the content of id_rsa.pub, which is your public key. The other is your private key and should never be shared.
Note that if you already have one somewhere else, copying both id_rsa[.pub] files on your computer is fine too.
</pre>
:: 4. [OPTIONAL] If you already have a certificate, please add your DN to the mail. The DN is given by the content of "Identity" when issuing the command voms-proxy-info.

*You need to register to the t2b user mailing-list. This is where we make important announcements to our users.
*# Go to the [https://e-groups.cern.ch/e-groups/EgroupsManageOwnerAdmin.do request page]
*# search for the list beginning with '''belgian-t2-users'''
*# then hit the button subscribe
*# If any of this fails, ask someone to contact the mailinglist admins (or ask them to send a mail to this list with your request)

 

=== Next Steps ===
 
* If you need a grid certificate, follow this [[Getting_a_certificate_for_the_T2|twiki page]]
 
* Now that you have access to our T2, read the following:
*#[[Cluster_Presentation| Presentation of our cluster]] : please have a careful look at this page !
*# [[LocalSubmission|Local job submission]]
*: ... everything else you might need on the [[Main_Page|wiki front page]]

First access to t2b

2016-02-03T15:11:54Z

Romain Rougny: /* Next Steps */

ManageAllAdminScriptsWithGit

2016-02-03T12:50:37Z

Romain Rougny: /* Summary Of Git Commands */

__TOC__

* The admin scripts are managed by [http://git.iihe.ac.be Git]
* They should be mounted on all machines managed by quattor in '''/scripts'''

 
=== Git Workflow ===
==== Adding/Modifying the scripts from your computer ====
# Request an account from git admins
# add your ssh key to your profile
# Do an '''git clone git@git.iihe.ac.be:iihe-scripts.git''' to download the content of iihe-scripts locally If you already have the script directory locally, you can first make sure you are on the master branch with '''git checkout master''' , and then pull the eventual updates with '''git pull''' If you have made some changes to the master branch (wich you shouldn't!!), use '''git stash''' to ''store'' the changes and get a vanilla master, and when on the new branch (next step) use '''git stash apply''' to apply the changes to the current branch.
# Make a new branch from master '''git checkout -b BranchImWorkingOn'''. The branch name should reflect what you plan on doing. Never use directly the ''master'' branch !! You can check this worked with '''git branch -ra''' that lists local & remote branches.
# Make your new scripts or modifications
# If you made new files/directories, prepare them for atomic commit: '''git add -N myfile'''
# Select the chunks of code you want to commit: '''git add -p''' , (explanations of letters [[ManageAllAdminScriptsWithGit#Summary_Of_Git_Commands|here]]). Be as small and precise as possible for each commits.
# If you want to revert the adding chunks to the index you just did : '''git reset -p'''
# Do the commit for the chunks you selected '''git commit -m"short message" [-m"long message"]'''. The long message is optional.
# To list the last commits '''git log -NumberOfCommits'''
# Last, you need to push the local changes to the remote server : '''git push origin BranchImWorkingOn'''
# Make a merge request on the [http://git.iihe.ac.be git site]: [[File: git_merge_request.png|700px|center]]

==== Summary Of Git Commands ====
<pre>
# First
git clone git.blahblah
git checkout master
git pull -p # this updates the local branch taking origin
git checkout -b newbranch
#or git branch -m oldbranch newbranch

# if working on wrong branch
git stash
# then go to new branch
git stash apply

git branch -ra ==> lists repo & local branches

# Atomic commit (by hunks):
git add -N myfile #first prepare file before patchadding
git add -p
y - indexer cette partie
n - ne pas indexer cette partie
a - indexer cette partie et toutes celles restantes dans ce fichier
d - ne pas indexer cette partie ni aucune de celles restantes dans ce fichier
g - sélectionner une partie à voir
/ - chercher une partie correspondant à la regexp donnée
j - laisser cette partie non décidée, voir la prochaine partie non encore décidée
J - laisser cette partie non décidée, voir la prochaine partie
k - laisser cette partie non décidée, voir la partie non encore décidée précédente
K - laisser cette partie non décidée, voir la partie précédente
s - couper la partie courante en parties plus petites
e - modifier manuellement la partie courante
? - afficher l'aide

==> this add hunks to index, needs git commit -m""
git reset -p ==> revert the adding to index

git commit -m"" [-m"long message"]

git diff ==> local / index
git diff HEAD ==> local / repo
git diff --cached ==> index / repo

git log -3 ==> lists commit

# Pushing
git push origin newbranch ==> pushes local branch to repo & creates new repo branch

</pre>

 

=== Adding /scripts to a machine and getting access to it ===
* You need to mount with nfs (or autofs) the /storage of tesla
* Then create a symlink from '''/storage_mnt/storage/group/admin/iihe-scripts''' to '''/scripts'''
* Or you can just include the template in '''config/nfs/storage''' in your machine, which does all this.
* It is in read-only for non-root users of the '''admins''' group, so you cannot make any modifications there.
* If you want access as non-root user, you need to add yourself in [http://freeipa.wn.iihe.ac.be freeipa] [accessible through tunnel only] to the '''admins''' group.

 
=== Old Wiki page for using SVN ===
*[[ManageAllAdminScriptsWithSVN| here]]

ManageAllAdminScriptsWithGit

2016-02-03T12:48:09Z

Romain Rougny: /* Adding/Modifying the scripts from your computer */

__TOC__

* The admin scripts are managed by [http://git.iihe.ac.be Git]
* They should be mounted on all machines managed by quattor in '''/scripts'''

 
=== Git Workflow ===
==== Adding/Modifying the scripts from your computer ====
# Request an account from git admins
# add your ssh key to your profile
# Do an '''git clone git@git.iihe.ac.be:iihe-scripts.git''' to download the content of iihe-scripts locally If you already have the script directory locally, you can first make sure you are on the master branch with '''git checkout master''' , and then pull the eventual updates with '''git pull''' If you have made some changes to the master branch (wich you shouldn't!!), use '''git stash''' to ''store'' the changes and get a vanilla master, and when on the new branch (next step) use '''git stash apply''' to apply the changes to the current branch.
# Make a new branch from master '''git checkout -b BranchImWorkingOn'''. The branch name should reflect what you plan on doing. Never use directly the ''master'' branch !! You can check this worked with '''git branch -ra''' that lists local & remote branches.
# Make your new scripts or modifications
# If you made new files/directories, prepare them for atomic commit: '''git add -N myfile'''
# Select the chunks of code you want to commit: '''git add -p''' , (explanations of letters [[ManageAllAdminScriptsWithGit#Summary_Of_Git_Commands|here]]). Be as small and precise as possible for each commits.
# If you want to revert the adding chunks to the index you just did : '''git reset -p'''
# Do the commit for the chunks you selected '''git commit -m"short message" [-m"long message"]'''. The long message is optional.
# To list the last commits '''git log -NumberOfCommits'''
# Last, you need to push the local changes to the remote server : '''git push origin BranchImWorkingOn'''
# Make a merge request on the [http://git.iihe.ac.be git site]: [[File: git_merge_request.png|700px|center]]

==== Summary Of Git Commands ====
<pre>

# First
git clone git.blahblah
git checkout -b newbranch ==> making a new branch from master. Safe, will not delete files.

git branch -ra ==> lists repo & local branches
git push origin --delete mybranch ==> will delete remote branch

# Atomic commit (by hunks):
git add -N myfile #first prepare file before patchadding
git add -p
y - indexer cette partie
n - ne pas indexer cette partie
a - indexer cette partie et toutes celles restantes dans ce fichier
d - ne pas indexer cette partie ni aucune de celles restantes dans ce fichier
g - sélectionner une partie à voir
/ - chercher une partie correspondant à la regexp donnée
j - laisser cette partie non décidée, voir la prochaine partie non encore décidée
J - laisser cette partie non décidée, voir la prochaine partie
k - laisser cette partie non décidée, voir la partie non encore décidée précédente
K - laisser cette partie non décidée, voir la partie précédente
s - couper la partie courante en parties plus petites
e - modifier manuellement la partie courante
? - afficher l'aide

==> this add hunks to index, needs git commit -m""
git reset -p ==> revert the adding to index

git commit -m"" [-m"long message"]

git diff ==> local / index
git diff HEAD ==> local / repo
git diff --cached ==> index / repo

git log -3 ==> lists commit

# Pushing
git push origin newbranch ==> pushes local branch to repo & creates new repo branch
</pre>

 

=== Adding /scripts to a machine and getting access to it ===
* You need to mount with nfs (or autofs) the /storage of tesla
* Then create a symlink from '''/storage_mnt/storage/group/admin/iihe-scripts''' to '''/scripts'''
* Or you can just include the template in '''config/nfs/storage''' in your machine, which does all this.
* It is in read-only for non-root users of the '''admins''' group, so you cannot make any modifications there.
* If you want access as non-root user, you need to add yourself in [http://freeipa.wn.iihe.ac.be freeipa] [accessible through tunnel only] to the '''admins''' group.

 
=== Old Wiki page for using SVN ===
*[[ManageAllAdminScriptsWithSVN| here]]

Getting a certificate for the T2

2016-01-22T16:24:14Z

Romain Rougny:

If you need grid access on the T2, please follow all the steps below: 

=== First time certificate ===
# [[Obtaining_a_certificate | Get a certificate (new)]]. ''If you already have a CERN grid certificate, you can temporarily use it. But for accounting reasons, we need you to get a Belgian certificate ''
# Extract your certificate from your browser: '''[[Chrome]]''' | '''[[Firefox]]'''
# [[certificate_to_UI | Put your certificate on the UIs]]
# [[Register_to_the_CMS_VO|Register to the CMS VO]]
# [[SiteDB | Check if your certificate is ok on SiteDB]]. Note the DN.
# Send a mail to the T2B support (grid_adminATlistserv.vub.ac.be) with your DN in order to have write access on the T2.
# [[CERN_certificate_management | Check that your certificate is the only one registered on the CERN website.]]
# [[Check_Certificate_UIs | Check if everything works fine on the mX machines]]

=== Certificate renewal ===
As we switched the company who provides us with certificates, you need to follow the 'First time certificate' to have a new one, this until December 2016 at least.

Getting a certificate for the T2

2016-01-22T15:05:17Z

Romain Rougny:

If you need grid access on the T2, please follow all the steps below: 

=== First time certificate ===
# [[Obtaining_a_certificate | Get a certificate (new)]]. ''If you already have a CERN grid certificate, you can temporarily use it. But for accounting reasons, we need you to get a Belgian certificate ''
# Extract your certificate from your browser: '''[[Chrome]]''' | '''[[Firefox]]'''
# [[certificate_to_UI | Put your certificate on the UIs]]
# [[Register_to_the_CMS_VO|Register to the CMS VO]]
# [[SiteDB | Check if your certificate is ok on SiteDB]]. Note the DN.
# Send a mail to the T2B support (grid_adminATlistserv.vub.iihe.ac.be) with your DN in order to have write access on the T2.
# [[CERN_certificate_management | Check that your certificate is the only one registered on the CERN website.]]
# [[Check_Certificate_UIs | Check if everything works fine on the mX machines]]

=== Certificate renewal ===
As we switched the company who provides us with certificates, you need to follow the 'First time certificate' to have a new one, this until December 2016 at least.

Getting a certificate for the T2

2016-01-22T15:04:38Z

Romain Rougny:

If you need grid access on the T2, please follow all the steps below: 

=== First time certificate ===
# [[Obtaining_a_certificate | Get a certificate (new)]]. ''If you already have a CERN grid certificate, you can temporarily use it. But for accounting reasons, we need you to get a Belgian certificate ''
# Extract your certificate from your browser: '''[[Chrome]]''' | '''[[Firefox]]'''
# [[certificate_to_UI | Put your certificate on the UIs]]
# [[Register_to_the_CMS_VO|Register to the CMS VO]]
# [[SiteDB | Check if your certificate is ok on SiteDB]]. Note the DN.
# Send a mail to grid_adminATlistserv.vub.iihe.ac.be with your DN in order to have write access on the T2.
# [[CERN_certificate_management | Check that your certificate is the only one registered on the CERN website.]]
# [[Check_Certificate_UIs | Check if everything works fine on the mX machines]]

=== Certificate renewal ===
As we switched the company who provides us with certificates, you need to follow the 'First time certificate' to have a new one, this until December 2016 at least.

Cluster Overview

2016-01-21T15:46:00Z

Romain Rougny: /* Overview */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
:: The '''File Server''' provides the user home on the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity with regular backups.

 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Mass Storage'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX

 

== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 01/2016 : Not enabled yet
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 2344 
| nowrap="nowrap" align="center" | 768 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-01-21T15:07:26Z

Romain Rougny: /* Job management */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Storage Machines'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX
:* The '''User Storage''': it provides the home of the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity.

 
== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 01/2016 : Not enabled yet
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 2344 
| nowrap="nowrap" align="center" | 768 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-01-21T15:06:02Z

Romain Rougny: /* Usefull links */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Storage Machines'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX
:* The '''User Storage''': it provides the home of the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity.

 
== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 01/2016 : Not enabled yet
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 2344 
| nowrap="nowrap" align="center" | 768 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[http://gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-01-21T15:05:47Z

Romain Rougny: /* Usefull links */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Storage Machines'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX
:* The '''User Storage''': it provides the home of the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity.

 
== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 01/2016 : Not enabled yet
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 2344 
| nowrap="nowrap" align="center" | 768 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==
[gangliat2b.iihe.ac.be/ganglia/ Ganglia Monitoring] : status of all our servers. 
[http://mon.iihe.ac.be/jobview/overview.html JobView Monitoring ] : summary of the cluster usage.

Cluster Overview

2016-01-21T14:54:33Z

Romain Rougny: /* Backup */

== Overview ==

The cluster is composed 3 groups of machines :
 
* The '''User Interfaces (UI)'''
::This is the cluster front-end, to use the cluster, you need to log into those machines
::::Servers : [ m0 , m1 , m2 , m3 ] , [ m5 , m6 , m7 , m8 , m9 ]
 
* The '''Computing Machines :'''
** The '''Computing Element (CE):''' This server is the brain of the batch system : it manages all the submitted jobs, and send them to the worker nodes.
::::Servers : cream02

:* The '''Worker Nodes (WN): ''' This is the power of the cluster : they run multiple jobs in parallel and send the results & status back to the CE.
::::Servers : nodeXX-YY
 
* The '''Storage Machines'''
** The '''Storage Element''': it is the brain of the cluster storage. Grid accessible, it knows where all the files are, and manages all the storage nodes.
::::Server : maite
:* The '''Storage Nodes''': This is the memory of the cluster : they contain big data files. In total, they provide ~2300 TB of grid-accessible storage.
::::Servers : beharXXX
:* The '''User Storage''': it provides the home of the UIs. It is a highly efficient & redundant storage node of ~70 TB capacity.

 
== How to Connect ==

To connect to the cluster, you need to have sent us your public ssh key.
In a terminal, type the following:
ssh -X -o ServerAliveInterval=100 username@m0.iihe.ac.be
:''Tip: the ''-o ServerAliveInterval=100'' option is used to keep your session alive for a long period of time ! You should not be disconnected during a whole day of work.''

After a successful login, you'll see this message :

@@@@@@@@ @@@@ @@@@@ @@@@@@@
@@ @ @ @@ @ @@
@@ @ @@@@ @@@@@ @@@@
@@ @@ @@ @ @@@@
@@ @ @@ @ @@
@@ @@@@@@ @@@@@@ @@@@@@@
@ IIHE

Welcome to the t2b cluster ! You are on the following UI: m2 
You can find more info on our wiki page: http://t2bwiki.iihe.ac.be
To contact us: grid_admin@listserv.vub.ac.be 
Please remember this machine will allow you only 600s (10 minutes)
of cpu time per processes.
________________________________________________________________________
Your Quota on /user: 43% used (282G left)
There are 2 users here | Load: 7.51 /8 CPUs (2%) | Mem: 80% used

Please observe all the information in this message:
* The wiki link, where you should go first to find the information
* The email used for the cluster support (please use this one rather than personal mail, this way everyone on the support team can answer and track the progress.)
* The cpu time limit imposed per process, as we divided our UIs into 2 groups.
:: '''The light task''' UIs (max '''CPU''' time = 10 minutes) : they are used for crab/local job submission, writing code, building debugging ...
::<pre>m0.iihe.ac.be, m1.iihe.ac.be, m2.iihe.ac.be, m3.iihe.ac.be</pre>
:: '''The CPU-intensive''' UIs (max '''CPU''' time = 5 hours) : they are available for CPU-intensive and long tasks, although you should prefer using local job submission ...
::<pre>m5.iihe.ac.be, m6.iihe.ac.be, m7.iihe.ac.be, m8.iihe.ac.be and m9.iihe.ac.be</pre>
* The quota you have left on /user
* Information about how heavily this UI is used. If any of them is red (ie above optimal usage), please consider using another UI. Please be mindful of other users and don't start too many processes, epsecially if the UI is already under charge.

 
== Data Storage & Directory Structure ==

There are 2 main directories to store your work and data:
* '''/user [/$USER]''' : this is your home directory. You have an enforced quota there, as it is an expensive storage with redundancy and daily backups.
* '''/pnfs [/iihe/cms/store/user/$USER]''' : this is where you can store a large amount of data, and is also [[GridStorageAccess|grid-accessible]]. If you need more than 2 TB, please contact us. THere is no backups there, so be careful of what you do !
 
There are other directories than you might want to take notice of:
* '''/group''' : same as /user , but if you need to share/produce in a group.
* '''/scratch''' : a temporary scratch space for your job. Use $TMPDIR on the WNs, it is cleanned after each job :)
* '''/cvmfs''' : Centralised CVMFS software repository from CERN. It should contain most of the software you will need.
* '''/swmgrs''' : local area for shared software not in /cvmfs . You can use a [[OtherSoftware|nice tool]] to find the software and versions available.

 
== Batch System ==

=== Queues ===

The cluster is decomposed in queues

{| width="1064" cellspacing="1" cellpadding="5" border="1" align="center"
|-
! scope="col" | 
! scope="col" | localgrid
! scope="col" | highmem
! scope="col" | express
|-
! scope="row" | Description
| nowrap="nowrap" align="center" | default queue, all available nodes except express 
| nowrap="nowrap" align="center" | Subgroup of localgrid with WNs having 4GB Mem / Slot 01/2016 : Not enabled yet
| nowrap="nowrap" align="center" | Limited walltime 
|-
! scope="row" | # CPU's (Jobs)
| nowrap="nowrap" align="center" | 2344 
| nowrap="nowrap" align="center" | 768 
| nowrap="nowrap" align="center" | 16 
|-
! scope="row" | Walltime limit
| nowrap="nowrap" align="center" colspan="2" | 95 hours
| nowrap="nowrap" align="center" |20 minutes 
|-
! scope="row" | Memory limit
| nowrap="nowrap" align="center" | 2 Gb 
| nowrap="nowrap" align="center" | 4 Gb 
| nowrap="nowrap" align="center" | 2 Gb 
|-
! scope="row" | Max # jobs running / User
| nowrap="nowrap" align="center" colspan="2"| 400 jobs 
| nowrap="nowrap" align="center" | 3 jobs 
|-
! scope="row" | Max # jobs sent to the batch system / User
| nowrap="nowrap" align="center" colspan="2"| 2500 jobs 
| nowrap="nowrap" align="center" | 100 jobs 
|}

 

=== Job submission ===

To submit a job, you just have to use the '''qsub''' command :
<pre>qsub myjob.sh
</pre>
''OPTIONS''

*-q queueName : choose the queue (default: any)

*-N jobName : name of the job

*-I : (capital i) pass in interactive mode

*-m mailaddress : set mail address (use in conjonction with -m) : MUST be @ulb.ac.be or @vub.ac.be

*-m [a|b|e] : send mail on job status change (a = aborted , b = begin, e = end)

*-l : resources options

 

=== Job management ===

To see all jobs (running / queued), you can use the '''qstat''' command, or go to the [http://mon.iihe.ac.be/jobview/overview.html JobView] page to have a summary of what's running.
<pre>qstat
</pre>
''OPTIONS''

* -u username : list only jobs submitted by username

* -n : show nodes where jobs are running

* -q : show the job repartition on queues

 

'''If you want to test the batch system, you can follow the workbook [[LocalSubmission|here]]

 

== Backup ==
'''Backup of /user & /group''' 
There is no way for users to access the backups. Just send a mail to grid_admin@listserv.vub.iihe.ac.be specifying the file(s)/folder(s) you want, and the date you want it from.

Our zfs server allows us to do regular snapshots:
* There is one done everyday at midnight, we keep the last seven of those (so a complete week).
* We also have one snapshot per week, and keep the last 4 of them (so a month).
* Additionnaly, the /user is backed up every day to a secondary machine, in case our main fileserver goes down.
 

== Usefull links ==