BigSubmission: Difference between revisions

From T2B Wiki
Jump to navigation Jump to search
(Created page with "== Submitting more than 2500 jobs to the cluster == Use the script '''big-submission''' on the UIs. Here's the help: <pre> > big-submission -h This script will execute qs...")
 
No edit summary
 
(3 intermediate revisions by one other user not shown)
Line 9: Line 9:
   at least 1500 jobs present every 900 seconds for the user on the cluster.
   at least 1500 jobs present every 900 seconds for the user on the cluster.


Usage:
USAGE:
   big-submission myqsubcommands.txt
   big-submission myqsubcommands.txt


   with myqsubcommands.txt looking like:
   with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh


   big-submission -h : will print this help.
   big-submission -h : will print this help.
DETAILS:
  * The script works in the background, so you can deconnect from your session
    while it continues sending jobs.
  * Commented lines starting with '#' as well as empty lines will be omitted.
  * Each time a job is sent, it is deleted from myqsubcommands.txt,
    ie the file always contains jobs not sent. This way, if you kill
    big-submission, you know what's left to send.
  * The lines that failed will be tried once more at the end.
  * The script is meant to run once per user. If you want to add
    some commands to send, [stop big-submission], add them to
    myqsubcommands.txt, [restart big-submission].
    Editing the command list live is possible although not recommended,
    and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
  * 3 files are always created for each execution of the script:
    - myqsubcommands.txt.DATE : a backup containing all qsub lines to submit.
    - big-submission.DATE.err : a list of all qsub lines that failed.
    - big-submission.DATE.log : a log file of all actions executed.
  * To read the logs, one suggests using `less -r logfile`, this way colors will be
    printed nicely.


</pre>
</pre>

Latest revision as of 17:21, 17 February 2017

Submitting more than 2500 jobs to the cluster

Use the script big-submission on the UIs. Here's the help:

> big-submission -h


  This script will execute qsub lines taken from a file, making sure there's
  at least 1500 jobs present every 900 seconds for the user on the cluster.

USAGE:
  big-submission myqsubcommands.txt

  with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
 # qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh

  big-submission -h : will print this help.


DETAILS:

  * The script works in the background, so you can deconnect from your session
    while it continues sending jobs.
  * Commented lines starting with '#' as well as empty lines will be omitted.
  * Each time a job is sent, it is deleted from myqsubcommands.txt,
    ie the file always contains jobs not sent. This way, if you kill
    big-submission, you know what's left to send.
  * The lines that failed will be tried once more at the end.
  * The script is meant to run once per user. If you want to add
    some commands to send, [stop big-submission], add them to
    myqsubcommands.txt, [restart big-submission].
    Editing the command list live is possible although not recommended,
    and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
  * 3 files are always created for each execution of the script:
     - myqsubcommands.txt.DATE : a backup containing all qsub lines to submit.
     - big-submission.DATE.err : a list of all qsub lines that failed.
     - big-submission.DATE.log : a log file of all actions executed.
  * To read the logs, one suggests using `less -r logfile`, this way colors will be
    printed nicely.