BigSubmission: Difference between revisions

From T2B Wiki
Jump to navigation Jump to search
No edit summary
 
(2 intermediate revisions by one other user not shown)
Line 15: Line 15:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh


Line 23: Line 23:
DETAILS:
DETAILS:


   * Commented lines starting with '#' will be omitted.
  * The script works in the background, so you can deconnect from your session
    while it continues sending jobs.
   * Commented lines starting with '#' as well as empty lines will be omitted.
   * Each time a job is sent, it is deleted from myqsubcommands.txt,
   * Each time a job is sent, it is deleted from myqsubcommands.txt,
     ie the file always contains jobs not sent. This way, if you kill
     ie the file always contains jobs not sent. This way, if you kill
     big-submission, you know what's left to send.
     big-submission, you know what's left to send.
  * The lines that failed will be tried once more at the end.
   * The script is meant to run once per user. If you want to add
   * The script is meant to run once per user. If you want to add
     some commands to send, [stop big-submission], add them to  
     some commands to send, [stop big-submission], add them to
     myqsubcommands.txt, [restart big-submission].
     myqsubcommands.txt, [restart big-submission].
     Editing the command list live is possible although not recommended,
     Editing the command list live is possible although not recommended,
     and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
     and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
  * 3 files are always created for each execution of the script:
    - myqsubcommands.txt.DATE : a backup containing all qsub lines to submit.
    - big-submission.DATE.err : a list of all qsub lines that failed.
    - big-submission.DATE.log : a log file of all actions executed.
  * To read the logs, one suggests using `less -r logfile`, this way colors will be
    printed nicely.




</pre>
</pre>

Latest revision as of 17:21, 17 February 2017

Submitting more than 2500 jobs to the cluster

Use the script big-submission on the UIs. Here's the help:

> big-submission -h


  This script will execute qsub lines taken from a file, making sure there's
  at least 1500 jobs present every 900 seconds for the user on the cluster.

USAGE:
  big-submission myqsubcommands.txt

  with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
 # qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh

  big-submission -h : will print this help.


DETAILS:

  * The script works in the background, so you can deconnect from your session
    while it continues sending jobs.
  * Commented lines starting with '#' as well as empty lines will be omitted.
  * Each time a job is sent, it is deleted from myqsubcommands.txt,
    ie the file always contains jobs not sent. This way, if you kill
    big-submission, you know what's left to send.
  * The lines that failed will be tried once more at the end.
  * The script is meant to run once per user. If you want to add
    some commands to send, [stop big-submission], add them to
    myqsubcommands.txt, [restart big-submission].
    Editing the command list live is possible although not recommended,
    and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
  * 3 files are always created for each execution of the script:
     - myqsubcommands.txt.DATE : a backup containing all qsub lines to submit.
     - big-submission.DATE.err : a list of all qsub lines that failed.
     - big-submission.DATE.log : a log file of all actions executed.
  * To read the logs, one suggests using `less -r logfile`, this way colors will be
    printed nicely.