BigSubmission - Revision history

Rougny Romain rrougny@ua.ac.be at 17:21, 17 February 2017

2017-02-17T17:21:26Z

← Older revision		Revision as of 17:21, 17 February 2017
Line 31:		Line 31:
	* The lines that failed will be tried once more at the end.		* The lines that failed will be tried once more at the end.
	* The script is meant to run once per user. If you want to add		* The script is meant to run once per user. If you want to add
	some commands to send, [stop big-submission], add them to		some commands to send, [stop big-submission], add them to
	myqsubcommands.txt, [restart big-submission].		myqsubcommands.txt, [restart big-submission].
	Editing the command list live is possible although not recommended,		Editing the command list live is possible although not recommended,
	and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`		and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
			* 3 files are always created for each execution of the script:
			- myqsubcommands.txt.DATE : a backup containing all qsub lines to submit.
			- big-submission.DATE.err : a list of all qsub lines that failed.
			- big-submission.DATE.log : a log file of all actions executed.
	* To read the logs, one suggests using `less -r logfile`, this way colors will be		* To read the logs, one suggests using `less -r logfile`, this way colors will be
	printed nicely.		printed nicely.



	</pre>		</pre>

Romain Rougny: /* Submitting more than 2500 jobs to the cluster */

2016-09-12T09:25:50Z

Submitting more than 2500 jobs to the cluster

← Older revision		Revision as of 09:25, 12 September 2016
Line 23:		Line 23:
	DETAILS:		DETAILS:

			* The script works in the background, so you can deconnect from your session
			while it continues sending jobs.
	* Commented lines starting with '#' as well as empty lines will be omitted.		* Commented lines starting with '#' as well as empty lines will be omitted.
	* Each time a job is sent, it is deleted from myqsubcommands.txt,		* Each time a job is sent, it is deleted from myqsubcommands.txt,
Line 33:		Line 35:
	Editing the command list live is possible although not recommended,		Editing the command list live is possible although not recommended,
	and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`		and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
	* To read the logs, ~~I suggest~~ using `less -r logfile`, this way colors will be		* To read the logs, one suggests using `less -r logfile`, this way colors will be
	printed nicely.		printed nicely.




	</pre>		</pre>

Romain Rougny: /* Submitting more than 2500 jobs to the cluster */

2016-09-08T08:20:35Z

Submitting more than 2500 jobs to the cluster

← Older revision		Revision as of 08:20, 8 September 2016
Line 15:		Line 15:
	qsub -q localgrid /path/to/myscript1.sh		qsub -q localgrid /path/to/myscript1.sh
	qsub -q localgrid /path/to/myscript2.sh		qsub -q localgrid /path/to/myscript2.sh
	# qsub -q localgrid /path/to/myscript2.sh		# qsub -q localgrid /path/to/myscript2.sh
	qsub -q localgrid /path/to/myscript2.sh		qsub -q localgrid /path/to/myscript2.sh

Line 23:		Line 23:
	DETAILS:		DETAILS:

	* Commented lines starting with '#' will be omitted.		* Commented lines starting with '#' as well as empty lines will be omitted.
	* Each time a job is sent, it is deleted from myqsubcommands.txt,		* Each time a job is sent, it is deleted from myqsubcommands.txt,
	ie the file always contains jobs not sent. This way, if you kill		ie the file always contains jobs not sent. This way, if you kill
	big-submission, you know what's left to send.		big-submission, you know what's left to send.
			* The lines that failed will be tried once more at the end.
	* The script is meant to run once per user. If you want to add		* The script is meant to run once per user. If you want to add
	some commands to send, [stop big-submission], add them to		some commands to send, [stop big-submission], add them to
Line 32:		Line 33:
	Editing the command list live is possible although not recommended,		Editing the command list live is possible although not recommended,
	and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`		and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`
			* To read the logs, I suggest using `less -r logfile`, this way colors will be
			printed nicely.




	</pre>		</pre>

Romain Rougny: /* Submitting more than 2500 jobs to the cluster */

2016-08-25T16:02:20Z

Submitting more than 2500 jobs to the cluster

← Older revision		Revision as of 16:02, 25 August 2016
Line 9:		Line 9:
	at least 1500 jobs present every 900 seconds for the user on the cluster.		at least 1500 jobs present every 900 seconds for the user on the cluster.

	~~Usage~~:		USAGE:
	big-submission myqsubcommands.txt		big-submission myqsubcommands.txt

	with myqsubcommands.txt looking like:		with myqsubcommands.txt looking like:
	qsub -q localgrid /path/to/myscript1.sh		qsub -q localgrid /path/to/myscript1.sh
			qsub -q localgrid /path/to/myscript2.sh
			# qsub -q localgrid /path/to/myscript2.sh
	qsub -q localgrid /path/to/myscript2.sh		qsub -q localgrid /path/to/myscript2.sh

	big-submission -h : will print this help.		big-submission -h : will print this help.


			DETAILS:

			* Commented lines starting with '#' will be omitted.
			* Each time a job is sent, it is deleted from myqsubcommands.txt,
			ie the file always contains jobs not sent. This way, if you kill
			big-submission, you know what's left to send.
			* The script is meant to run once per user. If you want to add
			some commands to send, [stop big-submission], add them to
			myqsubcommands.txt, [restart big-submission].
			Editing the command list live is possible although not recommended,
			and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`


	</pre>		</pre>

Romain Rougny: Created page with "== Submitting more than 2500 jobs to the cluster == Use the script '''big-submission''' on the UIs. Here's the help:

 > big-submission -h     This script will execute qs..."

2016-08-11T12:29:00Z

Created page with "== Submitting more than 2500 jobs to the cluster == Use the script '''big-submission''' on the UIs. Here's the help: <pre> > big-submission -h This script will execute qs..."

New page

== Submitting more than 2500 jobs to the cluster ==

Use the script '''big-submission''' on the UIs. Here's the help:
<pre>
> big-submission -h

This script will execute qsub lines taken from a file, making sure there's
at least 1500 jobs present every 900 seconds for the user on the cluster.

Usage:
big-submission myqsubcommands.txt

with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh

big-submission -h : will print this help.

</pre>