ExplainingApel - Revision history

Admin: /* Problems encountered */

2024-07-22T09:46:40Z

Problems encountered

← Older revision		Revision as of 09:46, 22 July 2024
Line 39:		Line 39:

	= Problems encountered =		= Problems encountered =
	* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files''' and in the table '''ProcessedFiles''', remove the records of the blah files that have their '''Parsed''' field = 0 otherwise you will have doublons (if there is no record for a batch file in the '''ProcessedFiles''' table, the parser will reparse the file).		* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files''' and in the table '''ProcessedFiles''', remove the records of the blah files that have their '''Parsed''' field = 0 otherwise you will have doublons (if there is no record for a batch file in the '''ProcessedFiles''' table, the parser will reparse the file). To avoid the problem with missing '''userFQAN''' field in blah records in the future, the script that generates the blah files was altered in the following way:

	To avoid the problem with missing '''userFQAN''' field in blah records in the future, the script that generates the blah files was altered in the following way:

	<pre>		<pre>
Line 50:		Line 48:
	> '"userFQAN=%s" ', "x509UserProxyFirstFQAN",		> '"userFQAN=%s" ', "x509UserProxyFirstFQAN",
	</pre>		</pre>

			This change will be wiped out by an update of the package htcondor-ce-apel.

	* Some records malformed in batch files (missing fields: 8 instead of 11) -> the following fields are sometimes missing in the history file: RemoteWallClockTime, JobStartDate, ResidentSetSize_RAW		* Some records malformed in batch files (missing fields: 8 instead of 11) -> the following fields are sometimes missing in the history file: RemoteWallClockTime, JobStartDate, ResidentSetSize_RAW

Admin: /* Problems encountered */

2024-07-22T09:43:42Z

Problems encountered

@@ Line 40: / Line 40: @@
 = Problems encountered =
 * Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files''' and in the table '''ProcessedFiles''', remove the records of the blah files that have their '''Parsed''' field = 0 otherwise you will have doublons (if there is no record for a batch file in the '''ProcessedFiles''' table, the parser will reparse the file).
 * Some records malformed in batch files (missing fields: 8 instead of 11) -> the following fields are sometimes missing in the history file: RemoteWallClockTime, JobStartDate, ResidentSetSize_RAW

Admin: /* Problems encountered */

2024-07-17T16:29:50Z

Problems encountered

← Older revision		Revision as of 16:29, 17 July 2024
Line 39:		Line 39:

	= Problems encountered =		= Problems encountered =
	* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files'''		* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files''' and in the table '''ProcessedFiles''', remove the records of the blah files that have their '''Parsed''' field = 0 otherwise you will have doublons (if there is no record for a batch file in the '''ProcessedFiles''' table, the parser will reparse the file).
	* Some records malformed in batch files (missing fields: 8 instead of 11) -> the following fields are sometimes missing in the history file: RemoteWallClockTime, JobStartDate, ResidentSetSize_RAW		* Some records malformed in batch files (missing fields: 8 instead of 11) -> the following fields are sometimes missing in the history file: RemoteWallClockTime, JobStartDate, ResidentSetSize_RAW

Admin: /* Problems encountered */

2024-07-17T16:25:46Z

Problems encountered

← Older revision		Revision as of 16:25, 17 July 2024
Line 40:		Line 40:
	= Problems encountered =		= Problems encountered =
	* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files'''		* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files'''
	* Some records malformed in batch files (missing fields: 8 instead of 11)		* Some records malformed in batch files (missing fields: 8 instead of 11) -> the following fields are sometimes missing in the history file: RemoteWallClockTime, JobStartDate, ResidentSetSize_RAW

Admin: /* Generation of job and summary records */

2024-07-17T11:41:01Z

Generation of job and summary records

← Older revision		Revision as of 11:41, 17 July 2024
Line 22:		Line 22:
	* fetch benchmark information from LDAP database;		* fetch benchmark information from LDAP database;
	* join EventRecords and BlahdRecords into JobRecords;		* join EventRecords and BlahdRecords into JobRecords;
	* summarise jobs		* summarise jobs;
	* unload JobRecords or SummaryRecords into filesystem		* unload JobRecords or SummaryRecords into filesystem.

	= Sending the job and summary records to the remote accounting server =		= Sending the job and summary records to the remote accounting server =

Admin: /* Problems encountered */

2024-07-17T11:40:32Z

Problems encountered

← Older revision		Revision as of 11:40, 17 July 2024
Line 39:		Line 39:

	= Problems encountered =		= Problems encountered =
	* Missing userFQAN field in blah records		* Missing userFQAN field in blah records -> use the script '''fix_blah_file.py''' in '''iihe-scripts/ce/correct_blah_files'''
	* Some records malformed in batch files (missing fields: 8 instead of 11)		* Some records malformed in batch files (missing fields: 8 instead of 11)

Admin: /* The Apel database */

2024-07-17T11:39:02Z

The Apel database

← Older revision		Revision as of 11:39, 17 July 2024
Line 13:		Line 13:
	After the blah and batch files have been generated, the script condor_ce_apel.sh will call '''/usr/bin/apelparser''', a Python script whose role is to update the local Apel Mysql database with the content of the blah and batch files.		After the blah and batch files have been generated, the script condor_ce_apel.sh will call '''/usr/bin/apelparser''', a Python script whose role is to update the local Apel Mysql database with the content of the blah and batch files.

	= The ~~Apel~~ database =		= The APEL database =
	In our case, it's a MySQL database whose name is '''apelclient'''. It's created and intialized automatically when the CE is deployed thanks to Puppet. It contains table like '''BlahdRecords''' to store the records from blah files, '''EventRecords''' to store records from batch files, and '''JobRecords''' to store the records that are generated by the join performed by the '''apelclient''' script (see next section). In this database, you'll also find views (their names begins with a 'V') that give a fast access to data. Many operations on these tables are actually performed through MySQL procedures.		In our case, it's a MySQL database whose name is '''apelclient'''. It's created and intialized automatically when the CE is deployed thanks to Puppet. It contains table like '''BlahdRecords''' to store the records from blah files, '''EventRecords''' to store records from batch files, and '''JobRecords''' to store the records that are generated by the join performed by the '''apelclient''' script (see next section). In this database, you'll also find views (their names begins with a 'V') that give a fast access to data. Many operations on these tables are actually performed through MySQL procedures.

Admin: /* Context */

2024-07-17T11:38:08Z

Context

← Older revision		Revision as of 11:38, 17 July 2024
Line 2:		Line 2:
	Our CE is an HTCondor CE, and the underlying batch system is htcondor.		Our CE is an HTCondor CE, and the underlying batch system is htcondor.

	These CEs are equipped with an ~~Apel~~ software stack. The role of this machinery is to extract information about jobs from logfiles, and feed a local database with them. Everyday, the database is read, records are extracted from it and sent to a remote Apel accounting server.		These CEs are equipped with an APEL software stack. The role of this machinery is to extract information about jobs from logfiles, and feed a local database with them. Everyday, the database is read, records are extracted from it and sent to a remote Apel accounting server.

	= From HTCondor job history files to batch and blah files =		= From HTCondor job history files to batch and blah files =

Admin at 15:46, 16 July 2024

2024-07-16T15:46:20Z

← Older revision		Revision as of 15:46, 16 July 2024
Line 37:		Line 37:
	* '''/etc/apel/parser.cfg'''		* '''/etc/apel/parser.cfg'''
	* '''/etc/apel/sender.cfg'''.		* '''/etc/apel/sender.cfg'''.

			= Problems encountered =
			* Missing userFQAN field in blah records
			* Some records malformed in batch files (missing fields: 8 instead of 11)

Admin: /* Sending the job and summary records to the remote accounting server */

2024-07-16T15:36:55Z

Sending the job and summary records to the remote accounting server

← Older revision		Revision as of 15:36, 16 July 2024
Line 29:		Line 29:

	The script that is run to send these records is '''/usr/bin/ssmsend'''. It is configured by file '''/etc/apel/sender.cfg'''. The protocol we use to send the records is called '''AMS''' (instead of '''STOMP''').		The script that is run to send these records is '''/usr/bin/ssmsend'''. It is configured by file '''/etc/apel/sender.cfg'''. The protocol we use to send the records is called '''AMS''' (instead of '''STOMP''').

			Please note that for this step to work, you need to declare the apel service of the CE in '''GOCDB'''.

	= Configuration files =		= Configuration files =