Maintenance script: Created page with " == Troubleshooting the Mtop machine == Note: these instructions are for root users only! PageOutline === Restarting TopDB Critical services after mTop reboot === ..."

2015-08-26T12:28:49Z

Created page with " == Troubleshooting the Mtop machine == Note: these instructions are for root users only! PageOutline === Restarting TopDB Critical services after mTop reboot === ..."

New page

== Troubleshooting the Mtop machine ==
Note: these instructions are for root users only!

[[PageOutline]]

=== Restarting TopDB Critical services after mTop reboot ===
Note: for all these commands you need to be root-user
<pre>
ssh -X username@mtop.iihe.ac.be
sudo su -
</pre>
The webserver (apache)
- Start
<pre>
/etc/init.d/httpd start
</pre>
- Restart
<pre>
/etc/init.d/httpd restart
</pre>
- Checking if it works
<pre>
ps U apache | grep httpd | wc -l (should be > 1)
</pre>
MySQL DataBase Server
- Start
<pre>
/etc/init.d/mysqld start
</pre>
- Restart
<pre>
test
</pre>
=== Glite Installation ===

==== Outdated Glite Installation ====

Possible error:
<pre>
crab: Checking the status of all jobs: please wait

crab: Status Query failed with message : error executing
GLiteStatusQuery
Problem loading python LB API.
Your default python is 2.4.3 (#1, Nov 10 2010, 16:40:04)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)]
</pre>

Solution:

<pre>

sudo su -

cd /opt/
tar -czvf /root/glite-backup-DD-MM-YYYY.tar.gz glite
rm -rfv glite
(make shure your private ssh key for the M machines is installed under /root/.ssh/id_rsa_USER)
scp -r -i /root/.ssh/id_rsa_USER username@m7:/opt/glite .

</pre>

Check that the version is the same as on m7:

<pre>
Compare glite-wms-job-status --version on mtop and m7
</pre>

==== CRL out of date ====

Possible error: (Also encountered when submitting crab jobs)

<pre>
[dhondt@mtop ~/AutoMaticTopTreeProducer]$ voms-proxy-init --voms cms:/cms/becms --valid 190:00
Cannot find file or dir: /user/dhondt/.glite/vomses
Enter GRID pass phrase:
Your identity: /C=BE/O=BEGRID/OU=ELEM/OU=VUB/CN=Jorgen DHondt
Creating temporary proxy ................................................... Done
Contacting lcg-voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch] "cms" Failed

Error: Could not establish authenticated connection with the server.
GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:
globus_gss_assist: Error during context initialization
globus_gsi_callback_module: Could not verify credential
globus_gsi_callback_module: Could not verify credential
globus_gsi_callback_module: Invalid CRL: The available CRL has expired
</pre>

Solution

<pre>
sudo su -
/opt/external/usr/sbin/fetch-crl -l /etc/grid-security/certificates/ -o /etc/grid-security/certificates/
</pre>

=== Unable to create proxy ===

Error message:

<pre>
[mmaes@mtop2 ~]$ voms-proxy-init --voms cms:/cms/becms --valid 190:00
Cannot find file or dir: /user/mmaes//.glite/vomses
Enter GRID pass phrase:
Your identity: /C=BE/O=BEGRID/OU=ELEM/OU=VUB/CN=Michael Maes
Creating temporary proxy ............................ Done
Contacting lcg-voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch] "cms" Failed

Error: Could not establish authenticated connection with the server.
globus_gss_assist token :-1: read failure: Operation not permitted
</pre>

Possible Solution (as root)
<pre>
Update the system clock: ntpdate ntp.telenet.be
</pre>
=== Connection to Grid Infrastructure ===

==== Connection problem to behar*.iihe.ac.be ====

Possible error:

<pre>
Connection problems to behar* using srmls/srmcp etc.
</pre>

Solution

<pre>
On mtop:
sudo su -
route
on m7
/sbin/route

Compare the two outputs to see if there are some behar routes missing on mtop. If so add them like this:

route add -host beharXXX.iihe.ac.be gw beharXXX.wn.iihe.ac.be

Then add this line to /root/routes

</pre>

{{TracNotice|{{PAGENAME}}}}

MtopTroubleShooting - Revision history

Maintenance script: Created page with " == Troubleshooting the Mtop machine == Note: these instructions are for root users only! PageOutline === Restarting TopDB Critical services after mTop reboot === ..."