OLIVIER DEVROEDE omdevroe@vub.ac.be: /* Chimera commands */

2017-12-13T13:31:05Z

Chimera commands

← Older revision		Revision as of 13:31, 13 December 2017
Line 11:		Line 11:
	cation='behar081_5';" --output list-of-files-that-should-be-on-behar081_5-according-to-Chimera		cation='behar081_5';" --output list-of-files-that-should-be-on-behar081_5-according-to-Chimera
	</pre>		</pre>

			Far more chimera shortcuts can be found in [https://indico.desy.de/indico/event/11773/contribution/6/material/slides/1.pdf this talk] by Dmitry Litvintsev

	=== postgres tuning ===		=== postgres tuning ===
	==== benchmark ====		==== benchmark ====

OLIVIER DEVROEDE omdevroe@vub.ac.be at 11:04, 6 December 2017

2017-12-06T11:04:20Z

← Older revision		Revision as of 11:04, 6 December 2017
Line 5:		Line 5:
	*SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html		*SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html

			=== Chimera commands ===
			Find all files that are supposed to be on a certain pool
			<pre>
			psql --no-align --tuples-only --username postgres --dbname chimera --command "select inode2path(ipnfsid) from t_inodes t1, t_locationinfo t2 where t1.inumber=t2.inumber and t2.ilo
			cation='behar081_5';" --output list-of-files-that-should-be-on-behar081_5-according-to-Chimera
			</pre>
	=== postgres tuning ===		=== postgres tuning ===
	==== benchmark ====		==== benchmark ====

Maintenance script: Created page with " === monitoring === SRM monitor http://maite.iihe.ac.be:8098/billing/xml/ dcache head page http://maite.iihe.ac.be:2288 *SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2...."

2015-08-26T12:28:26Z

Created page with " === monitoring === *SRM monitor http://maite.iihe.ac.be:8098/billing/xml/ *dcache head page http://maite.iihe.ac.be:2288 *SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2...."

New page

=== monitoring ===
*SRM monitor http://maite.iihe.ac.be:8098/billing/xml/
*dcache head page http://maite.iihe.ac.be:2288
*SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html

=== postgres tuning ===
==== benchmark ====
*http://edoceo.com/liber/db-postgresql-benchmark
*switch to postgres user
<pre>
su - postgres
</pre>
*initialise the benchmark db
**<tt>-s</tt> for maximum of active clients
<pre>
createdb testdb
pgbench -i testdb -s 25
</pre>
*run the benchmark
<pre>
pgbench -c 10 -t 100 testdb
</pre>

<pre>
#!/bin/bash

## number of clients
clients="1 10 25 50 75 100"
#clients="1 2"
## number of transactions
trans="100 500 1000 2000 5000 10000 50000"
#trans="100 200"
## scaling factor used for intialisastion
scaling_fact="1 5 10 15 20"
#scaling_fact="1 2"

## number of retries
stats=100
#stats=5

for sc in $scaling_fact
do
dropdb testdb >& /dev/null
createdb testdb >& /dev/null
pgbench -i testdb -s $sc >& /dev/null
for c in $clients
do
for t in $trans
do
tot_i=0
for s in <tt>seq 1 $stats</tt>
do
pgbench -c $c -t $t testdb >& /tmp/out-pgbench
echo "pgbench -c $c -t $t testdb" >> /tmp/out-pgbench
tps=<tt>cat /tmp/out-pgbench|grep tps|grep including|sed 's/.* $[0-9]\+$.*/\1/'</tt>
echo "TPS_INTERMEDIATE= $tps Run $s Clients $c Transfers $t Scaling $sc"
tot_i=$(($tot_i + $tps))
done
tot_i=$(($tot_i/$stats))
echo "TPS= $tot_i Clients $c Transfers $t Scaling $sc"
done
done
done

</pre>
==== tuning ====
==== disk layout ====
*WAL logs on a different disk or not?
**4 disk setup
**1 device in 4 disk RAID10
***higher throughput, interesting for reading (i guess)
**2 devices in 2 * 2 disk RAID1
***WAL/OS on a separate disk, WAL on own partition
***OS/other software doesn't do a lot of IO (or lets hope it's not all fsync'ed)
***write-through for WAL device
***small sized partition for WAL is fine
****http://www.postgresql.org/docs/8.2/interactive/wal-configuration.html
****current production system (postgres 8.1) has 7 of these files (as in documentation)
***no journalling on WAL partition
***mount with noatime and symlink to pg_xlog
***DB on another
***write-back on
***XFS
==== disk benchmarks ====
*small script to test iozone performance.
<pre>
#!/bin/bash

## size in MB
size_step=5
size_steps_start=2
size_steps_nr=2

## max number of files, step of 1
steps=2
dest="/var/lib/pgsql/test /var/tmp/test"

out=iozone-out-<tt>date +%s</tt>

for d in $dest
do
for j in <tt>seq $size_steps_start $(($size_steps_start+$size_steps_nr))</tt>
do
size=$(($j*$size_step))
ioz="iozone -eM -s${size}M -r64k -i0 -i1 "

for i in <tt>seq 1 $steps</tt>
do
files=""
rm -Rf $d >& /dev/null
mkdir -p $d >& /dev/null
for t in <tt>seq 1 $i</tt>
do
files="$files $d/out$t"
done
echo "BEGIN" >> $out
echo <tt>date</tt> >> $out
echo "$ioz -t $i -F $files" >> $out
$ioz -t $i -F $files >> $out 2>&1
echo <tt>date</tt> >> $out
echo "END" >> $out
done
rm -Rf $d >& /dev/null
done
done

</pre>
*on solaris, replace seq sequences with
<pre>
perl -e "\$,=' '; print 1 .. $i;"
</pre>
==== links ====
*http://edoceo.com/liber/db-postgresql-performance
*http://www.postgresql.org/docs/8.2/interactive/kernel-resources.html
*http://www.powerpostgresql.com/PerfList
*http://www.gmod.org/wiki/index.php/PostgreSQL_Performance_Tips

=== postgres DBs ===
==== dump ====
==== pg_dump ====
*This is compressed and allows usage of pg_restore (eg to only restore the data).
<pre>
pg_dump -F c -f <destinationfile> <name of db>
</pre>

==== pg_dumpall ====
*brute force
*no selective restore (easily) possible
*the <tt>gzip</tt> part is for compression
<pre>
pg_dumpall | gzip > <destination file>
</pre>

==== recover ====
==== pg_restore ====
**example 1: only restore data
**assumes the schema is there through some other means
**name of DB is extracted from filedump
**needs output from pg_dump with <tt>-F c</tt>
<pre>
pg_restore -a -F c -d <name of DB> <name of file>
</pre>

==== from pg_dumpall ====
*needs minimal postgres setup
**best to stop postgresql server, move /var/lib/pgsql/data and reinitialise
*as user postgres
<pre>
su - postgres
zcat <path to gzipped full dump> | psql postgres
</pre>

==== overwrite current DBs ====
<pre>
db=test1
pg_dump $db > ${db}-new
psql $db < /path/to/$db
</pre>

==== dropping current DBs ====
*this can cause problems. reverted to previous state with the <tt>-new</tt> saves.
<pre>
db=test1
pg_dump $db > ${db}-new
dropdb $db
createdb -T template0 $db
psql $db < /path/to/$db
</pre>

==== Upgrade dcache 1.7 to 1.8 on same machine for quattor users ====
*trigger a stop for ncm-dcache
<pre>
touch /opt/d-cache/DONT_RUN_NCM_DCACHE
</pre>
*stop dcache/pnfs
<pre>
/etc/init.d/dcache-pool stop
/etc/init.d/dcache-core stop
/etc/init.d/pnfs stop
</pre>
*make a backup
**first step of procedure below
*make the upgrade
**build and notify the new configuration
**ncm-dcache (version > 3.0.0-11) won't do anything.
*do all remaining steps below.
*before rerunning ncm-dcache, remove the file <tt>/opt/d-cache/DONT_RUN_NCM_DCACHE</tt>
==== Move to new server using quattor/ncm-postgresql/ncm-dcache ====
*Do a full dump on the original host (compressed for space savings):
<pre>
mkdir -p /var/post_backup/
chown postgres.postgres /var/post_backup/
dest=/var/post_backup/full_<tt>date '+%s'</tt>
tar cvf ${dest}-pnfsdb.tar /opt/pnfsdb
su - postgres -c "pg_dumpall | gzip > $dest.gz"
</pre>
*On the new machine:
**setup exact same config (so don't add new pnfs-dbs yet!)
**stop dcache, pnfs and postgresql
**remove postgres DBs
<pre>
cd /var/lib/pgsql
mv data data-unused-<tt>date +%s</tt>
</pre>
**reinitialise DBs
**<tt>postgres < 8.2</tt>
<pre>
/etc/init.d/postgresql start
</pre>
**<tt>postgres >= 8.2</tt>
<pre>
/etc/init.d/postgresql initdb
/etc/init.d/postgresql start
</pre>
**copy postgres DB to machine and inject real data
<pre>
su - postgres
zcat <full_...gz> | psql postgres
</pre>
**if this is also a major upgrade
**major means eg from 1.7.0-X to 1.8.0-Y (unless release notes say otherwise)
**drop non-pnfs DBs
***become postgres user
<pre>
su - postgres
</pre>
***list all DBs
<pre>
psql -l
</pre>
***look for dcache DBs that are not pnfs DBs
***typical examples are <tt>dcache</tt>,<tt>billing</tt> and <tt>replicas</tt>
***the <tt>companion</tt> DB should be left alone
***pnfs DBs are <tt>admin</tt>,<tt>data1</tt>, whatever added yourself
***base postgres DBs (should be left alone!) <tt>postgres</tt>,<tt>template0</tt>,<tt>template1</tt>
***you can always redo this if you still have the postgres dump
***drop the dcache non-pnfs DBs
<pre>
for i in dcache billing replicas
do
dropdb $i
done
</pre>
**run ncm-postgres (as root), it will recreate the required dcache DBs
<pre>
ncm-ncd --co postgresql
</pre>
**restore the pnfsdb files (as root)
<pre>
cd /
mv /opt/pnfsdb /opt/pnfsdb-orig-<tt>date +%s</tt>
tar xvf <path to pnfsdb.tar file>
</pre>
**you should be able to start pnfs and list the content of the directories again
**you can now also rerun ncm-dcache

=== problems ===
==== Staging forever ====
*Try to locate the pools of the file
**is the pool still up
**is the file stil in pool/data
**can you copy it
**LAST MEASURE: restart dcache-pool service
*in PoolManager
**rc ls <pnfsid>.*
**what state is it in
***something with Pool2Pool
***wait, it's transferring
***if stays there, go to destination pool and try to kill the p2p transfer
***if that fail, try with rc destroy <the whole pnfsid identifier in rc ls> ed <tt>rc destroy 000C00000000000000BEEFE0@0.0.0.0/0.0.0.0-*/*</tt>

==== TransferManager error too many transfers! ====
<pre>
02/13 14:40:48 Cell(SRM-maite@srm-maiteDomain) : org.dcache.srm.scheduler.NonFatalJobFailure: org.dcache.srm.SRMException: TransferManager errortoo many transfers!
</pre>

<pre>
cd RemoteGsiftpTransferManager
set max transfers <#max transfers>
</pre>

==== 99 Repository got lost ====
When the pool is gone, it can be due to a lot of things.
*XFS problem
**check <tt>dmesg</tt> or <tt>/var/log/messages</tt> for error messages like
<pre>
XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /usr/src/redhat/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff88251392
</pre>
<pre>
xfs_force_shutdown(sdb1,0x8) called from line 4267 of file /usr/src/redhat/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_bmap.c. Return address = 0xffffffff8825e2d0
Filesystem "sdb1": Corruption of in-memory data detected. Shutting down filesystem: sdb1
Please umount the filesystem, and rectify the problem(s)
</pre>
**if it's the case do the following (and in this order)
**stop dcache on the node
<pre>
/etc/init.d/dcache-pool stop
/etc/init.d/dcache-core stop
</pre>
**unmount the storage
<pre>
umount /storage/1
</pre>
**try to remount it
<pre>
mount /storage/1
</pre>
**check dmesg if it was succesful. the last line should read something like
<pre>
XFS mounting filesystem sdb1
Ending clean XFS mount for filesystem: sdb1
</pre>
**if not, run xfs_repair
<pre>
xfs_repair /dev/sdb1
</pre>
***if it was succesful, remount the storage and check dmesg if the mount was succesful
***restart dcache on the node
<pre>
/etc/init.d/dcache-pool start
/etc/init.d/dcache-core start
</pre>
***if not, it can end with eg
<pre>
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
</pre>
***then rerun xfs_reapir with -L option (wyou already retried mounting a few steps ago)
<pre>
xfs_repair -L /dev/sdb1
</pre>
***after it's done, remount and check dmesg for succesful mount
***restart dcache
<pre>
/etc/init.d/dcache-pool start
/etc/init.d/dcache-core start
</pre>

=== Inner working of the PSU ===
From an email by Patrick Fuhrmann:
<pre>
Ok here we go :
The procedure to find all matching pools :
The request comes with a precise
(StorageClass,dCacheClass,IPnumber)
In the future there will be the protocol as well.
Now we have to find the corresponding unit :
Store Unit :
We try to find an exact match of the incoming
StorageClass in the list of storage classes (unit -store).
If not found we try to find some combinations of
wildcards ( *@osm *.* ).
dCache unit :
Same as Store Unit but not used by anybody yet.
Network Unit :
The incoming IP number is compared with all
net units. We start with the 255.255.255.255 and
proceed with smaller masks until we reach the
0.0.0.0 or we find a match. (like ip routers do).
If we find a match we stop. As said in the previous e-mail,
if 131.169.1.1 comes in and there are two net-units
0.0.0.0/0.0.0.0 and 131.169.0.0/255.255.0.0 we choose
131.169.0.0/255.255.0.0 (At this point we don't know
anything about links yet). If there is no 131.1.. net-unit
the result of this step is the 0.0.0.0/0.0.0.0 net-unit.

Now we try to find ALL links which require a Store-Unit
and a NetUnit AND match the units found in the above step.
Next we add those links which only require Store-Unit or
Net-Unit and match the unit found above. (As a matter of fact
the is this additional criterium, which is the transfer
direction but this don't help us to understand the mechasm)
Now we take all links found above and sort according to the
preference of this transfer direction. For big setups this
results in a matrix of links
pref 100 : link1 link2 link3
pref 50 : link4 link5
pref ....
Now we resolve the links into pools (via pool groups)
Result is
pref100 : pool1, pool2 .....
.....
If at least one pool of the highest preference is available
we send the resulting set of pools (of highest preference
to the CM) rest is clear
If non of highest preference is available we step to the next
preference (a.s.o)
This mechanism allows to construct all possible setups.
As said before : The treatment of the net-unit seems
strange but is really necessary if you think about it
for awhile.
</pre>
=== Transfer tests ===
See all remote transfers:
cd RemoteGsiftpTransferManager
ls
=== Problems and Solutions ===
While runing ExDigiStatistics with some PU data, the system is really tested.
==== ulimit -n ====
Some of the jobs gave the following error message:
Server error message for [35]: "Unexpected Exception :
java.net.SocketException: Too many open files" (errno 33).
Failed open file in the dCache.
Apparently this is due to a a restricted number of file descriptors. You can find out the total amount with
ulimit -n
A tip from Michael Ernst: (ulimit -n)/(# of active movers) = 5 or more (now it's 16 ;).
The best thing is to put this in the dcache-pool script and the restart the pool.
mv /opt/d-cache/bin/dcache-pool /opt/d-cache/bin/dcache-pool-orig;cat /opt/d-cache/bin/dcache-pool-orig|sed
's/start)/start)\n\tulimit -n 16384/' > /opt/d-cache/bin/dcache-pool;chmod +x
/opt/d-cache/bin/dcache-pool;/etc/init.d/dcache-pool restart
==== Detected Tx Unit Hang ====
One of the pools got the following kernel message:
e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
This resulted in a redetection of the network with the wrong network speed. Obviously, the adapter was a bit overloaded. A network restart fixed the problem.
==== Channel bonding ====
Using 2 e1000 pci adapters, the maximum number of packets rose from 70k to 90k. Obviously the hardware was a bottleneck (3 adapters on pci, single P4), but still some result nonetheless.
The setup was pretty basic and easy:
*following the setup from <kernel>/Documentation/networking/bonding.txt
*SL3 supports bonding config with initscripts. It's pretty easy and straightforward.
*don't forget to recompile ifenslave when using non-standard kernel.
*i tried bonding-mode abl (wich needs no special switch settings), but the network driver requires set_dev_mac_address (not on the r8169!!), this also needs miimon setting.
--
=== Links ===
*IIHE DCache http://maite.iihe.ac.be:2288/

{{TracNotice|{{PAGENAME}}}}

DCache - Revision history

OLIVIER DEVROEDE omdevroe@vub.ac.be: /* Chimera commands */

OLIVIER DEVROEDE omdevroe@vub.ac.be at 11:04, 6 December 2017

Maintenance script: Created page with " === monitoring === *SRM monitor http://maite.iihe.ac.be:8098/billing/xml/ *dcache head page http://maite.iihe.ac.be:2288 *SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2...."

Maintenance script: Created page with " === monitoring === SRM monitor http://maite.iihe.ac.be:8098/billing/xml/ dcache head page http://maite.iihe.ac.be:2288 *SRM manual https://sdm.lbl.gov/srm-wg/doc/SRM.v2...."