Installing dcache pool
/!\ This is obsoleted by this page
On quattor svn
- copy/paste the content of an existing dcache pool profile (e.g. profile_behar053.iihe.ac.be)
- as we want to use private only address
- make sure the gridftp door is disabled:
variable DOOR_GRIDFTP = false;
- and do the necessary change (static route in maite) to ensure public FQDN (e.g behar053.iihe.ac.be) is mapped to private FQDN (e.g behar053.wn.iihe.ac.be)
- suggestion: maybe better than static routes, add rules to /etc/hosts in maite. This should be done via quattor of course.
- and do the necessary change (static route in maite) to ensure public FQDN (e.g behar053.iihe.ac.be) is mapped to private FQDN (e.g behar053.wn.iihe.ac.be)
- in the profile tpl uncomment the line that will make spma failing to allow us to setup the XFS for the pool partitions
"/software/packages"=pkg_del("ganglia");
- make sure add all the corresponding changes (e.g. in databases.tpl, os_version_db.tpl,... ) in both CB5 and CB6
- suggestion: use search command on svn (example: search on behar053)
- commit, runcheck and prepare for installation (aii-shellfe usual commands)
On the pool
- on the RAID adapter BIOS configure the RAID arrays
- use stripe size of 128kB
- At most 12 disks in RAID6
- enable cache write back feature
- do this for every RAID array
- after saving the RAID building process may take hours/days to complete
- restart the box and install (first pass)
- spma will fail because we explicitly introduce an rpm conflict
- copy the content of the variable MAKE_FS_SCRIPT to make_xfs.sh
- adapt the script to the new hardware specs
- variable sunit=256 for 128kB stripe size
- variable data_disks=10 for 12 disks in RAID6
- run the script
chmod 755 make_xfs.sh ./make_xfs.sh
Notice it may not work entirely at the first try because of various reasons. Make sure all the steps are correctly executed.
If not do the necessary changes and commit these changes back to the profile tpl for the record and for further use
- prepare the pool to restart
- in the profile tpl comment the line blocking spma and commit to svn
#"/software/packages"=pkg_del("ganglia");
- run the following command to allow quattor to install the rest of the componenents
mv ks-post-install.log ks-post-install.log.1 reboot
- After it restarts follow the quattor configuration (second pass) with:
tail -f /var/log/ncm-cdispd.log
- When the configuration is finished look at /storage/1/. If it is empty you will have to configure the dcache manually:
- scp ccqg:./dCacheSetup /opt/d-cache/config/
- adapt in this file the java version according to the one actually installed
java=/usr/java/jdk1.6.0_26/bin/java
- edit /opt/d-cache/etc/node_config and paste the following content
ADMINDOOR=no ADMIN_NODE=maite.iihe.ac.be DCACHE_BASE_DIR=/opt/d-cache DCACHE_HOME=/opt/d-cache DCAP=yes DIRDOMAIN=no GPLAZMASERVICE=no GRIDFTP=no GSIDCAP=yes HTTPDOMAIN=no INFOPROVIDER=no LMDOMAIN=no NAMESPACE=pnfs NODE_TYPE=pool NUMBER_OF_MOVERS=100 PNFSMANAGER=no PNFS_INSTALL_DIR=/opt/pnfs PNFS_OVERWRITE=no PNFS_ROOT=/pnfs PNFS_START=yes POOLMANAGER=no POOL_PATH=/opt/d-cache/etc SERVER_ID=iihe SRM=no UTILITYDOMAIN=no XROOTD=no
- run the install script
/opt/d-cache/install/install.sh
- edit /opt/d-cache/config/behar053.poollist where behar053 is an example replace accordingly:
[root@behar053 ~]# cat /opt/d-cache/config/behar053.poollist behar053_1 /storage/1/pool sticky=allowed recover-space recover-control recover-anyway lfs=precious tag.hostname=behar053 behar053_2 /storage/2/pool sticky=allowed recover-space recover-control recover-anyway lfs=precious tag.hostname=behar053 behar053_3 /storage/3/pool sticky=allowed recover-space recover-control recover-anyway lfs=precious tag.hostname=behar053
- create the pools
/opt/d-cache/bin/dcache pool create 18438g /storage/1/pool /opt/d-cache/bin/dcache pool create 18438g /storage/2/pool /opt/d-cache/bin/dcache pool create 14750g /storage/3/pool
Recompute the size (18438g and 14750g) for 3TB disks and keep a safe margin (few MB smaller that the expected netto formatted size)
22300 for /1 ;27880 for /2 and /3
- view the result
[root@behar053 ~]# /opt/d-cache/bin/dcache pool ls Pool Domain LFS Size Free Path behar053_1 behar053Domain precious 18438 18438 /storage/1/pool behar053_2 behar053Domain precious 18438 18438 /storage/2/pool behar053_3 behar053Domain precious 14750 14750 /storage/3/pool Disk space is measured in GiB.
- add necessary lines in maite in /opt/d-cache/config/PoolManager.conf and restart dcache deamons
/opt/d-cache/bin/dcache restart
- back on the pool, configure dcache via quattor
ncm-ncd --co dcache
- configure symlink via quattor
ncm-ncd --co symlink
- last steps
scp ccqg:./dcache-core /etc/rc.d/init.d/dcache-core scp ccqg:./dcache-pool /etc/rc.d/init.d/dcache-pool chkconfig --add dcache-core chkconfig --add dcache-pool
- reboot
- after it restarts follow the configuration
tail -f /var/log/ncm-cdispd.log
- you may have to stop gridftp it is not yet the case
/opt/d-cache/bin/dcache status /opt/d-cache/bin/dcache stop /opt/d-cache/bin/dcache start /opt/d-cache/bin/dcache stop gridftp ncm-ncd --co dcache
- run on maite
ncm-ncd --co dcache