DcachePoolConfig1912

From T2B Wiki
Jump to navigation Jump to search

On quattor svn

  • copy/paste the content of an existing dcache pool profile (e.g. profile_behar080.iihe.ac.be)
  • as we want to use private only address
    • make sure the gridftp door is disabled:
variable DOOR_GRIDFTP = false;
    • and do the necessary change (static route in maite) to ensure public FQDN (e.g behar080.iihe.ac.be) is mapped to private FQDN (e.g behar080.wn.iihe.ac.be)
      • suggestion: maybe better than static routes, add rules to /etc/hosts in maite. This should be done via quattor of course.
  • in the profile tpl uncomment the line that will make spma failing to allow us to setup the XFS for the pool partitions
"/software/packages"=pkg_del("ganglia");
  • make sure add all the corresponding changes (e.g. in databases.tpl, os_version_db.tpl,... ) in both CB5 and CB6
    • suggestion: use search command on svn (example: search on behar080)
  • commit, runcheck and prepare for installation (aii-shellfe usual commands)

On the pool

  • on the RAID adapter BIOS configure the RAID arrays
    • use stripe size of 128kB
    • At most 12 disks in RAID6
    • enable cache write back feature
    • do this for every RAID array
    • after saving the RAID building process may take hours/days to complete
  • restart the box and install (first pass)
  • spma will fail because we explicitly introduce an rpm conflict
  • copy the content of the variable MAKE_FS_SCRIPT to make_xfs.sh
  • adapt the script to the new hardware specs
    • variable sunit=256 for 128kB stripe size
    • variable data_disks=10 for 12 disks in RAID6
  • run the script
chmod 755 make_xfs.sh
./make_xfs.sh

Notice it may not work entirely at the first try because of various reasons. Make sure all the steps are correctly executed. 

If not do the necessary changes and commit these changes back to the profile tpl for the record and for further use
  • prepare the pool to restart
    • in the profile tpl comment the line blocking spma and commit to svn
#"/software/packages"=pkg_del("ganglia");
    • run the following command to allow quattor to install the rest of the componenents
mv ks-post-install.log  ks-post-install.log.1
reboot
  • After it restarts follow the quattor configuration (second pass) with:
tail -f /var/log/ncm-cdispd.log
  • When the configuration is finished look at /storage/1/. If it is empty you will have to configure the dcache manually:
    • adapt in this file the java version according to the one actually installed
java=/usr/java/jdk1.6.0_26/bin/java
    • edit /opt/d-cache/etc/dcache.conf and paste the following content
dcache.layout=${host.name}
dcache.namespace=pnfs

# The following is taken from the old dCacheSetup file.
# Some configuration parameters may no longer apply.


dcache.home=/opt/d-cache


dcache.java.memory.heap=6144m
dcache.java.memory.direct=1024m
net.wan.port.min=20000
net.wan.port.max=25000
net.lan.port.min=33115
net.lan.port.max=33145
pool.dcap.port=0
dcache.java.options.extra=-server 

broker.host=maite.iihe.ac.be

metaDataRepository=org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository
metaDataRepositoryImport=org.dcache.pool.repository.meta.file.FileMetaDataRepository
ftpPort=${portBase}126
dCapGsiPort=${portBase}128
parallelStreams=10
srmBufferSize=1048576
srmTcpBufferSize=1048576
dcache.log.dir=/var/log/dcache
billingToDb=yes


    • edit /opt/d-cache/etc/layouts/behar080.conf and paste the following content
[behar080Domain]
[behar080Domain/pool]
name=behar080_1
path=/storage/1/pool
waitForFiles=${path}/setup
lfs=precious
tags=hostname=behar080
[behar080Domain/pool]
name=behar080_2
path=/storage/2/pool
waitForFiles=${path}/setup
lfs=precious
tags=hostname=behar080
[behar080Domain/pool]
name=behar080_3
path=/storage/3/pool
waitForFiles=${path}/setup
lfs=precious
tags=hostname=behar080
[behar080Domain/pool]
name=behar080_4
path=/storage/4/pool
waitForFiles=${path}/setup
lfs=precious
tags=hostname=behar080
[behar080Domain/pool]
name=behar080_5
path=/storage/5/pool
waitForFiles=${path}/setup
lfs=precious
tags=hostname=behar080
[behar080Domain/pool]
name=behar080_6
path=/storage/6/pool
waitForFiles=${path}/setup
lfs=precious
tags=hostname=behar080

[dcap-behar080Domain]
[dcap-behar080Domain/dcap]

[gsidcap-behar080Domain]
[gsidcap-behar080Domain/gsidcap]
    • create the pools
/opt/d-cache/bin/dcache pool create --size=27880G --lfs=precious /storage/1/pool behar082_1 behar082Domain
/opt/d-cache/bin/dcache pool create --size=27880G --lfs=precious /storage/2/pool behar082_2 behar082Domain
/opt/d-cache/bin/dcache pool create --size=22300G --lfs=precious /storage/3/pool behar082_3 behar082Domain
/opt/d-cache/bin/dcache pool create --size=27880G --lfs=precious /storage/4/pool behar082_4 behar082Domain
/opt/d-cache/bin/dcache pool create --size=27880G --lfs=precious /storage/5/pool behar082_5 behar082Domain
/opt/d-cache/bin/dcache pool create --size=22300G --lfs=precious /storage/6/pool behar082_6 behar082Domain

  Recompute the size (18438g and 14750g) for 3TB disks and keep a safe margin (few MB smaller that the expected netto formatted size)

  22300 for /1 ;27880 for /2 and /3
    • view the result
[root@behar082 pool]# /opt/d-cache/bin/dcache pool ls
POOL       DOMAIN         SIZE   FREE   PATH            
behar082_1 behar082Domain 27880G 27944G /storage/1/pool 
behar082_2 behar082Domain 27880G 27944G /storage/2/pool 
behar082_3 behar082Domain 22300G 22355G /storage/3/pool 
behar082_4 behar082Domain 27880G 27944G /storage/4/pool 
behar082_5 behar082Domain 27880G 27944G /storage/5/pool 
behar082_6 behar082Domain 22300G 22355G /storage/6/pool
    • Start dcache deamons
/opt/d-cache/bin/dcache start
    • back on the pool, configure dcache via quattor
ncm-ncd --co dcache
    • configure symlink via quattor
ncm-ncd --co symlink
    • last steps
scp ccqg:./dcache-core /etc/rc.d/init.d/dcache-core
scp ccqg:./dcache-pool /etc/rc.d/init.d/dcache-pool
chkconfig --level 2345 dcache-core on
chkconfig --level 2345 dcache-pool on
    • reboot
    • after it restarts follow the configuration
tail -f /var/log/ncm-cdispd.log
    • you may have to stop gridftp it is not yet the case
/opt/d-cache/bin/dcache status
/opt/d-cache/bin/dcache stop
/opt/d-cache/bin/dcache start
/opt/d-cache/bin/dcache stop gridftp
ncm-ncd --co dcache
    • run on maite
ncm-ncd --co dcache


Template:TracNotice

Retrieved from ‘https://t2bwiki.iihe.ac.be/index.php?title=DcachePoolConfig1912&oldid=82’