DeleteObsoleteFiles

How to Delete Obsolete files

The DCache system should be checked regularly for files that should not be there. There are 3 types that need to be checked: not well deleted files,phedex and orphaned.

1: Not well deleted files

We saw a lot of files who have no more entry in pathfinder (so no path on pnfs to it) but still have info in cacheinfoof. To find all these, follow these instructions:

On ccq, a script was made to find all the files residing on all the pools and then transfer all these lists to maite. There is one list per pool

cd /root/Olivier/Clean_Behars/
rm -f file_list_behar0*
./fetch_file_list

on maite a script was made to find the bad files and prepare a script to clean up.

cd /var/ToClean
rm -f *txt
./DoAll

This script will attempt to clean all the pools that are listed in /root/damadori/OldFiles/Storage So make sure this file is up-to date.

After each pool, the script will give some feedback:

the file is ready for deletion
just run the following command:
ssh -F /root/.ssh/dcache_admin_config admin@localhost -T < todelete_behar030_2.txt

You need to issue these commands one by one to actually delete the files.

2: Discrepancies with respect to Phedex

Make a list of what is on DCache

go to maite.

cd /pnfs/iihe/cms/ph/sc4/
find . -name * -type f >> files_on_maite.txt

This file will serve as input for the Storage Consistency Check. Copy it to frontier.

Make difference with what is in DBS

got to frontier

su - phedex
cd ./phedex/sw/slc4_amd64_gcc345/cms/PHEDEX/PHEDEX_3_2_9/Utilities/
./StorageConsistencyCheck -db /scratch/phedex/phedex/gridcert/DBParm:Prod/BELGIUM --lfnlist /scratch/phedex/phedex/PHEDEX/Utilities/files_on_maite.txt -se_name 'maite.iihe.ac.be' --verbose > consistancycheck.txt

Delete

go to maite again and put the file in ~/Deletions (first empty this dir)

I like to split the file containing what has to go in smaller pieces of 200 to 500 files each:

split -l 300 Difference.txt

Then start deleting

cd /pnfs/iihe/cms/ph/sc4/store
xargs  -a ~/Deletions/xab rm

3: Orphaned

Sometimes files remain on the pool although the entry is deleted into the pnfs database. One needs to make the difference of what is really on the pool and what is on pns.

= On ccq

cd /root/distrib-scripts 
./distrib_exec_ls behars

Make sure that the file 'behars' contains all the physical pool nodes. The script will make a file list per pool and transfer all these files to maite.

On maite

cd /opt/pnfs/tools/
./find-pnfsIDsonpool-butnot-onpnfs.sh

Now you can find all the files to delete (per pool) in /root/Diffs/results