how to delete these scratch files after sort

ashcar · Post by **ashcar** » Mon Oct 23, 2006 3:32 pm

I have long list of directories in my project folder which i suspect are scratch files after sort,
/proj/dsadm/Ascential/DataStage/Projects/Dvl

so-O3nUa
so-_mnia
so3vQjEa
so4JQa7a
so4xedya
so5hYbaa
so9-Qoya
soPDijMa..................................

these directories cannot be deleted even by the owner of directory or by dsadm, nor can you peek inside those directories.
What are those directories?
Is it a residue after sort files are done ?
why does datastage not clear them once sort is completed ?
is there anyway that scratch files like these can be deleted ?

ray.wurlod · Post by **ray.wurlod** » Mon Oct 23, 2006 4:15 pm

WHY can't they be deleted? What's the permission on them? What's the permission on the parent directory? Are they still in use (use fuser to find out)? What error message is generated if you try to delete one of them?

chulett · Post by **chulett** » Mon Oct 23, 2006 5:26 pm

The best way to delete these sort temp files from the Project is to not create them there in the first place!

There is a property in the Sort stage that tells it where to put them when it runs. By default it is blank which means the Project. Never leave it blank. I always set that property to "/tmp" and then the SA's have processes in place to clean things out from there after a certain number of days.

ps. If you don't have the permissions you need to delete them, ask a 'super-user' to do it for you. Or seek The Sword of a Thousand Truths.

kumar_s · Post by **kumar_s** » Mon Oct 23, 2006 6:28 pm

There are chances for the files to be created and not cleared by the jobs when the jobs aborted for that run.

chulett · Post by **chulett** » Mon Oct 23, 2006 7:11 pm

And sometimes they're just not cleared, period. Best to create them off someplace else where you don't have to worry about them.

ray.wurlod · Post by **ray.wurlod** » Tue Oct 24, 2006 9:20 am

... unless /tmp is very small, which it often is on Solaris especially (and they mount swap there also!)

ashcar · Post by **ashcar** » Tue Oct 24, 2006 12:05 pm

ray.wurlod wrote:... unless /tmp is very small, which it often is on Solaris especially (and they mount swap there also!)

Our filesystem space for /tmp in AIX 5.3 is very small so instead will take the Crontab approach to find and delete allover the DataStage projects.

every user who ran sort jobs had directories with their name and permissions to owner only

/proj/dsadm/Ascential/DataStage/Projects/Dvl
drwx--S--- 2 tice dstage 256 Jun 07 13:38 sod5em7a
drwx--S--- 2 xxxxxxx dstage 4096 Mar 29 2006 soegImia
drwx--S--- 2 xxxxx dstage 256 Jun 07 14:59 sofxYeMa
drwx--S--- 2 xxxxx dstage 4096 Oct 12 2005 sog_mpya

these are old directories and so asked the SA to change persmissions so dsadm can open/delete them
find them using
$ find . -name [io]pf__* -ctime +30 -exec dirname {} \; | sort -u

Proposed Crontab entry: find . -name [io]pf__* -ctime +30 -exec rm {} \;
**** --- this removes the files but leaves the directory. . . May have to do a rm -r $(dirname {})

the dirname is used with [io]pf__* in commands above to make sure only sort temp file directories are deleted .

All these started because we ran out of disk space but never knew who was hogging space

ray.wurlod · Post by **ray.wurlod** » Tue Oct 24, 2006 1:55 pm

You might need hard quotes around the wildcarded -name option for the find command

Code: Select all

find . -name '[io]pf__*' -ctime +30 -exec rm {} \;