Performance issues - help with clean-up of files in /Scratch

[edmtenzing]

Hi all,

I am a Data Warehouse Analyst using IBM Datastage v8.0 Build 1.2. This past month we have noticed a degradation in job run performance, and some of our more critical processes are taking a very long time to run (in comparison to run-times observed throughout last year).

I have been reading IBM support articles and help on DSxchange to identify the cause, and some suggested fixes for performance issues point to cleaning up the scratch directory. We have a clean-up script which removes all .log files older than 7 days and .txt files older than 61 days. (Extra: note that our datastage application is running on Solaris 10). The shell script we run executes the following commands:

find /app/IBM/InformationServer/Server/Scratch/ \( -name '*.log' \) -
mtime +7 -exec rm {} \;

find /app/dw/prd/log/cleanup/ \( -name 'run_CleanUp_Scratch_Dir*.txt'
\) -mtime +61 -exec rm {} \;

However we have no process to clean up ctl, bad and part files and we have now accumulated 3500 of these dating back to 2009. What is the preferred method for cleaning up these files? Can they be deleted periodically once they reach a certain age as above or is there more to consider when removing these?

Your help is most appreciated.

Thanks

[ray.wurlod]

Just extend your script, or the commands in the script, to pick up the files with the additional suffixes.

[edmtenzing]

Hi Ray, thanks for your response. I'm aware I can extend the script to clean up these files, but my question is more around at what point can they be deleted? i.e. older than X days? I wouldn't want to be cleaning up .ctl files or the like while/if they are being used by the application. Cheers

[chulett]

The moment the job completes those files are obsolete, so use something reasonable as the time component for them. Throw them into your "seven days" pile worst case.