Datastage server Maintainence

kris007 · Post by **kris007** » Thu Aug 24, 2006 1:04 pm

Hi,
We have been developing lot of jobs and our development phase is near completion. But we are facing lots of memory issues, almost all jobs are aborting because of scratch space being full and even the disk space(secondary storage) is also full.I would like to know if there is any standard procedure that is followed to clear/maintain the scratch and disk space and also if there are any other issues that needs to be taken care of.Can any one please throw some light on it.
Thanks

DSguru2B · Post by **DSguru2B** » Thu Aug 24, 2006 1:09 pm

You can check for files which are 7 days old and delete them. You can put this functionality as part of your unix script, if you have one. Clear out the log files regularly.

clickart · Post by **clickart** » Thu Aug 24, 2006 2:23 pm

We too faced a similar issue. We were creating temporary datasets in our jobs which were not deleted and started eating up the space. You can have a UNIX script to delete such temp files or datasets and put it as the last stage in your sequencer in an Execute command stage. This way you can ensure that the temp files are deleted immediately after a successful run.

You can put this logic at the script level but if you put it in the Sequencer, you can "see" it.

kris007 · Post by **kris007** » Thu Aug 24, 2006 2:24 pm

DSguru2B wrote:You can check for files which are 7 days old and delete them. You can put this functionality as part of your unix script, if you have one. Clear out the log files regularly.

Should I also delete datasets that way. And log files is there any otherway to delete other than from director.Can you describe more about them.
Thanks

DSguru2B · Post by **DSguru2B** » Thu Aug 24, 2006 2:28 pm

Offcourse you can. You can clear all files, even delete them at the unix level. For log files, there was a job on ADN that was built to clear all the log files. I am not exactly sure if its still there. 7 days is just a suggestion, you can do it 10 days, 15 days, whatever you like. Its known as the clean up mechanism. My standard practice is to archive the files immediately after the cycle has finished and then cleanup the archived files after 7 days.

kris007 · Post by **kris007** » Thu Aug 24, 2006 2:45 pm

The files in the Scratch space are they temparory dataset or are they sequential files?I want to know this because I cannot clear scratch space from Manager. So from command line should I use orchadmin commands to clear the dataset.

DSguru2B · Post by **DSguru2B** » Thu Aug 24, 2006 2:49 pm

Cant you use unix rm command. I am not exactly sure with PX datasets weather you require orchadmin commands or just the unix rm command would suffice. Can you go in the command line and try deleting one file ?

kris007 · Post by **kris007** » Thu Aug 24, 2006 2:56 pm

DSguru2B wrote:Cant you use unix rm command. I am not exactly sure with PX datasets weather you require orchadmin commands or just the unix rm command would suffice. Can you go in the command line and try deleting one file ?

Well I can delete using UNIX rm command but if the file happens to be a dataset then I would only be deleting teh configuration file and that leaves the real data file hanging somewhere using up the space.
Ideally I would think once the job is finished running it should clear up all the temparory files but looks like there are lot of files left in the scratch space not sure whats causing this.

DSguru2B · Post by **DSguru2B** » Thu Aug 24, 2006 3:08 pm

See, i am not exactly sure about datasets. Never faced a problem as such. Think we need some more experienced advice.

sri1dhar · Post by **sri1dhar** » Thu Aug 24, 2006 6:59 pm

kris007 wrote:[Well I can delete using UNIX rm command but if the file happens to be a dataset then I would only be deleting teh configuration file and that leaves the real data file hanging somewhere using up the space.

The real data files are created in your resource directory.

kumar_s · Post by **kumar_s** » Thu Aug 24, 2006 9:45 pm

Do you have dataset in the temp directory?
Check with "orchadmin describe {dataset}" command to know the details about the dataset.
If it has single data file and that too lies in the same directory, you can delete it. Else check the extension first, and later use "rm" or "orchadmin rm" accordingly.

krystlecsy · Post by **krystlecsy** » Thu Aug 24, 2006 10:02 pm

Is everyone talking about the datasets residing in the destination folder set in the DataStage jobs itself or the DataStage/Datasets folder?

We too face the full space problem. What are the folders to clean up for maintenance? Seems that even when it is cleared everyday, the scratch space is still full!!

ray.wurlod · Post by **ray.wurlod** » Fri Aug 25, 2006 1:35 am

The orchadmin command has an rm subcommand that can be used to remove a Data Set. Provide it with the Data Set's control file name as its argument.

You can not readily purge DataStage logs from the command line without damaging the control records. Use the Director and/or the automatic purge capability.

kumar_s · Post by **kumar_s** » Fri Aug 25, 2006 1:53 am

... without damaging the control records.

Ray - Are you refering to RT_LOG*** file?

ray.wurlod · Post by **ray.wurlod** » Fri Aug 25, 2006 2:09 am

Yes. And RT_STATUS*** as well.

These are hashed files, and therefore directory/*30 structures, rather than operating system files.

They have corresponding VOC entries.

There are other VOC entries containing last modified information.