clearing scratch space

madhukar · Post by **madhukar** » Sun Mar 04, 2007 8:46 am

Hi All,

How can i clear the files under scratch directory
can i just remove with rm command?

Thx

DSguru2B · Post by **DSguru2B** » Sun Mar 04, 2007 10:29 am

Do an all term search on key words 'scratch space clean up'.

ray.wurlod · Post by **ray.wurlod** » Sun Mar 04, 2007 10:33 am

Are any jobs running that might be using the scratch space? Scratch space should be cleaned up automatically - if it's not you have jobs that are aborting, and should do something about that.

madhukar · Post by **madhukar** » Sun Mar 04, 2007 12:15 pm

yes ray, you got it right all my jobs were fine during development.

when testing with large volume of data jobs are aborting with SIGBUS errors.
and scratch space full.

Is there any way i can clean up those files (aborted jobs) and run the jobs again.

ray.wurlod · Post by **ray.wurlod** » Sun Mar 04, 2007 8:41 pm

Unless you're running out of space there's no rush to clean them up. But, if you want to, simply use rm when there's no jobs running.

kumar_s · Post by **kumar_s** » Sun Mar 04, 2007 11:43 pm

As noted by Ray, you can go ahead and clear the files avialble in TEMP dir or Scratch disk when the jobs are not running. Those are temporay files used by intermediate stages available in the jobs like SORT, Aggregator etc.,
It can be isolated to different node pool via Config file.

madhukar · Post by **madhukar** » Mon Mar 05, 2007 4:50 am

removed the files in scratch space but still it gives 99% full (previously also)
Am i missing anything?

DSguru2B · Post by **DSguru2B** » Mon Mar 05, 2007 8:02 am

You need more space. As Ray puts it, supply and demand.

ray.wurlod · Post by **ray.wurlod** » Mon Mar 05, 2007 8:10 am

madhukar wrote:removed the files in scratch space but still it gives 99% full (previously also)
Am i missing anything?

Yes, the 99%.
Use du -s to identify which directories are taking the resources.

kumar_s · Post by **kumar_s** » Mon Mar 05, 2007 5:08 pm

Also check the config file, if you have given a different path for Sort node pool and scratch space.