Determine Scratch Space to be allocated

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
cdw
Premium Member
Premium Member
Posts: 9
Joined: Wed Jul 01, 2009 10:02 am

Determine Scratch Space to be allocated

Post by cdw »

Hi,

We are in the design phase for an application and have the basic job designs ready.

The ADMIN team has requested for information on how much scratch space and resource disk space would we need for our project.

What is the best way to estimate this and provide requisite estimates?

We know the stages, datasets etc that we will be using. Based on this, we did a primary calculation of record length * no of records per stage type.
But during actual execution it is possible that several jobs may run in parallel. Also causing multiple stages (operator processes) to run in parallel.

How then do we determine the actual scratch and resource disk that should be sufficient during execution?

Regards,
Bakul
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

You should also take into account the size of buffering between stages used (if your target is significantly slower than your source) as well as repartitioning, sorts (both implicit and explicit) and space used in lookups.
Post Reply