Estimating Memory Size

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Munish
Participant
Posts: 89
Joined: Sun Nov 19, 2006 10:34 pm

Estimating Memory Size

Post by Munish »

Hi All,
We need to estimate the virtual memory size (Scratch) for our project.
Can anyone provide some guideline or best practice to evaluate that.
Do we have any formulas to get those figures.
We are assuming that memory size is decided by
1. Input Data size.
2. Number of stages
3. Number of client accessing server.

Any input will be appreciated.

Thanks and regards,
Munish
MK
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

Do different levels of search on key words like "memory estimate" or "scratch space estimate". This has been discussed before.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Do you want to estimate virtual memory requirements or scratch disk requirements? They are not the same things. For example, scratch disk can be used when allocated memory is exceeded, even though there is plenty of free virtual memory - for example in the Sort stage you can limit the amount of virtual memory (by default to 20MB per partition); as soon as this is exceeded then the scratch disk is used. "True" virtual memory, on the other hand, is swapped out to the swap disk when process exchange occurs, and pages out to paging disk when exhausted in the one process. I don't believe you're clear on your terminology.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply