Performance Issue
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
One job, but with multiple Transformer stages (say, not more than four lookups per Transformer stage) would also be OK. You can enable inter-process row buffering (and, if desired, interpolate IPC stages to make it obvious that there are separate processes involved). Of course, splitting into multiple processes is not a great gain if you only have a single CPU.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Thanks for your responses. Ray i have a question
which one is the best one
In Single job with multiple Tranasformer stages or spliting the big job into multile jobs
which one is the best one
In Single job with multiple Tranasformer stages or spliting the big job into multile jobs
ray.wurlod wrote:One job, but with multiple Transformer stages (say, not more than four lookups per Transformer stage) would also be OK. You can enable inter-process row buffering (and, if desired, interpolate IPC stages to make it obvious that there are separate processes involved). Of course, splitting into multiple processes is not a great gain if you only have a single CPU.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I have to agree with Ray, One job, multiple transformations.
Load as many Hashed tables into RAM (enable caching) as possible.
If volumes are huge you can even containerise the lookup transforms, split the stream using a link partitioner, process each stream through the lookups and then use a link collector to pring it all together.
This is not required if the hashed files load to ram. Use Administrator to increase the cache buffer size.
I like to hammer the cpu out of a box, remember a lost cpu cycle is a waster cpu cycle. Try to keep the box at about 5% idle.
Load as many Hashed tables into RAM (enable caching) as possible.
If volumes are huge you can even containerise the lookup transforms, split the stream using a link partitioner, process each stream through the lookups and then use a link collector to pring it all together.
This is not required if the hashed files load to ram. Use Administrator to increase the cache buffer size.
I like to hammer the cpu out of a box, remember a lost cpu cycle is a waster cpu cycle. Try to keep the box at about 5% idle.
Andrew
Think outside the Datastage you work in.
There is no True Way, but there are true ways.
Think outside the Datastage you work in.
There is no True Way, but there are true ways.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
How many rows? Rows/sec is an almost meaningless metric for a whole lot of reasons. For example, the start-up time and close-down time are counted in the elapsed time, even though no rows are processed. For a small number of rows, you might also be waiting for the (IPC) buffers to fill.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi Ray,
Not only row/sec figure, the elapse time is also taking longer than no inter process row option.
Hi ArndW,
Just happened addressing to ray on reading his advice. I really appreciage your help.
Development server has 1 processor, production has 4, so expected to have better performance in production. Can it be concluded processor issue here?
Thanks all for advice,
Not only row/sec figure, the elapse time is also taking longer than no inter process row option.
Hi ArndW,
Just happened addressing to ray on reading his advice. I really appreciage your help.
Development server has 1 processor, production has 4, so expected to have better performance in production. Can it be concluded processor issue here?
Thanks all for advice,