operator combination
Moderators: chulett, rschirm, roy
operator combination
Where can i enable operator combination in datastage?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Thanks ray. Let me rephrase the question. I want to know where can i enable or disable that option. I got to know that if i use many lookups continuously for each reference table and if i enable operator combination option, all those will be combined and treated as one.
If i disable it, i will get the performance benefit using pipelining concept but no of lookup stages increases.
So performance remains the same for both the cases itseems. If operator combination is enabled by default, whats the use of separate lookups, because we use it separately in case of huge data in reference tables. lookup operates with the help of memory. And by default its entire partitioning. What if memory gets occupied to process one reference table itself. We go for separate lookups.
But op combination makes it one. So lookup will fail because of "out of memory"?
Just wanted to check by disabling and enabling that option. So asked for where it is.
If i disable it, i will get the performance benefit using pipelining concept but no of lookup stages increases.
So performance remains the same for both the cases itseems. If operator combination is enabled by default, whats the use of separate lookups, because we use it separately in case of huge data in reference tables. lookup operates with the help of memory. And by default its entire partitioning. What if memory gets occupied to process one reference table itself. We go for separate lookups.
But op combination makes it one. So lookup will fail because of "out of memory"?
Just wanted to check by disabling and enabling that option. So asked for where it is.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The ability to be combinable can be set in individual stages on the Advanced tab. Global operator combination can, as Craig notes, be set using the appropriate environment variable, which I always add to parallel jobs as a parameter.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Definitely "maybe". There are too many other factors. Disabling operator combination is primarily to address systems that are CPU bound (processors being asked to do too much work because two many operators have been combined into one). But it's fair to say that this could also lead to an "out of memory" problem. I'd be looking elsewhere for the out of memory issue, though, particularly at memory-hungry stage types such as Lookup or hash mode Aggregator. Read the entire error message to determine from which stage (operator) the error was thrown.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.