operator combination

dsscholar · Post by **dsscholar** » Fri Sep 23, 2011 3:27 am

Where can i enable operator combination in datastage?

ray.wurlod · Post by **ray.wurlod** » Fri Sep 23, 2011 5:35 am

It should be enabled by default. What makes you think it isn't?

dsscholar · Post by **dsscholar** » Fri Sep 23, 2011 1:43 pm

Thanks ray. Let me rephrase the question. I want to know where can i enable or disable that option. I got to know that if i use many lookups continuously for each reference table and if i enable operator combination option, all those will be combined and treated as one.

If i disable it, i will get the performance benefit using pipelining concept but no of lookup stages increases.

So performance remains the same for both the cases itseems. If operator combination is enabled by default, whats the use of separate lookups, because we use it separately in case of huge data in reference tables. lookup operates with the help of memory. And by default its entire partitioning. What if memory gets occupied to process one reference table itself. We go for separate lookups.

But op combination makes it one. So lookup will fail because of "out of memory"?

Just wanted to check by disabling and enabling that option. So asked for where it is.

chulett · Post by **chulett** » Fri Sep 23, 2011 1:59 pm

Like most everything else in PX Land, there is an APT variable to control that: add APT_DISABLE_COMBINATION to your mapping and set it to True.

ray.wurlod · Post by **ray.wurlod** » Fri Sep 23, 2011 4:00 pm

The ability to be combinable can be set in individual stages on the Advanced tab. Global operator combination can, as Craig notes, be set using the appropriate environment variable, which I always add to parallel jobs as a parameter.

dsscholar · Post by **dsscholar** » Sat Sep 24, 2011 4:34 am

Thanks chulett and ray.

Hi all,

My query has some other clarifications like "out of memory". Any details on that.

ray.wurlod · Post by **ray.wurlod** » Sat Sep 24, 2011 3:30 pm

Definitely "maybe". There are too many other factors. Disabling operator combination is primarily to address systems that are CPU bound (processors being asked to do too much work because two many operators have been combined into one). But it's fair to say that this could also lead to an "out of memory" problem. I'd be looking elsewhere for the out of memory issue, though, particularly at memory-hungry stage types such as Lookup or hash mode Aggregator. Read the entire error message to determine from which stage (operator) the error was thrown.