DSXchange

Hi All
One of my Jobs is throwing the following Warning

APT_ParallelSortMergeOperator,0: Unbalanced input from partition 1: 10000 records buffered

It doesn't say which stage or where the problem is coming..There are many stages in the Job. Is there some way i can figure out which stage partition is throwing this warning

Any inputs will be really helpful

Thanks
Pavan

Add environment variable : APT_DISABLE_COMBINATION and set it value to true. Compile your job and run again.

There's no evidence that operator combination is occurring!

There's no evidence that operator combination is occurring!

This is an alert message from the Collector advising you that data on the inbound partitions are not evenly balanced. This, in turn, suggests that you have needed to use a key-based or range-based partitioning algorithm upstream of the Collector, and it has resulted in a badly skewed distribution of records over available partitions. If that is the case, and you really need that partitioning, then there is nothing you can do about the alert being generated (though you might investigate other collection algorithms). This is a situation where demoting the severity of the logged message might be indicated.

This may help you.

Advanced Runtime in job level or project level
-nosortinsertion

Hi Pavan,
You can try with APT_DUMP_SCORE to analyse the partitioned data.

Hi,

Nice answer for analysing partitions use APT_DUMP_SCORE

but from my side i had one solution, i also faced this problem but in my situation the job gave warnings and as well as it is aborted,
my case is i use one 'sort stage' then 'remove duplicate stage' but the keys in two stages are not same this is the problem why the job is aborting but we are not supposed to change the keys for business so i implemented copy stage in the middle and i used like this,

if ur problem is similar to the above sentences then proceed for the solution

suppose my keys in sort stage are k1 and remove duplicate stage keys are k2,k3
step1:
i added k1 as key in sort stage at properies tab of stage page and in same sort stage at input page and at partitioning tab at input partiiotning type i selected hash and i added k2,k3 as keys
step2:
i added copy stage
step3:
i added k2,k3 as keys in remove duplicate stage

am i confusing you