Hi All
One of my Jobs is throwing the following Warning
APT_ParallelSortMergeOperator,0: Unbalanced input from partition 1: 10000 records buffered
It doesn't say which stage or where the problem is coming..There are many stages in the Job. Is there some way i can figure out which stage partition is throwing this warning
Any inputs will be really helpful
Thanks
Pavan
UnBalance partition
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 133
- Joined: Tue Nov 23, 2004 11:24 pm
- Location: India
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
This is an alert message from the Collector advising you that data on the inbound partitions are not evenly balanced. This, in turn, suggests that you have needed to use a key-based or range-based partitioning algorithm upstream of the Collector, and it has resulted in a badly skewed distribution of records over available partitions. If that is the case, and you really need that partitioning, then there is nothing you can do about the alert being generated (though you might investigate other collection algorithms). This is a situation where demoting the severity of the logged message might be indicated.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Re: UnBalance partition
Hi Pavan,
You can try with APT_DUMP_SCORE to analyse the partitioned data.
You can try with APT_DUMP_SCORE to analyse the partitioned data.
-
- Participant
- Posts: 62
- Joined: Sat Mar 07, 2009 4:59 am
- Location: Chicago
- Contact:
Is Your job containing Remove duplicate stage
Hi,
Nice answer for analysing partitions use APT_DUMP_SCORE
but from my side i had one solution, i also faced this problem but in my situation the job gave warnings and as well as it is aborted,
my case is i use one 'sort stage' then 'remove duplicate stage' but the keys in two stages are not same this is the problem why the job is aborting but we are not supposed to change the keys for business so i implemented copy stage in the middle and i used like this,
if ur problem is similar to the above sentences then proceed for the solution
suppose my keys in sort stage are k1 and remove duplicate stage keys are k2,k3
step1:
i added k1 as key in sort stage at properies tab of stage page and in same sort stage at input page and at partitioning tab at input partiiotning type i selected hash and i added k2,k3 as keys
step2:
i added copy stage
step3:
i added k2,k3 as keys in remove duplicate stage
am i confusing you
Nice answer for analysing partitions use APT_DUMP_SCORE
but from my side i had one solution, i also faced this problem but in my situation the job gave warnings and as well as it is aborted,
my case is i use one 'sort stage' then 'remove duplicate stage' but the keys in two stages are not same this is the problem why the job is aborting but we are not supposed to change the keys for business so i implemented copy stage in the middle and i used like this,
if ur problem is similar to the above sentences then proceed for the solution
suppose my keys in sort stage are k1 and remove duplicate stage keys are k2,k3
step1:
i added k1 as key in sort stage at properies tab of stage page and in same sort stage at input page and at partitioning tab at input partiiotning type i selected hash and i added k2,k3 as keys
step2:
i added copy stage
step3:
i added k2,k3 as keys in remove duplicate stage
am i confusing you
Suresh Reddy
ETL Developer
Research Operations
"its important to know in which direction we are moving rather than where we are"
ETL Developer
Research Operations
"its important to know in which direction we are moving rather than where we are"