~~~Warning in job~~~
Moderators: chulett, rschirm, roy
~~~Warning in job~~~
Hi
My job runs perfectly well with a smaller volumes in the order of 2 or 3 millions. But once when the volume shoots up, my job throws the following warning continuously but job runs to completion without aborting.
APT_ParallelSortMergeOperator,2: Unbalanced input from partition 2: 10000 records buffered
How can I avoid this warning ?? Any Ideas welcome !!
Vignesh.
My job runs perfectly well with a smaller volumes in the order of 2 or 3 millions. But once when the volume shoots up, my job throws the following warning continuously but job runs to completion without aborting.
APT_ParallelSortMergeOperator,2: Unbalanced input from partition 2: 10000 records buffered
How can I avoid this warning ?? Any Ideas welcome !!
Vignesh.
Well, go to the Search page here and enter "Unbalanced input from partition" and check the Exact match option. You'll find it's been covered before.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Hi Vignesh,
This is due to wrong partitioning. You have done the HASH partitioning twice. If you have done HASH partitioning once use the SAME partitioning for the remaining stages until you do the ROUND ROBIN partitioning. Instead of SAME partitioning you have again done the HASH partitioning and that is the reason you get this warning.
HTH
--Rich
Pride comes before a fall
Humility comes before honour
This is due to wrong partitioning. You have done the HASH partitioning twice. If you have done HASH partitioning once use the SAME partitioning for the remaining stages until you do the ROUND ROBIN partitioning. Instead of SAME partitioning you have again done the HASH partitioning and that is the reason you get this warning.
HTH
--Rich
Pride comes before a fall
Humility comes before honour
Can this error happen even if you're using HASH Partitioning but twice but on two different Key values ??
richdhan wrote:Hi Vignesh,
This is due to wrong partitioning. You have done the HASH partitioning twice. If you have done HASH partitioning once use the SAME partitioning for the remaining stages until you do the ROUND ROBIN partitioning. Instead of SAME partitioning you have again done the HASH partitioning and that is the reason you get this warning.
HTH
--Rich
Pride comes before a fall
Humility comes before honour
--------
GaNoU
--------
GaNoU
--------
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Woow... so that mean that you can't use the same Dataset if you have to perform Join/Lookup/Merge... operations on different key values ?
I mean, each Time I had to do that, I used to "repartition" my "Reference" Dataset by hashing it on another key depending on the data I wanted to join with![Surprised :o](./images/smilies/icon_surprised.gif)
I mean, each Time I had to do that, I used to "repartition" my "Reference" Dataset by hashing it on another key depending on the data I wanted to join with
![Surprised :o](./images/smilies/icon_surprised.gif)
ray.wurlod wrote:Yes, of course, particularly if you are depending on the partitioning being the same, for example to perform a lookup.
--------
GaNoU
--------
GaNoU
--------
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
No, it doesn't mean that at all. You can use the same Data Set, but you do have to ensure that the lookup will find all the keys it needs to, and that means that the stream and reference inputs must be partitioned identically (and sorted identically for Join and Merge stages) on the keys being used for the lookup. Or, if it's a small enough Data Set, use Entire partitioning.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
OK... I thought I had used PX 2 years without understanding anything... Saved !!
So let's get back to the subject :
Far earlier, Rich said that the Error "Unbalanced Input From partition 0..." is due to a Hash partitioning made Twice.
Is it the real problem then ?
According to my usage, and what I can read, you can re-Hash whenever you want it, even if it is not optimum (Same partition should be used if you are using the same Hash Keys).
In my case, I have a stream input, coming from a Flat File. I'm using the partition method to Hash on the same Key as my reference Dataset (the reference input is using same partition as it has been partitionned earlier).
Every other link is same... except for a writing in a dataset where I use Round Robin.
And I'm getting the famous error "Unbalanced Input From Partition 0...".
Any Idea ??
![Wink :wink:](./images/smilies/icon_wink.gif)
So let's get back to the subject :
![Twisted Evil :twisted:](./images/smilies/icon_twisted.gif)
Far earlier, Rich said that the Error "Unbalanced Input From partition 0..." is due to a Hash partitioning made Twice.
Is it the real problem then ?
![Question :?:](./images/smilies/icon_question.gif)
According to my usage, and what I can read, you can re-Hash whenever you want it, even if it is not optimum (Same partition should be used if you are using the same Hash Keys).
In my case, I have a stream input, coming from a Flat File. I'm using the partition method to Hash on the same Key as my reference Dataset (the reference input is using same partition as it has been partitionned earlier).
Every other link is same... except for a writing in a dataset where I use Round Robin.
And I'm getting the famous error "Unbalanced Input From Partition 0...".
Any Idea ??
![Question :?:](./images/smilies/icon_question.gif)
ray.wurlod wrote:No, it doesn't mean that at all. You can use the same Data Set, but you do have to ensure that the lookup will find all the keys it needs to, and that means that the stream and reference inputs must be partitioned identically (and sorted identically for Join and Merge stages) on the keys being used for the lookup. Or, if it's a small enough Data Set, use Entire partitioning.
--------
GaNoU
--------
GaNoU
--------
Okay, Just solved the problem.
It seems that PX version 7.1, while using the Join Stage, automatically sort your input data.
I'm used to partition my data before the Join stage and check the Sort box as well... that's why I get the Warning !!
All I had to do to solve it, was to uncheck the sort box.
Maybe this will help some of you.
It seems that PX version 7.1, while using the Join Stage, automatically sort your input data.
I'm used to partition my data before the Join stage and check the Sort box as well... that's why I get the Warning !!
All I had to do to solve it, was to uncheck the sort box.
Maybe this will help some of you.
ganive wrote:OK... I thought I had used PX 2 years without understanding anything... Saved !!![]()
So let's get back to the subject :![]()
Far earlier, Rich said that the Error "Unbalanced Input From partition 0..." is due to a Hash partitioning made Twice.
Is it the real problem then ?![]()
According to my usage, and what I can read, you can re-Hash whenever you want it, even if it is not optimum (Same partition should be used if you are using the same Hash Keys).
In my case, I have a stream input, coming from a Flat File. I'm using the partition method to Hash on the same Key as my reference Dataset (the reference input is using same partition as it has been partitionned earlier).
Every other link is same... except for a writing in a dataset where I use Round Robin.
And I'm getting the famous error "Unbalanced Input From Partition 0...".
Any Idea ??![]()
ray.wurlod wrote:No, it doesn't mean that at all. You can use the same Data Set, but you do have to ensure that the lookup will find all the keys it needs to, and that means that the stream and reference inputs must be partitioned identically (and sorted identically for Join and Merge stages) on the keys being used for the lookup. Or, if it's a small enough Data Set, use Entire partitioning.
--------
GaNoU
--------
GaNoU
--------
Hi,ganive wrote:Okay, Just solved the problem.
It seems that PX version 7.1, while using the Join Stage, automatically sort your input data.
I'm used to partition my data before the Join stage and check the Sort box as well... that's why I get the Warning !!
All I had to do to solve it, was to uncheck the sort box.
Maybe this will help some of you.
ganive wrote:OK... I thought I had used PX 2 years without understanding anything... Saved !!![]()
So let's get back to the subject :![]()
Far earlier, Rich said that the Error "Unbalanced Input From partition 0..." is due to a Hash partitioning made Twice.
Is it the real problem then ?![]()
According to my usage, and what I can read, you can re-Hash whenever you want it, even if it is not optimum (Same partition should be used if you are using the same Hash Keys).
In my case, I have a stream input, coming from a Flat File. I'm using the partition method to Hash on the same Key as my reference Dataset (the reference input is using same partition as it has been partitionned earlier).
Every other link is same... except for a writing in a dataset where I use Round Robin.
And I'm getting the famous error "Unbalanced Input From Partition 0...".
Any Idea ??![]()
ray.wurlod wrote:No, it doesn't mean that at all. You can use the same Data Set, but you do have to ensure that the lookup will find all the keys it needs to, and that means that the stream and reference inputs must be partitioned identically (and sorted identically for Join and Merge stages) on the keys being used for the lookup. Or, if it's a small enough Data Set, use Entire partitioning.
As far as i know, the join stage wont sort automaticlly unless AUTO partition is specified otherwise.
-Kumar