Page 1 of 1
Wrong Results in Aggregator
Posted: Tue Jul 08, 2008 7:59 am
by ppavani_km
Hi,
In my input data i have amount coloumn as data type Decimal(15,2).
the amount may contain either +'ve or -'ve values.
When I am trying to sum up the amount coloumn using aggregator first it raised up warning saying precesion exceeding.
so that i have changed the data type from decimal to double before aggregator.then it is not showing up any warnings.
but on each time i run the job the aggregated amount coloumn is showing wrong results.
I even used Modify stage also before aggregator but still it is showing wrong resuts.
test data.
ID AMOUNT(15,2)
10 150.00
10 -100.00
10 -50.00
20 140
30 45
30 -45
so after aggregation it should come like this without any warnings.
ID AMOUNT(15,2)
10 0
20 140
30 0
Please help.
Posted: Tue Jul 08, 2008 9:01 am
by OddJob
What values did you actually receive?
RE:Wrong Results in Aggregator
Posted: Tue Jul 08, 2008 9:14 am
by ppavani_km
For each run i am getting different values like as follows
ID AMOUNT(15,2)
10 150
20 140
30 0
or
ID AMOUNT(15,2)
10 -150
20 140
30 -45
OddJob wrote:What values did you actually receive?
Posted: Tue Jul 08, 2008 9:56 am
by OddJob
Can you detail the sort and partitioning you're applying?
Posted: Tue Jul 08, 2008 10:41 am
by ppavani_km
I just took the auto option for ID coloumn.
MY problem is with the amount field lenth and data type.
OddJob wrote:Can you detail the sort and partitioning you're applying?
Posted: Tue Jul 08, 2008 6:21 pm
by John Smith
ppavani_km wrote:I just took the auto option for ID coloumn.
MY problem is with the amount field lenth and data type.
OddJob wrote:Can you detail the sort and partitioning you're applying?
Try partitioning your data by the ID column.
Posted: Tue Jul 08, 2008 6:31 pm
by keshav0307
use Hash partition and sorting on the the aggregate column (ID in this case)
Posted: Tue Jul 08, 2008 6:50 pm
by yatin.mittal
You should have the data in sorted order on the primary set (on which you are aggregating) before applying the aggregate function. If its not in sorted form, then it can give wrong results.
Posted: Wed Jul 09, 2008 7:16 am
by ppavani_km
I tried doing Hash partitioning on ID coloumn still showing wrong results.
Posted: Wed Jul 09, 2008 7:12 pm
by keshav0307
can you try running this job in single node and see the result.
Posted: Wed Jul 09, 2008 10:33 pm
by DS_FocusGroup
use a sort stage before the aggregator stage and hash partition on the keys which you are using for aggregation in the same order. also sort ascending in the same order. should work!
Posted: Wed Jul 09, 2008 10:34 pm
by DS_FocusGroup
use a sort stage before the aggregator stage and hash partition on the keys which you are using for aggregation in the same order. also sort ascending in the same order. should work!
Posted: Wed Jul 09, 2008 10:35 pm
by DS_FocusGroup
use a sort stage before the aggregator stage and hash partition on the keys which you are using for aggregation in the same order. also sort ascending in the same order. should work!