Page 1 of 1
Aggregator stage Inconsistent output
Posted: Wed Oct 28, 2009 4:07 pm
by nvuradi
Hi,
The requirement is to aggregate the Amount column based on Amount Type.
I am dealing with the data of around 45 million and currently it has only 2 amount types.
I am getting different results each time i run the job.
I used to Sort method with a preceding Sort stage and Sort on Amount type with Hash partition. I also tried using a single node configuration and sequential node also but i am getting the same result. Is it a bug in dstage?
Any help regarding this will be highly appreciated.
Posted: Wed Oct 28, 2009 4:28 pm
by chulett
Not sure what you mean by 'it only has 2 amount types'. And is your source static? If it is a database table, perhaps you're seeing normal changes over time in the data.
Posted: Wed Oct 28, 2009 4:40 pm
by ray.wurlod
Please confirm:
(a) the names of your grouping columns
(b) the names of your hash key columns
(c) the names of your sort key columns
(d) the Aggregator operation method (hash or sort)
Posted: Wed Oct 28, 2009 10:04 pm
by nvuradi
(a) the names of your grouping columns -Amount Type
(b) the names of your hash key columns - Amount Type
(c) the names of your sort key columns - Amount Type
(d) the Aggregator operation method (hash or sort) - Sort ...
And Amount Type has only 2 values in my data sample.
The data is static, I am reading it from a flat file
Posted: Thu Oct 29, 2009 1:06 am
by ArndW
The answers to the other questions are important as well. Also, how are you detecting that the output is different? Are there just 2 output rows and values are different? If so, what is the output column type?
Posted: Thu Oct 29, 2009 8:37 am
by nvuradi
ArndW wrote:The answers to the other questions are important as well. Also, how are you detecting that the output is different? Are there just 2 output rows and values are different? If so, what is the output col ...
I have given all the required information for the questions...
if i run for the first time i am getting say 10,000 amount but when i run it the next time it changes to 10,001. this one is just an example the real values are huge...
Posted: Thu Oct 29, 2009 8:54 am
by chulett
Floating point.
Posted: Thu Oct 29, 2009 9:27 am
by nvuradi
chulett wrote:Floating point. ...
Any idea on how to resolve this?
Posted: Thu Oct 29, 2009 9:40 am
by chulett
Lots of discussions here on the Aggregator and floating point issues. One such example is
here.
Posted: Thu Oct 29, 2009 10:10 am
by ArndW
nvuradi wrote:I have given all the required information for the questions...
I see that you went back and re-edited an earlier post to add that missing information.
Posted: Tue Nov 03, 2009 10:24 am
by budiman.bun
so how is it?
is it just because of floating point?
what if the output difference is huge in ammount.
let say, 1st run, the job return the total value 10,000
2nd run, the job return 12,000.
any ideas?
i'm using Sort operation method in Aggregator. The source is static, and from Oracle DB using Oracle Stage.
The output is to Oracle Stage.
is this a bug in Oracle Stage?
Thanks