Aggregator stage Inconsistent output

nvuradi · Post by **nvuradi** » Wed Oct 28, 2009 4:07 pm

Hi,
The requirement is to aggregate the Amount column based on Amount Type.

I am dealing with the data of around 45 million and currently it has only 2 amount types.

I am getting different results each time i run the job.

I used to Sort method with a preceding Sort stage and Sort on Amount type with Hash partition. I also tried using a single node configuration and sequential node also but i am getting the same result. Is it a bug in dstage?

Any help regarding this will be highly appreciated.

chulett · Post by **chulett** » Wed Oct 28, 2009 4:28 pm

Not sure what you mean by 'it only has 2 amount types'. And is your source static? If it is a database table, perhaps you're seeing normal changes over time in the data.

ray.wurlod · Post by **ray.wurlod** » Wed Oct 28, 2009 4:40 pm

Please confirm:
(a) the names of your grouping columns
(b) the names of your hash key columns
(c) the names of your sort key columns
(d) the Aggregator operation method (hash or sort)

nvuradi · Post by **nvuradi** » Wed Oct 28, 2009 10:04 pm

(a) the names of your grouping columns -Amount Type
(b) the names of your hash key columns - Amount Type
(c) the names of your sort key columns - Amount Type
(d) the Aggregator operation method (hash or sort) - Sort ...

And Amount Type has only 2 values in my data sample.

The data is static, I am reading it from a flat file

ArndW · Post by **ArndW** » Thu Oct 29, 2009 1:06 am

The answers to the other questions are important as well. Also, how are you detecting that the output is different? Are there just 2 output rows and values are different? If so, what is the output column type?

nvuradi · Post by **nvuradi** » Thu Oct 29, 2009 8:37 am

ArndW wrote:The answers to the other questions are important as well. Also, how are you detecting that the output is different? Are there just 2 output rows and values are different? If so, what is the output col ...

I have given all the required information for the questions...

if i run for the first time i am getting say 10,000 amount but when i run it the next time it changes to 10,001. this one is just an example the real values are huge...

chulett · Post by **chulett** » Thu Oct 29, 2009 8:54 am

Floating point.

nvuradi · Post by **nvuradi** » Thu Oct 29, 2009 9:27 am

chulett wrote:Floating point. ...

Any idea on how to resolve this?

chulett · Post by **chulett** » Thu Oct 29, 2009 9:40 am

Lots of discussions here on the Aggregator and floating point issues. One such example is here.

ArndW · Post by **ArndW** » Thu Oct 29, 2009 10:10 am

nvuradi wrote:I have given all the required information for the questions...

I see that you went back and re-edited an earlier post to add that missing information.

budiman.bun · Post by **budiman.bun** » Tue Nov 03, 2009 10:24 am

so how is it?
is it just because of floating point?
what if the output difference is huge in ammount.
let say, 1st run, the job return the total value 10,000
2nd run, the job return 12,000.

any ideas?
i'm using Sort operation method in Aggregator. The source is static, and from Oracle DB using Oracle Stage.
The output is to Oracle Stage.
is this a bug in Oracle Stage?

Thanks