BigInt through Aggregator

[Kirtikumar]

Hi,

I was trying to pass Salary field having datatype BigInt through aggregator. For the rows where Salary field has a number with length > 10 numeric characters , it was generating one magic number.

Design of job is as follows:

Code: Select all

Seq.File1 -----> Transformer -----> Aggregator --------> Seq.File2

Transformer concatenates 2 columns FName and LName as Name.

The data in Seq.File1 was as follows.

Code: Select all

FName	LName	Salary
----------------------------------------
Aa	   Bb	   768689900990000
Bb	   cC	   990099000
Vv	   Xx	   9900990000
Hh	   Ss	   99009900001

The output generated in the Seq.File2 is as follows:

Code: Select all

Name		 Salary
----------------------------------------
Aa,Bb	   2147483647
Bb,cC	   990099000
Vv,Xx	   9900990000
Hh,Ss	   2147483647

One interesting thing to note here, the magic no. is genareted only if Salary is having more than 10 characters. I have thouroughly debugged the job, and reached the conclusion that this no. gets modified in aggregator stage.

Has anyone faced this problem earlier? What is the solution to this problem?

[Sainath.Srinivasan]

The 'magic number' is the max value possible in any integer type (2 power n - 1). You may have to increase the size of the storage. i.e. move to numeric, float, double etc.

[lax]

This is the problem Of EXACTNUMERICS.


See the prev post for more information

http://dsxchange.com/viewtopic.php?t=91108

[Sainath.Srinivasan]

If you notice, the length of the numeric values are 15 digit length only.

The resolution is to use Char instead of BigInt.

[Kirtikumar]

The problem of BigInt occurs only if I put the aggregator. If I remove aggregator stage the problem disappears. And this magic numbers are generated only if any number is having more than 10 digits. And one more thing is, the problem of EXACTNUMERIC is not affecting this, as numbers with 11-14 digits are also facing the same problem.

I tried using Numeric 19,0 datatype for aggregator stage only. Its working and generating proper results. But why this aggregator stage is facing this problem? Is it a bug?

If you notice, the length of the numeric values are 15 digit length only.

The resolution is to use Char instead of BigInt.

[Kirtikumar]

Thanks Sainath for your invaluable inputs.

The work around you have suggested is working perfectly and we have a done testing for all possible combinations.

Reason why am I posting something again on this topic is, this problem is with the stage and not with EXACTNUMERIC . We have done fix on EXACTNUMERIC by setting it to 19 and restarted the DS Server. After fixing also, this magic number is generated. The only way to solve the issue is the one suggested by you.

One more thing we have found is the similar problem exists for sort stage and the workaround it is the same i.e. to define the column as char 20. This problem also gets solved if we use Numeric 20.

[Sainath.Srinivasan]

Double cheers for posting the result. It will be useful for other people searching the forum for similar issue.