StringToDecimal Conversion Error

nvalia · Post by **nvalia** » Wed Sep 13, 2006 8:22 am

Hi,

I am reading "amount" from a fixed width seq file as a Char e.g (2259 or 36153.35) and then the only transformation I do before writting to an Oracle table is do type conversion using StringToDecimal.

For some strange reason it stores these values as 2259.580000000000 and 36153.349999999999 respectively. (oracle data type is number 24,12). The first one is correct where as the second one is incorrect.

any idea why?

Reg,
NV

keshav0307 · Post by **keshav0307** » Wed Sep 13, 2006 8:32 am

Have you used proper Rtype, like Fix_zeros etc.

nvalia · Post by **nvalia** » Wed Sep 13, 2006 8:44 am

NO..Fix_Zero is for decimaltostring conversion..

I have tried doing this conversion in a stage varibale and then used the SV in the derivation..

ray.wurlod · Post by **ray.wurlod** » Wed Sep 13, 2006 2:30 pm

The conversion is attempting to give you 12 decimal places. In the second case you are getting rounding errors to an acceptable degree of precision (which you can prevent with coding).

The first one looks like a bug; maybe you need to test for the absence of a decimal place holder and add a decimal place and at least one decimal place digit before converting. For example:

Code: Select all

If Index(InLink.TheString, ".", 1) = 0 Then StringToDecimal(InLink.TheString : ".00") Else StringToDecimal(InLink.TheString)