Dealing with special characters

[chulett]

P.S The default NLS is set to ISO8859-1

Whose default NLS is set to that? The DataStage client or your database? I'm suspecting the former.

It would be best to let us know details like which database you are using and what its NLS setting is, also if your DataStage installation is NLS enabled or not.

[yaminids]

Hi Craig,

The DataStages' NLS is defaulted to ISO8859-1 and the installation is NLS enabled.
We are using Oracle 9i database and I am not sure about the default character set of the db (I think it would be UTF8)

Thanks
Yamini

[yaminids]

Craig,

Can you please answer the following questions?

1) Does DataStage store characters with special characters differently than the database
2) Does DataStage reduce the field length whenever it finds a special character in the data?

Thanks lot in advance
Yamini

[ArndW]

What about LEN(TRIM({string})) to see if the "missing" character is part of the text or the buffered data. I suspect it isn't the text, but your padding.

[yaminids]

Hi ArndW,

You are right. When I calculated the length, the field is showing the correct length as in source.
Can you please through some light on the issue of the padding of the field?

Thanks
Yamini

[ArndW]

You haven't stated where your data is coming from. Is it a text file or a database? DataStage will not just strip one character from this string; so I would use a hex display editor if the source is a flat file or the substring functions in your query language directly to make 100% of the field length first.

[ray.wurlod]

Can you do the same measurements using ByteLen() rather than Len()?

Comparing the two (ByteLen and Len) will allow you to detect rows where there an apparent problem. Resolving it won't be quite so easy.

Awaiting your ByteLen() values.