AS400/flat file sources, NLS, UTF8 & a new NLS-related e

ray.wurlod · Post by **ray.wurlod** » Wed Apr 09, 2008 2:26 am

Can you try it using the ISO8859-1 map?

It is the first character in your data that is giving the problem - the "?" is an indicator of "unmappable character". In hex it is 0xC9. The next two characters (0x4B and 0x4F) are being mapped as "K" and "O" respectively, which is correct.

0xC9 is 201 in Decimal - see if you can find a listing of "ASCII" codes that goes all the way up to 255 to determine what this character (Char(201)) should be.

Q1LPRD is a space filled field.

wwalker · Post by **wwalker** » Wed Apr 09, 2008 3:43 am

In fact, I already did that, and testing seems to indicate it is working.

You better be careful, Ray. People might start to think you know your stuff.

Thanks

Wade

wwalker · Post by **wwalker** » Wed Apr 09, 2008 6:11 am

Additional insight as I continue to explore..

I find that the NLS code "MNEMONICS" seems to work for both...but I am reluctant to use this because :

a) I am not familiar with it (never heard of it, actually)
b) the standard at my client for QA and higher is ISO8859-1

ray.wurlod · Post by **ray.wurlod** » Wed Apr 09, 2008 6:52 am

MNEMONICS is a pseudo map that recognizes "ASCII representations" such as <N?>.

You may find MNEMONICS-1 even better, as it recognizes "ISO885901" representations such as <o>> (LATIN SMALL LETTER O WITH CIRCUMFLEX).

wwalker · Post by **wwalker** » Wed Apr 09, 2008 7:54 am

Ray,

Interesting observation about MNEMONIC-1...but how would one get their flippers on that character set if they do not have it..

Is it just a matter of installing it onto the NLS server?

..and if I understand fully, NMEMONIC-1 would be considered a superset of ISO8859-1 and UTF8 ? By using this I should then resolve my issue of character sets for the French Market?

...and, in the interest of exposing my NLS ignorance further, isn't the UNICODE character set the one to use to avoid this issue, when using the DS processes in multiple markets, such as in my current project? or, what would be the best/most practical choice for character set, since we are already using NLS?

I seem to recall using UNICODE (or was it UTF8??) when I moved a process built for the Italian market to the Japanese market for a different client....

Thanks for your help.

W

ray.wurlod · Post by **ray.wurlod** » Wed Apr 09, 2008 3:04 pm

Ultimately - and particularly in Japan where there are many different encodings of characters - you need to know how the character set is encoded. ASCII and ISO8859-x are "single byte character sets" so it should be a simple matter of reviewing which map(s) handle all the character in your data.

The maps convert the external data characters into an idiosyncratic "UV-UTF8" encoding for use within DataStage, and back again as they leave DataStage.

(Note that parallel jobs use a newer, different mechanism.)