Dealing with Japanese characters

yaminids · Post by **yaminids** » Thu Dec 16, 2010 2:44 am

Friends,

Currently I'm working on designing a job that transfers data between 2 Oracle databases. Both the databases are set to UTF8 NLS character set. I could clearly see the Japanese characters (す下る)when I login to the source db.

The problem is that the Japanese characters are being replaced by "????" in the destination db when I execute the DataStage job that transfers the data.

The design of the job is very simple
ODBC ==>Transformer==>ODBC

Am I doing something wrong

Thanks in advance
Yamini

sultan@cts · Post by **sultan@cts** » Thu Dec 16, 2010 3:17 am

Is UTF 8 characte set is enabled at job level?
Try to remove the ustring (entended property of meta data).

yaminids · Post by **yaminids** » Thu Dec 16, 2010 2:09 pm

UTF8 is set as the default NLS at the project level. Also, can you please tell me how I can remove the ustring?

Thanks
Yamini

ray.wurlod · Post by **ray.wurlod** » Thu Dec 16, 2010 3:25 pm

If you remove the ustring you definitely will not be able to process Japanese characters. However, if you wish to do so, click on it - it's a drop down list in which the other option is empty. Choose that option.

yaminids · Post by **yaminids** » Thu Dec 16, 2010 4:33 pm

Ray,

We do want to process the Japanese characters. Can you please let me know the correct way to process these characters while populating data in the destination table

Thanks
Yamini

ray.wurlod · Post by **ray.wurlod** » Thu Dec 16, 2010 5:38 pm

You need to obtain a definitive statement concerning how these characters are encoded in the target table (for example SHIFT-JIS, UTF-8), then ensure that all DataStage mappings and all database mappings are compatible. For example if your target is Oracle, make sure that the NLS_LANG environment variable is correctly set (check the "environment variables" entry in the job log).

yaminids · Post by **yaminids** » Thu Dec 16, 2010 6:30 pm

Ray,

Data in the source is encoded in UTF-8 format. I have changed the NLS at the project level to UTF8. Also, added an environment variable NLS_LANG with "AMERICAN_AMERICA.AL32UTF8" as the value.

I'm still not able to process the Japanese characters. Can you please point me to the right direction

Thanks
Yamini

yaminids · Post by **yaminids** » Mon Jan 10, 2011 2:16 am

I finally figured out the problem. All the columns containing Japanese characters were listed as "VARCHAR". The job started to process the characters as soon as I changed the definition of the columns to "NVARCHAR"

ray.wurlod · Post by **ray.wurlod** » Mon Jan 10, 2011 2:17 am

Well done. I guess we'd assumed you'd used NVarChar.

DSXchange

Dealing with Japanese characters

Dealing with Japanese characters

Re: Dealing with Japanese characters

Re: Dealing with Japanese characters