Regarding Loading chinese characters

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 7:09 am

Hi,
I am receiving chinese characters in the feed file. and i am loading this file in DB2 table. In DS i am using UTF 8 in Job properties in NLS tab as well as in stages. Data is getting loaded into the table 1. Next job is loading data from table 1 into Dataset. then from Dataset to Table2, This is also loading correct. Next job is creating txt file. here i am getting warning APT_CombinedOperatorController,0: I am using UTF 8 in Job properties as well as in stage level.

No records are getting rejected all are getting loaded into the txt file.

Can any body suggest me how to proceed to remove the warnings.

ArndW · Post by **ArndW** » Wed Sep 30, 2009 7:12 am

Please post your complete and unedited warning message.

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 7:14 am

edOperatorController,0: Invalid character(s) ([xE6]) found converting string (code point(s): 2009-08-12 8.25 [xE6][xAD][xA3][xE5][x9C][xA8][xE7][xAD][x89][xE5][xAE][xA2][xE6][x88][xB7][xE7][x9A][x84][xE4][xB8][xA4][xE4][xB8][xAA][xE5][x85][xAC][xE5][x8F][xB8][xE7][x9A][x84][xE6][x9D][x90][xE6][x96][x99][xE3][x80][x82]8.24 Cao Yan[xE5][xB7][xB2][xE5][x9C][xA8][xE7][xB3][xBB]...) from codepage UTF-8 to Unicode, substituting.

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 7:15 am

APT_CombinedOperatorController,0: Invalid character(s) ([xE6]) found converting string (code point(s): 2009-08-12 8.25 [xE6][xAD][xA3][xE5][x9C][xA8][xE7][xAD][x89][xE5][xAE][xA2][xE6][x88][xB7][xE7][x9A][x84][xE4][xB8][xA4][xE4][xB8][xAA][xE5][x85][xAC][xE5][x8F][xB8][xE7][x9A][x84][xE6][x9D][x90][xE6][x96][x99][xE3][x80][x82]8.24 Cao Yan[xE5][xB7][xB2][xE5][x9C][xA8][xE7][xB3][xBB]...) from codepage UTF-8 to Unicode, substituting.

ArndW · Post by **ArndW** » Wed Sep 30, 2009 7:17 am

Now add "APT_DISABLE_COMBINATION" to your job and set the value to "true" for one run and the warning will tell you the actual stage generating the message, please post that.

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 7:22 am

Db2Api_PRELOAD_PLD_COLLECTIONS_TGT,0: Invalid character(s) ([xE6]) found converting string (code point(s): 2009-08-12 8.25 [xE6][xAD][xA3][xE5][x9C][xA8][xE7][xAD][x89][xE5][xAE][xA2][xE6][x88][xB7][xE7][x9A][x84][xE4][xB8][xA4][xE4][xB8][xAA][xE5][x85][xAC][xE5][x8F][xB8][xE7][x9A][x84][xE6][x9D][x90][xE6][x96][x99][xE3][x80][x82]8.24 Cao Yan[xE5][xB7][xB2][xE5][x9C][xA8][xE7][xB3][xBB]...) from codepage UTF-8 to Unicode, substituting.

Db2Api_PRELOAD_PLD_COLLECTIONS_TGT this is my source stage

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 7:38 am

Database is set up to UTF-8. and in other two jobs i am loading the data into DB2 table as well as reading the data.

ArndW · Post by **ArndW** » Wed Sep 30, 2009 7:45 am

In the extended attribute for the string do you have "unicode" set?

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 7:47 am

That is set to unicode to extended column in stage

ArndW · Post by **ArndW** » Wed Sep 30, 2009 8:01 am

Ok, what about looking at it differently. What should the string be after the date? It looks like it is garbled, with single-byte extended Latin characters instead of initial bytes of multibyte Chinese ones.

manojbh31 · Post by **manojbh31** » Wed Sep 30, 2009 11:22 pm

Please find the data which i am loading for the above warning.

2009-08-12 8.25 正在等客户的两个公司的材料。8.24 Cao Yan已在系统中void掉发票了。8.20 客户说已退回发票，正在等OO关于合同转让的邮件。8.19 OO要给客户做合同转让，估计要下周才能有所进展，不�

Data is getting loaded into seq file. But i getting above warning. and datatype in table VARGRAPHIC and Datastage i am reading it as VARCHAR.

ArndW · Post by **ArndW** » Thu Oct 01, 2009 1:16 am

What if you were to change your job and all stages in it to UTF-8 to avoid convertions altogether, does the warning still appear?

manojbh31 · Post by **manojbh31** » Tue Oct 06, 2009 3:56 am

Hi,

This got resolved, i was substr function in the source SQL query. I came to know that we cannot use any functions for different language in SQL statement if it the job is used as UTF