Chinese Characters

[samarvind]

Hi everyone,

I have a requirement where I need to load the Chinese characters from a file into an Oracle table using DataStage server job and it works fine when the NLS is set to UTF-8.

Now, the problem is when I view the data from the sequential file stage or from Oracle table, I am unable to see the Chinese characters as it is getting changed to the following characters "???" but to DataStage it is internally correct. Also, I am facing some problems in the OCI stage where I am unable to give the Chines characters in the SQL WHERE clause.

My OS is Windows XP.

Could you please suggest any ideas and thanks in advance for your response.

Arvind Sampath

[balajisr]

You will not be able to view chinese characters using view data. That's how it works.

[LenGreenwood]

You need to have your client Windows code page set to 936 (for Simplified Chinese) or 950 (for Traditional Chinese) before the DataStage clients will be able to display or enter your Chinese characters correctly.

On XP this is normally done from Control Panel/Regional and Language options/Advanced/Language for non-Unicode programs. This may involve installing Chinese fonts and other such stuff you may not already have installed on Windows. You need to think about whether you want to go this route, as it may affect other s/w you have installed on the client.

If you are running a version of DS prior to 8.1, and your server is installed on an English machine, there may will then be a problem whereby the necessary client/server map are not installed. To get from the client running in code pgae 936 to the DataStage server (which runs in UTF8 internally) requires a specific map to be installed. If DataStage was installed on a server that had a Chinese locale set at the time, the correct maps and settings should have been loaded so that the connection happens automatically. But if not, you may well get an error trying to connect the client to the server once you have switched the code page. (I can give you some instructions on how to fix that if you really want to do it, but it is rather long-wonded).

[samarvind]

You need to have your client Windows code page set to 936 (for Simplified Chinese) or 950 (for Traditional Chinese) before the DataStage clients will be able to display or enter your Chinese characters correctly.

On XP this is normally done from Control Panel/Regional and Language options/Advanced/Language for non-Unicode programs. This may involve installing Chinese fonts and other such stuff you may not already have installed on Windows. You need to think about whether you want to go this route, as it may affect other s/w you have installed on the client.

If you are running a version of DS prior to 8.1, and your server is installed on an English machine, there may will then be a problem whereby the necessary client/server map are not installed. To get from the client running in code pgae 936 to the DataStage server (which runs in UTF8 internally) requires a specific map to be installed. If DataStage was installed on a server that had a Chinese locale set at the time, the correct maps and settings should have been loaded so that the connection happens automatically. But if not, you may well get an error trying to connect the client to the server once you have switched the code page. (I can give you some instructions on how to fix that if you really want to do it, but it is rather long-wonded).

Thanks a lot for your suggestion LenGreenwood. I am interested in all possible options to try out asian languages like Chinese in server edition, so, could you please give your instructions which I can try it out.

[LenGreenwood]

OK, here are some instructions I wrote elsewhere for Japanese clients - can be adapted for Chinese, as explained below. Note that you only need to do any of this if you are connecting from a client whose codepage is set to, say 932, to a server that has been installed in English; on a version of DataStage earlier than 8.1 ========>

Let's assume you have Designer installed on a Japanese Windows system, or one where you have gone into Control Panel/Regional and Language options/Advanced and selected "Japanese" from the drop-down list of "language to match the language version of the non-Unicode programs you want to use" (parts of Designer are non-Unicode). Let's also assume you have DS Server installed on a non-Japanese Unix or Windows system.

The server has a table that links Windows codepages, such as 932 for Japanese clients, to an NLS "client/server map" held in DSEngine. The problem pre-8.1 is that: (a) not all known maps are automatically registered in that table; (b) there is a known bug that means the default map which gets assigned may be unusable. Only a server installed on a Japanese system will have the map for 932 registered; if you connect from a 932 client to such a system what you may see is a message along the lines of "Invalid Universe mark characters have been sent from the server" every time you try and login. That's because the invalid default map is, unfortunately, being picked up.

This particular problem has been fixed in 8.1. It still does not help the fact that the server does not have a mapping for the 932 code page if it is a non-Japanese install - so attempts to transfer Japanese characters between client and server result in lots of "?"s being shown instead. Note that the same problems occur for Chinese (936, 950) and Korean (949) clients, and possibly also for non-Latin1 clients such as Polish (1250), Russian (1251), Greek (1252), Turkish (1254), Hebrew (1255), or Arabic (1256).

At 8.1 is DS install has changed so that all code page maps are built and registered on all servers. Prior to that, you can manually register the required maps, as follows. Note that where the examples have 932 below, you can substitute another code page number from the list above.

In a Unix command session :

Code: Select all

cd $DSHOME
bin/dssh

...or in a Windows command session:

Code: Select all

cd C:\IBM\InformationServer\Server\DSEngine
bin\dssh

The appropriate commands will take you into the DSEngine command environment, where the prompt is ">". From then onwards the instructions are independent of O/S type. At the ">" prompt, enter:

Code: Select all

COPY FROM NLS.CLIENT.MAPS WIN:1252,WIN:932

Should get a message "1 record copied". Then type:

Code: Select all

ED NLS.CLIENT.MAPS WIN:932
2
R/1252/932/
FILE

Should get a message "WIN:932" filed in file "NLS.CLIENT.MAPS" This has created a record called WIN:932 that links code page 932 to the map named MS932-CS, which is probably not built on your server.

So, then type:

Code: Select all

NLS.BUILD MAP MS932-CS NO.PAGE

Should eventually see a message about "Binary file is: NLS.MAP.INSTALL/MS932-CS" Then type:

Code: Select all

Q

to get out of DSEngine shell and back to the O/S command shell.

You have to stop & restart DSEngine to get this new map loaded. But once that is done, a Japanese client should be able to attach to the server.

[samarvind]

Thanks a ton for your time and consideration for explaining me in detail. I will implement it and let you know in case of any queries.

[mandyli]

Hi

We also faced same kind of issueswith chinese char.
What we did in the source file format is Unicode format so I can able to see view the data from Datastage view data.

For target we used UTF8 maps. Any way in the server job may be you can't see the chinese data. But you can able to view the data from SQLdeveloper or TOAD with chineses char.

So now our jobs loading data from china source file into ORACLE table.

Don't change anything datastage side try to NLS setup in database side.


Thanks
Man

[mandyli]

Hi

We also faced same kind of issueswith chinese char.
What we did in the source file format is Unicode format so I can able to see view the data from Datastage view data.

For target we used UTF8 maps. Any way in the server job may be you can't see the chinese data. But you can able to view the data from SQLdeveloper or TOAD with chineses char.

So now our jobs loading data from china source file into ORACLE table.

Don't change anything datastage side try to NLS setup in database side.


Thanks
Man

[ragunathan13]

Hi LenGreenwood,

We are also having the same requirement of handling chinese characters.We have changed our system settings to incorporate chinese characters.We are able to get the chinese characters as it is in the output file.But we are not able to see them when we choose view data option in datastage.

We tried executing the script given by you in our server dssh.But we faced error after executing the below line

COPY FROM NLS.CLIENT.MAPS WIN:1252,WIN:932

saying NLS.CLIENT.MAPS as invalid.

Could you please give us more information on the above?

[ray.wurlod]

Sounds like your DataStage server was installed without NLS, or that you were not in the UV account when you executed the COPY command.

[chulett]

We have changed our system settings to incorporate chinese characters.

Also, what exactly does this mean? You changed what?

[ragunathan13]

We have changed our system settings to incorporate chinese characters.

By this I meant I have enabled/checked the option "Install East Asian Languages" in the Date, Time, Language, and Regional Options dialog box inside control panel.

Please let us know if any other settings have to be changed.Both in system as well as on datastage side.

Thanks in advance.....

[ragunathan13]

Hi Ray...

I checked the UV.ACCOUNTS table in the command box of the DS Administrator.The UVACCOUNT is set to the path "c:\#directory path#\engine" whereas the dssh.exe application is available in "c:\#directory path#\engine\bin" path.

I am doubting that this might be the reason for me trying execute the command without being in the uvaccount.(Based on your comment "Sounds like your DataStage server was installed without NLS, or that you were not in the UV account when you executed the COPY command.")
If I am correct Could you please direct me if there is way to move the dssh application to the UVaccount path(c:\#directory path#\engine)?

Can you also tell me where NLS.CLIENT.MAPS file will be available?

I have also checked the uvconfig file which is under c:\#directory path#\engine and found that nls settings is enabled and the value is given as 1.
The chinese character set NLS Map MS936 is also being supported according to the config files.But still we are not able to view data in chinese.

Please let us know if there is any other way/any other setting that has to be enable.

Thanks....

[LenGreenwood]

Going back to the statement earlier:

We tried executing the script given by you in our server dssh.But we faced error after executing the below line

COPY FROM NLS.CLIENT.MAPS WIN:1252,WIN:932

saying NLS.CLIENT.MAPS as invalid.

...we need to know the exact error message you got after executing the COPY command.

NLS.CLIENT.MAPS is an entry in the VOC file of the account that gets started after you execute bin\dssh when attached to the DSEngine home directory. At least it is if you installed the server with NLS on (which it appears from the uvconfig file that you did). I assume you were attached to the correct directory (on a 7.5 Windows system it is often C:\Ascential\DataStage\Engine) before executing bin\dssh.exe - presumably you must have been in that directory otherwise bin\dssh.exe would not have been the correct relative path.

So, what was the actual error message? You did enter the whole command in upper-case, didn't you - as that is very important too.

[ragunathan13]

Hi,

The error that we got is as below:
Unable to opn "NLS.CLIENT.MAPS" file.

Our UVACCOUNT path is set to c:\Datastage\engine.In this folder we have an VOC file which has NLS.CLIENT.MAPS defined in it.

But the dssh through which we executed is available in c:\Datastage\engine\bin(not the UVACCOUNT path) and this folder also contains a VOC file which does not contain any NLS.CLIENT.MAPS.

We checked the availability of NLS.CLIENT.MAPS by using below query in dssh:
SELECT * from VOC where NAME like '%NLS%'

Please tell us if the VOC files in the two different paths mentioned above are the same or different?Is there anyway to execute the commands mentioned by you by being in the UVACCOUNT path without having dssh.exe in that path?