Hi all.
We are presently importing chinese and japanese data in large scale.
None of the existing codepage mappings seem to fullfill our needs, and we can expect new characters to pop up from time to time.
Basically we need to export one of the japanese/chinese codepages and edit it in an external editor
My plan is then to import and register it again under new name (first time)
For add-ons of new characters we would just over write the existing.
I HAVE looked in the NLS manual , but it only seems cover a manual insert character by character using the NLS.ADMIN.
I have been there as well, and it feels "less than impressive".
On DS Server in the Datastage\Engine\nls\maps\MAP.TABLES library a number of files are present.
Could you just copy any one of these files and the insert linie by line the new mappings??
How do you eventually register a new mapping, if done in this way??
Any experiences and/or advices would be welcome.
Kind Rgds. Jan
NLS export existing map for edit
Moderators: chulett, rschirm, roy
jinm,
I've made my own Japanese maps and manually added all the gaiji characters before, and the effort was all manual within DataStage/Universe. The tools might be unwieldy and archaic but they do work quite well - but only once you've gone through the painful process of using them.
I'm curious as to how the codepages don't match up with your incoming data - particularly the Japanese is highly standardized with the exception of those company-specific Gaiji. Adding entries outside of this codeblock should be rare and I haven't seen that done anywhere yet.
I suppose that a completely new data source (i.e. a new HW platform with it's own encoding) would entail a lot of work. The NLS information is stored in hashed files, which could even be accessed via DataStage jobs, so you have a builtin method of extracting mapping information into a format you like and could conceivably write back changed data as well.
I've made my own Japanese maps and manually added all the gaiji characters before, and the effort was all manual within DataStage/Universe. The tools might be unwieldy and archaic but they do work quite well - but only once you've gone through the painful process of using them.
I'm curious as to how the codepages don't match up with your incoming data - particularly the Japanese is highly standardized with the exception of those company-specific Gaiji. Adding entries outside of this codeblock should be rare and I haven't seen that done anywhere yet.
I suppose that a completely new data source (i.e. a new HW platform with it's own encoding) would entail a lot of work. The NLS information is stored in hashed files, which could even be accessed via DataStage jobs, so you have a builtin method of extracting mapping information into a format you like and could conceivably write back changed data as well.
</a>Hi ArndW
Thank you for a good reply - not the one I hoped to see, but still
)
Well it is more like we are loading into a "non-unicode" environment
We don't want to get mapping errors, and a lot of the characters must be converted to pre-defined characters.
The nice thing would be to take it out and map it in a small programme "seach/replace style" .
In this way you could keep it for further editing every time we come across a new character.
As for now we edit a basic routine to be used within the DS Jobs that does the conversion, but in the long run this is not a good solution considering that the jobs that use the routine must be recompiled every time we change the routine code.
Thank you for a good reply - not the one I hoped to see, but still
)Well it is more like we are loading into a "non-unicode" environment
We don't want to get mapping errors, and a lot of the characters must be converted to pre-defined characters.
The nice thing would be to take it out and map it in a small programme "seach/replace style" .
In this way you could keep it for further editing every time we come across a new character.
As for now we edit a basic routine to be used within the DS Jobs that does the conversion, but in the long run this is not a good solution considering that the jobs that use the routine must be recompiled every time we change the routine code.
Hi,
Recompile the jobs using the routine simply since you changed the routine
why on earth would you do that 
it should work fine if it was compiled and tested OK 
Recompile the jobs using the routine simply since you changed the routine
why on earth would you do that it should work fine if it was compiled and tested OK Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Jinm,
as Roy has pointed out, the jobs that call a routine use only a pointer to it that is resolved at runtime, so you only need to re-compile the routine and any jobs that use it will get the new version; any jobs that are actively running using this routine will continue to use the old copy until they complete.
Setting up NLS maps is a lot of work, there is no doubt about it. It does sound like a better long-term solution than your current Uv/Basic one - I think that you will see a performance boost by using the builtin NLS functionality rather than the emulation.
as Roy has pointed out, the jobs that call a routine use only a pointer to it that is resolved at runtime, so you only need to re-compile the routine and any jobs that use it will get the new version; any jobs that are actively running using this routine will continue to use the old copy until they complete.
Setting up NLS maps is a lot of work, there is no doubt about it. It does sound like a better long-term solution than your current Uv/Basic one - I think that you will see a performance boost by using the builtin NLS functionality rather than the emulation.
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If you have NLS installed, there is a menu-driven system on the server for editing NLS maps and locales. It is invoked with the command NLS.ADMIN. However, edits are performed with the UniVerse line editor, with which you are probably not familiar.
The NLS manual describes this utility, as well as the database in which maps and locales are installed.
The NLS manual describes this utility, as well as the database in which maps and locales are installed.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Thanks for all the replies.
Perhaps I wasn't clear enough.
As to recompiling jobs. In environments other than developement we only have released jobs. These jobs take in the routine code as they were on the time of release. Consequently a changed routine will not be used in our validation and production environments unless a new release of the job in question is made.
Question: In case we migrate the changed routine, will a released job use the new routine code, or still only use the code embedded in the job when released??
As I pointed out I have used the NLS.ADMIN. It is far from ideal - my humble opinion - and I was trying to take the code page design out of the NLS.ADMIN , design and edit a new map ,and then from a command prompt do the registering of the newly designed code page. To me it seemed both easier, and a better way to document your mappings.
Perhaps I wasn't clear enough.
As to recompiling jobs. In environments other than developement we only have released jobs. These jobs take in the routine code as they were on the time of release. Consequently a changed routine will not be used in our validation and production environments unless a new release of the job in question is made.
Question: In case we migrate the changed routine, will a released job use the new routine code, or still only use the code embedded in the job when released??
As I pointed out I have used the NLS.ADMIN. It is far from ideal - my humble opinion - and I was trying to take the code page design out of the NLS.ADMIN , design and edit a new map ,and then from a command prompt do the registering of the newly designed code page. To me it seemed both easier, and a better way to document your mappings.
Hi Arn
Sorry to disagree.
Quote: "all that they contain are links to the routine" Un-quote.
Yes. Provided the routine is present in the given DS Project.
To illustrate: PROJECT_D (devl) is located at server AA100
We release the DS Job which embeds the code in the routines used within the job.
Now we migrate to PROJECT_V (validation) taking ONLY the released job and nothing else to PROJECT_V located at server AA200.
Routine code is till only in PROJECT_D and no matter how many changes I make in the code no way can the DS job catch and make use of these changes, since they reside in a different project and in this case on a different server.
My question then was: In case we migrate the changed ROUTINE to PROJECT_V will the DS Job then make use of the code embedded within the job or will it use the code from the routine, that is now suddelny present within the project.
Kind Rgds Jan
Sorry to disagree.
Quote: "all that they contain are links to the routine" Un-quote.
Yes. Provided the routine is present in the given DS Project.
To illustrate: PROJECT_D (devl) is located at server AA100
We release the DS Job which embeds the code in the routines used within the job.
Now we migrate to PROJECT_V (validation) taking ONLY the released job and nothing else to PROJECT_V located at server AA200.
Routine code is till only in PROJECT_D and no matter how many changes I make in the code no way can the DS job catch and make use of these changes, since they reside in a different project and in this case on a different server.
My question then was: In case we migrate the changed ROUTINE to PROJECT_V will the DS Job then make use of the code embedded within the job or will it use the code from the routine, that is now suddelny present within the project.
Kind Rgds Jan
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The job contains the name (strictly the "Catalog name") of the routine.
When the job attempts to invoke the routine, DataStage searches in the Catalog for the location of that routine. An error is generated if nothing is found.
If a routine of that name has been cataloged, and it has the correct number of arguments, then that routine is loaded into memory and executed, and its address in memory is cached within the job's data area until the job finishes.
Thus, the job invokes the most recently cataloged version of the routine.
If a new version of the routine is imported (or installed), then that new version is cataloged as part of that process.
(The "Catalog" is actually the VOC file, but the VOC file is also used for other purposes.)
It IS possible to make routines globally available to all projects on a DataStage server by cataloging them in a slightly different fashion. This techinque is not documented in the DataStage manuals, because "they" want you to keep all projects fully self-contained.
However, the technique can be found in the UniVerse BASIC manual, which can be downloaded from IBM's UniVerse library web site.
When the job attempts to invoke the routine, DataStage searches in the Catalog for the location of that routine. An error is generated if nothing is found.
If a routine of that name has been cataloged, and it has the correct number of arguments, then that routine is loaded into memory and executed, and its address in memory is cached within the job's data area until the job finishes.
Thus, the job invokes the most recently cataloged version of the routine.
If a new version of the routine is imported (or installed), then that new version is cataloged as part of that process.
(The "Catalog" is actually the VOC file, but the VOC file is also used for other purposes.)
It IS possible to make routines globally available to all projects on a DataStage server by cataloging them in a slightly different fashion. This techinque is not documented in the DataStage manuals, because "they" want you to keep all projects fully self-contained.
However, the technique can be found in the UniVerse BASIC manual, which can be downloaded from IBM's UniVerse library web site.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Routine object code is not embedded within the job, it is stored in a different location (DS_ROUTINES and a pointer in the VOC). If the routines are transported as part of a release they are put in this location. If you subsequently change the routine then the jobs will be affected.jinm wrote:...We release the DS Job which embeds the code in the routines used within the job...
It has been a while since I used the packager wizard, so it might be best for you to test this with a dummy job & routine so that you can confirm (or perhaps refute) your suspicions. Have the routine just call DSLogInfo('I am the original',''), package & move the job and subsequently recompile the routine with DSLogInfo('I am no longer the original','') and re-run the job without compiling.
Thanks for the replies.
The use of the VOC files is still somewhat of a mystery, but we are getting there
Not to be ungrateful, but the original topic was to edit a code map in an external editor, and then import the code map into the DS Server.
And on that topic I would really appreciate if there is a way to do this.
Rgds Jan
The use of the VOC files is still somewhat of a mystery, but we are getting there
Not to be ungrateful, but the original topic was to edit a code map in an external editor, and then import the code map into the DS Server.
And on that topic I would really appreciate if there is a way to do this.
Rgds Jan
Jan,
the code maps are stored in Hashed files, which can be read from and written to in DataStage jobs. You can write a job to extract the map information into a form you wish to use, utilize your favorite external editor to manipulate the file contents and then re-import into DataStage.
There is no need need to stress that errors in this process may make the map unusable, that is self-evident.
the code maps are stored in Hashed files, which can be read from and written to in DataStage jobs. You can write a job to extract the map information into a form you wish to use, utilize your favorite external editor to manipulate the file contents and then re-import into DataStage.
There is no need need to stress that errors in this process may make the map unusable, that is self-evident.
Hey ArndW
The greatest reply so far.
I suspected esactly something like that.
This should imply that you would store the result of the "IMPORT NLS Map" job as a hashed fil, -
target directory = Datastage\Engine\nls\maps\MAP.TABLE
Load and register it in the DS Administrator, and you are running.
Thanks a lot.
Rgds Jan
The greatest reply so far.
I suspected esactly something like that.
This should imply that you would store the result of the "IMPORT NLS Map" job as a hashed fil, -
target directory = Datastage\Engine\nls\maps\MAP.TABLE
Load and register it in the DS Administrator, and you are running.
Thanks a lot.
Rgds Jan
Routine changes not used in released jobs !!
Quote: If you subsequently change the routine then the jobs will be affected.
This is not true I'm afraid.
After completing the NLS issue I tried this one.
Simple Routine Ans = 1
One job uses this ruotine. Fine
Release Job, and change routine to Ans = 2. (Yes- I compiled the routine)
When running the non-released job output is 2, but when running
the released job, output is still 1.
I cannot see that a given job will take advantage of the newest version of a routine, when my experience - as stated in the first entries show, that a released job will only use a routine as it was on the time the job was released.
Any advice would be welcome
Rgds Jan
This is not true I'm afraid.
After completing the NLS issue I tried this one.
Simple Routine Ans = 1
One job uses this ruotine. Fine
Release Job, and change routine to Ans = 2. (Yes- I compiled the routine)
When running the non-released job output is 2, but when running
the released job, output is still 1.
I cannot see that a given job will take advantage of the newest version of a routine, when my experience - as stated in the first entries show, that a released job will only use a routine as it was on the time the job was released.
Any advice would be welcome
Rgds Jan