Migration of DS7.1 to DS7.5.2:- Data quality issue

[abhishekachrekar]

Hi,

We are migrating an application from DS7.1 to DS7.5.2.
While testing we have come up some issues as described below:

The source data has some special characters like "" (Arrow).

In DS 7.1 this character comes correctly in the output, but in DS 7.5.2 it gets converted into some junk characters "â„–".

In both the environments we are refering to the same database environment. Hence there is not a slight possiblility of losing the integrity while loading in database.

The NLS settings of both the Datastage versions are same. (UTF-8 ).
The database NLS setting (common for both environment) is also UTF-8.

Can anyone please suggest why we cannot preserve special characters in DS7.5.2?

Regards,
Abhishek

[abhishekachrekar]

While running the process on D7.5.2 we get the following warnings.

Warning! One or more of your selected locales are not available.

Please invoke the commands "locale" and "locale -a" to verify your

selections and the available locales.

Continuing processing using the "C" locale.

It looks like something to do with NLS settings.
Is it the reason why the special characters are not preserved in DS7.5.2?

Regards,
Abhishek

[abhishekachrekar]

Hi,

Can anyone help me with my above given query?
Waiting desperately for your response

Regards,
Abhishek

[chulett]

In the meantime, you've not involved your official support provider or were they not able to help? Also, did you run the commands it mentioned?

[abhishekachrekar]

Hi Craig,

I checked with our solution support team (unix Guru in our project) but they are also not aware of this issue.

In addition we are also having some other weird problems as given below.
1> Our process archives the output file in a archive directory and appends the file name with the current timestamp. But in 7.5.2 setup the timestamp is missing.

E.g.
7.1 setup: Archived file name: AccountUpdate_20090430115303.dat.processed.200905061724

7.5.2 setup: AccountUpdate_20090430115303.dat.processed.

2> In our process we have some derivations which populate some date time fields in the output file. These fields are populated correctly in 7.1 setup but they are blanked in 7.5.2 setup.


I think some server settings might have been changed recently which I am not aware of. Today in the evening we are planning to reboot our server in the hope that it may solve the problem.

After running the locale -a command I get the following output.
C
POSIX
C.iso88591
C.iso885915
C.utf8
univ.utf8
ar_DZ.arabic8
ar_SA.arabic8
ar_SA.iso88596
bg_BG.iso88595
zh_CN.hp15CN
zh_CN.utf8
zh_HK.big5
zh_HK.utf8
zh_TW.big5
zh_TW.ccdc
zh_TW.eucTW
zh_TW.utf8
cs_CZ.iso88592
da_DK.iso88591
da_DK.iso885915@euro
da_DK.roman8
nl_NL.iso88591
nl_NL.iso885915@euro
nl_NL.roman8
en_GB.iso88591
en_GB.iso885915@euro
en_GB.roman8
en_US.iso88591
en_US.roman8
fi_FI.iso88591
fi_FI.iso885915@euro
fi_FI.roman8
fr_CA.iso88591
fr_CA.iso885915
fr_CA.roman8
fr_CA.utf8
fr_FR.iso88591
fr_FR.iso885915@euro
fr_FR.roman8
fr_FR.utf8
de_DE.iso88591
de_DE.iso885915@euro
de_DE.roman8
de_DE.utf8
el_GR.greek8
el_GR.iso88597
iw_IL.hebrew8
iw_IL.iso88598
hu_HU.iso88592
is_IS.iso88591
is_IS.iso885915@euro
is_IS.roman8
it_IT.iso88591
it_IT.iso885915@euro
it_IT.roman8
it_IT.utf8
ja_JP.SJIS
ja_JP.eucJP
ja_JP.kana8
ja_JP.utf8
ko_KR.eucKR
ko_KR.utf8
no_NO.iso88591
no_NO.iso885915@euro
no_NO.roman8
pl_PL.iso88592
pt_PT.iso88591
pt_PT.iso885915@euro
pt_PT.roman8
ro_RO.iso88592
ru_RU.iso88595
hr_HR.iso88592
sk_SK.iso88592
sl_SI.iso88592
es_ES.iso88591
es_ES.iso885915@euro
es_ES.roman8
es_ES.utf8
sv_SE.iso88591
sv_SE.utf8
sv_SE.iso885915@euro
sv_SE.roman8
th_TH.tis620
tr_TR.iso88599
tr_TR.turkish8
zh_CN.gb18030
zh_HK.hkbig5
el_GR.utf8

Regards,
Abhishek

[ray.wurlod]

It may be a locale file for the database (rather than for DataStage) that is missing.

[chulett]

And I specifically meant your DataStage support people. Good to check with other resources, but I would have started there.

[abhishekachrekar]

Hi Ray/Craig,

We are using the same Database for both DS7.1 and DS7.5.2 which is in UTF-8.

Also I have checked the Datastage Server Maps and the Parallel maps in both the setup. They are exactly same.

Also both the setup are on the same box and the same locale is used by both. Local -a command output is the same for both setups.

Regards,
Abhishek

[abhishekachrekar]

Hi All,

The issue is resolved now.
The datastage project was corrupted and hence we were facing all the above issues.
I recreated the project with the same settings as before and now all is working fine.

Regards,
Abhishek

[ray.wurlod]

What was the corruption? How did you discover it? How do you think it happened? What steps have you take to prevent it from occurring again?

[abhishekachrekar]

Actually the problems kept aggrevating over the time. First special characters were not visible, later we got the warning about locale, then we noticed timestamps were missing, then recently the job logs got corrupted. At this stage we were not able to run the process from directopr/designer but we were able to run in through scripts. Hence we recreated the project and now all works fine.

REgards,
Abhishek