How to handle huge number of logs?

saraswati · Post by **saraswati** » Thu Jan 26, 2012 11:38 pm

Hi,

There is a job which loads almost 80 million records.

This job takes long time to finish because it faces a lot of warning for the data.

Almost 20 million records must be getting warning.

But the challenge is that warning for each record is a different based on the data.

If there is any feature or way to handle the warning like that of using wildcard features.

For example:

FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 71855[xA0]) from codepage UTF-8 to Unicode, substituting.

FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 2623[xA0][xA0][xA0][xA0]) from codepage UTF-8 to Unicode, substituting.

FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 1593[xA0][xA0][xA0][xA0]) from codepage UTF-8 to Unicode, substituting.

saraswati · Post by **saraswati** » Thu Jan 26, 2012 11:44 pm

Let me give you some examples of the warning which will help you to understand the problem properly:

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE JORGE JUAN, 19 - - PISO 6[xFFFD] IZDA) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE ORENSE, 81 - PISO 3[xFFFD] PTA 4[xFFFD]) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE ORENSE, 81 - PISO 3[xFFFD] PTA 4[xFFFD]) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE RIO SIL, 37 - PLANTA 6[xFFFD] PTA D) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE RIO SIL, 37 - PLANTA 6[xFFFD] PTA D) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE PROVEN[xFFFD]A 88) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE PROVEN[xFFFD]A 88) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE MUNTANER, 438 - 4[xFFFD] 2[xFFFD]) to codepage ISO-8859-1, substituting.

TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE MUNTANER, 438 - 4[xFFFD] 2[xFFFD]) to codepage ISO-8859-1, substituting.

How to handle this type of warnings through datastage message handler?

It can't be handled through the job in other way.

qt_ky · Post by **qt_ky** » Thu Jan 26, 2012 11:53 pm

It can't be handled through the job in other way.

How did you determine that? Are you certain?

I would recommend avoid using the message handler as that will only mask the problems. Using a message handler is like taking the easy way out. It's best if you can find a way to prevent each and every warning in every job log. It sounds like your NLS settings may be off.

saraswati · Post by **saraswati** » Fri Jan 27, 2012 9:43 am

How do you think it can be handled through message handler as each warning is different based on the data?

ray.wurlod · Post by **ray.wurlod** » Fri Jan 27, 2012 3:31 pm

How many different MESSAGE ID values do you get? It's the MESSAGE ID that drives the message handler, not the content.

PaulVL · Post by **PaulVL** » Fri Jan 27, 2012 9:05 pm

You are getting the warning messages for a reason. Ask you DBA what character set the table is using, then configure your job to use the same character set.

Look at each stage as well.

solve the issue, don't bypass it.

saraswati · Post by **saraswati** » Tue Jan 31, 2012 1:48 pm

I had tried everything with the DBA.

I just want to bypass the warning.

The challenge is that every warning is different.But the starting few words of the warning are the same.

Can anyone please suggest....how to handle it?

Can we use some logic like wildcard logic to bypass these warnings?

saraswati · Post by **saraswati** » Tue Jan 31, 2012 1:50 pm

Like warning starting with "TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s)" should be bypass.

It was just an example.

jwiles · Post by **jwiles** » Tue Jan 31, 2012 2:26 pm

As Ray has asked, how many unique MESSAGE IDs are you seeing with these messages? Double click on the warnings in Director, there will be a Message Id field displayed.

The message text in the handler is just an example...

Regards,