How to handle huge number of logs?
Moderators: chulett, rschirm, roy
How to handle huge number of logs?
Hi,
There is a job which loads almost 80 million records.
This job takes long time to finish because it faces a lot of warning for the data.
Almost 20 million records must be getting warning.
But the challenge is that warning for each record is a different based on the data.
If there is any feature or way to handle the warning like that of using wildcard features.
For example:
FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 71855[xA0]) from codepage UTF-8 to Unicode, substituting.
FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 2623[xA0][xA0][xA0][xA0]) from codepage UTF-8 to Unicode, substituting.
FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 1593[xA0][xA0][xA0][xA0]) from codepage UTF-8 to Unicode, substituting.
There is a job which loads almost 80 million records.
This job takes long time to finish because it faces a lot of warning for the data.
Almost 20 million records must be getting warning.
But the challenge is that warning for each record is a different based on the data.
If there is any feature or way to handle the warning like that of using wildcard features.
For example:
FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 71855[xA0]) from codepage UTF-8 to Unicode, substituting.
FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 2623[xA0][xA0][xA0][xA0]) from codepage UTF-8 to Unicode, substituting.
FULL_LOAD_FILES,0: Invalid character(s) ([xA0]) found converting string (code point(s): 1593[xA0][xA0][xA0][xA0]) from codepage UTF-8 to Unicode, substituting.
Let me give you some examples of the warning which will help you to understand the problem properly:
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE JORGE JUAN, 19 - - PISO 6[xFFFD] IZDA) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE ORENSE, 81 - PISO 3[xFFFD] PTA 4[xFFFD]) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE ORENSE, 81 - PISO 3[xFFFD] PTA 4[xFFFD]) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE RIO SIL, 37 - PLANTA 6[xFFFD] PTA D) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE RIO SIL, 37 - PLANTA 6[xFFFD] PTA D) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE PROVEN[xFFFD]A 88) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE PROVEN[xFFFD]A 88) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE MUNTANER, 438 - 4[xFFFD] 2[xFFFD]) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE MUNTANER, 438 - 4[xFFFD] 2[xFFFD]) to codepage ISO-8859-1, substituting.
How to handle this type of warnings through datastage message handler?
It can't be handled through the job in other way.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE JORGE JUAN, 19 - - PISO 6[xFFFD] IZDA) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE ORENSE, 81 - PISO 3[xFFFD] PTA 4[xFFFD]) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE ORENSE, 81 - PISO 3[xFFFD] PTA 4[xFFFD]) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE RIO SIL, 37 - PLANTA 6[xFFFD] PTA D) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE RIO SIL, 37 - PLANTA 6[xFFFD] PTA D) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE PROVEN[xFFFD]A 88) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE PROVEN[xFFFD]A 88) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE MUNTANER, 438 - 4[xFFFD] 2[xFFFD]) to codepage ISO-8859-1, substituting.
TRNF_FULL_LOAD,3: Invalid character(s) ([xFFFD]) found converting Unicode string (code point(s): CALLE MUNTANER, 438 - 4[xFFFD] 2[xFFFD]) to codepage ISO-8859-1, substituting.
How to handle this type of warnings through datastage message handler?
It can't be handled through the job in other way.
How did you determine that? Are you certain?It can't be handled through the job in other way.
I would recommend avoid using the message handler as that will only mask the problems. Using a message handler is like taking the easy way out. It's best if you can find a way to prevent each and every warning in every job log. It sounds like your NLS settings may be off.
Choose a job you love, and you will never have to work a day in your life. - Confucius
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
As Ray has asked, how many unique MESSAGE IDs are you seeing with these messages? Double click on the warnings in Director, there will be a Message Id field displayed.
The message text in the handler is just an example...
Regards,
The message text in the handler is just an example...
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.