which one is more costly

ArunaDas_Maharana · Post by **ArunaDas_Maharana** » Fri Jan 28, 2011 2:19 pm

hi All,

I am getting xmls from my souce with CRLF characters after each tag, while parsing with xml input it fails.

As a wor around i added Erepacle for search and strip off CRLF in the datastage transformer after my ORAOC19 stage in the server job , the message are stored as clob.

Another way i tried was while extracting data from oracle used replace function something like:
SELECT
Replace(Replace( AGGR_DATA_T.AGGR_MSG_BODY_IMG,chr(13) ,' '), chr(10),' ')

I am expecting 60000 or more records in half an hour time frame(batch will run once in evrery half an hour)

Share your thoughts on this!

Thanks,
Aruna

chulett · Post by **chulett** » Fri Jan 28, 2011 4:34 pm

XML files? How are you reading them? If you use a Sequential File stage then the CR/LF pairs could be an issue but shouldn't be if you let the XML Input stage read them directly via the URL/File Path option.

ArunaDas_Maharana · Post by **ArunaDas_Maharana** » Fri Jan 28, 2011 8:29 pm

Yes you are correct! parsing directly while reading from the database won't give this issue

From the database the xmls are landed in the sequential file

I found one more option where if you validate your xml with schema then it will strip off the CRLF from the sequential file.

I can't change change the design at the moment, in your view with the current senario in terms of processing high volume which method you think will be best i am planning for some performance testing on below 2 options:
1. With schema validation
2. With replace function at the oracle stage while extracting the data.

could you please share your view.

chulett · Post by **chulett** » Sat Jan 29, 2011 9:32 am

Change the design. And I'm not suggestion reading directly from the database, I'm suggesting you let the XML Input stage read them. I'm assuming your job starts with a Sequential File stage that feeds the XML Input. If that's true, then you need to replace the Sequential File stage with a Folder stage with one column in it - the filename. Then set the XML stage to the "URL/Filepath" option and your problem should be solved.

If that's not true, please be more precise about your job design.

For reference, an entry from the excellent blog Ernie Ostic writes:

http://dsrealtime.wordpress.com/2007/12 ... -a-source/

ArunaDas_Maharana · Post by **ArunaDas_Maharana** » Tue Feb 01, 2011 8:32 pm

Thanks chulett

I will try the two options folder and external stage..