Huge XML file

pavan31081980 · Post by **pavan31081980** » Fri Jul 10, 2009 6:18 am

Hi!

I am reading a XML file thru an XML input stage and loading into oracle table.The size of the xml file is 85 MB.The job is running continoulsy for more than 120 minutes.

Could you please help me in reducing the runtime of the job.If any performance related measures need to be taken in the job?

Thanks in advance

Regards,
Pavan

chulett · Post by **chulett** » Fri Jul 10, 2009 6:44 am

That's not huge. You need to determine your bottleneck, have you broken your job into logical pieces to see where the actual problem is? For example, replace Oracle with a flat file and check the speed.

vmcburney · Post by **vmcburney** » Mon Jul 13, 2009 12:04 am

The XML Input stage will try to read the entire XML file into memory to validate the syntax before it does any processing so it could be a long time before it starts processing. If you start it on a number of nodes it might be trying to read the XML file on those different nodes. Try running it on a single node or in single thread debug mode to see if it runs faster.

Sreenivasulu · Post by **Sreenivasulu** » Mon Jul 13, 2009 12:57 am

Hi All,

XML Processing takes very long time in datastage. I think its built for processing only status messages not data files. we moed from XML to CSV because of the time delays

Regards
Sreeni

sajidkp · Post by **sajidkp** » Tue Jul 14, 2009 12:26 am

Huge XML (1GB and above )processing is always a bottleneck in ETL tools . you can find out some way to convert these xmls to flat files using some external programs or you can write a script/program to parse the XMLs . there will be good performance advantage