Large Xml Files

RelianceDss · Post by **RelianceDss** » Sat May 19, 2007 6:35 am

Hi,

We are facing problems reading large xml files(250 mbs). It gives following error
PLU_XML..XML_PLU: XML input document value is empty or NULL. Column Name = "Record"
I have tried both 2 parameter and single parameter(url path) approach.
Also the reading of xml is pretty slow.
1. Are there any other methods for better reading of xmls?
2. How to improve performance while xml reading?
3. Can anyone give me names of xml to flat file converter utilities?

I have read in many posts that large XMLs are insane, but this is what we are getting from the source and we need to process them at our end.

eostic · Post by **eostic** » Sat May 19, 2007 7:17 am

Hi...

How far do you get? Are you certain it's a size issue? How large an instance (of this same structured document) are you able to read succcessfully? 50m? 100m? 200m?

Ernie

wnogalski · Post by **wnogalski** » Sat May 19, 2007 7:30 am

DataStage has problems with XML files larger than 200MBs.

A better solution is to write a simple program which will parse the file and create f.e. a comma delimited sequential file from the XML.

HTH

chulett · Post by **chulett** » Sat May 19, 2007 7:49 am

I don't know how anyone can make a blanket statement about 200MBs being a limit for DataStage, there's way too many variables involved. I've succesfully processed files nearly twice that on my system.

Plus, whenever I've had 'size' problems they aren't nearly so nice - the job just falls over dead. Is this one large file that gives this message? Many 'large' files? If it is just one, I'd wonder if the problem lies in the file itself and not its size.

And not everyone can 'write a simple program' to parse XML... what would you suggest to do that? Seeing as how some companies make a living delivering XML tools, not really sure how 'simple' it would really be. [shrug]

ray.wurlod · Post by **ray.wurlod** » Sat May 19, 2007 3:17 pm

Back to basics, can you guarantee that the XML is well-formed?

RelianceDss · Post by **RelianceDss** » Wed May 23, 2007 5:04 am

ray.wurlod wrote:Back to basics, can you guarantee that the XML is well-formed?

--The XML are well formed for sure.
--I am able to process 100 Mbs of xml file but anything more than that fails. The tag which we are trying to read has 69 attributes.
It gives error followinf error message
"Abnormal termination of stage Xml_test..XML_Input_1 detected"
The box is a 4 cpu machine with 16GB physical mem. Its a sun4u sparc SUNW,Sun-Fire-V440. Os-SunOS 5.10

While testing I see to it that nothing else is running on the box.
Can we anywayz avoid using Folder stage while using XML-input stage?

chulett · Post by **chulett** » Wed May 23, 2007 7:05 am

If you have used the Folder stage with one column passing only the filename and then set the XML Input stage to 'URL/Filepath' the issue is not with the Folder stage.

Post your question to your Support provider. I think you'll find the issue is your O/S which has... quirks... with XML, including fun things like the 'square root of a negative number' problem, which from what I recall is unique to Sun.

mystuff · Post by **mystuff** » Tue May 29, 2007 6:31 am

I don't know how anyone can make a blanket statement about 200MBs being a limit for DataStage, there's way too many variables involved.

a) Can you give me an idea about those variables involved.. All I know is size of the file .... which probably depends on physical memory available.

b) Could there be a rough estimated depending on available physical memory? Like for 16GB preferably not more than 200MB of each XML file.. something like that?

chulett · Post by **chulett** » Tue May 29, 2007 6:36 am

I've never seen any metric for that. It seems to vary widely enough that all you can do is process files on your system until you reach your particular limit.

Perhaps Ernie might come along and have more light to shed on this.

eostic · Post by **eostic** » Wed May 30, 2007 9:49 pm

I haven't seen any good ways to calculate it, unfortunately. Perhaps it's something that could be dug up on the web (apache C++ xerces and xalan is being implemented deep inside the Stage). I suspect it's probably wildly variable depending on hierarchy, data types, element and attribute name lengths, values, etc.

Ernie

RelianceDss · Post by **RelianceDss** » Wed Jun 20, 2007 7:22 am

Hi,

We have taken a directional decision in not using xml-input plugin. We are trying to flatten XMLs using perl program where we have also added some business rules.
Our testing on big XML files(more than 100mbs) was giving inconsistent result and the performance was also pathetic.

Anywayz guys thnks a lot for all the suggestions.

rgrds