big XML files vs DS

ppalka · Post by **ppalka** » Fri Dec 16, 2005 4:57 am

Would be DS able to process a big XML data? There will be about 20milion records in one input file, each record has 75 columns.

Regards,
Piotrek

chulett · Post by **chulett** » Fri Dec 16, 2005 8:40 am

Process as in 'read' or 'create'?

Read? It would need to come in via a Folder stage and I sincerely doubt there is any way it would handle a file of that size.

Write? Yes. Hope you're not in any hurry, however.

And since it can't create a file larger than 2GB, you'd need to chunk it up but at least the stage can do that automatically for you.

ppalka · Post by **ppalka** » Fri Dec 16, 2005 9:38 am

Read using XML Input

If there would be difficult in processing such large files, is there any other way to extract data from xml file to flat file? And not writing a basic routine

Or maybe the easiest way is to use some external tool to process that file?

Regards,
Piotrek

alanwms · Post by **alanwms** » Fri Dec 16, 2005 6:00 pm

The XML Input stage would handle the XML parsing into rows and columns, based on the Xpath declarations in the description field. Remember that the stage needs to find all the begin and end tags in the proper order, so make sure you're working with well-formed XML (XMLSpy is a good tool).

ppalka · Post by **ppalka** » Fri Dec 16, 2005 6:15 pm

But the xml input stage implies using a folder stage to read xml file. And I am wonder that it would be a problem to read a huge files by that stage...

chulett · Post by **chulett** » Fri Dec 16, 2005 6:25 pm

Yes, as already noted there are definite limits to what can be brought in via the Folder stage. Don't forget that it reads in each file as a record, with the filename going in the first field and the entire contents of the file going into the second field, which is then parsed by the XML Input stage.

Best guess is a limit of a few hundred megabytes. They were built for "real time" processing, so small bite-sized packets of XML is what it was really meant to handle.

aartlett · Post by **aartlett** » Sun Dec 18, 2005 4:22 pm

ppalka,
Can datastage handle huge XML ... probably within it's limits.

The question is SHOULD datastage handle huge XML?

IMHO no. There are better tools to handleXML.

Search here for more info.

bunu1977 · Post by **bunu1977** » Sun Dec 18, 2005 10:06 pm

HI,
I have done one projetc where we have read 6 records with 420 columns.
I was able to read the data but it was very slow.

Regds,
Dilip

bunu1977 · Post by **bunu1977** » Sun Dec 18, 2005 10:06 pm

HI,
I have done one projetc where we have read 6M records with 420 columns.
I was able to read the data but it was very slow.