XML vs. Flat Files

rkumar28 · Post by **rkumar28** » Sun May 16, 2004 12:06 pm

Hi,

I wanted a recommendation related to a Flat File vs. XML. My company is debating on using an XML vs. flat file as a SOURCE in data stage. The data size will be around 300,000 rows.

I need recommendation on pros and cons of using XML over Flat files. Is using XML is beneficial than flat file in data stage.

Thanks for any advice and time.

ray.wurlod · Post by **ray.wurlod** » Sun May 16, 2004 3:58 pm

Flat file is fastest, if for no other reason than that it's easier to parse the data from it based on metadata definitions. You only need to process flat file metadata once (assuming the format/content doesn't change, which is not always the case!). With XML there's the overhead of processing the tags and verifying them, as well as processing the data. For the small added safety it offers in getting things right, I believe that the cost of using XML over text files is unwarranted. If you have the choice, go for flat files every time! Of course, if you don't have the choice, and your data are being delivered in XML format, then DataStage can handle that, too.

rkumar28 · Post by **rkumar28** » Sun May 16, 2004 5:38 pm

Do we have to write a parser for the XML file in data stage or data stage can handle XML read and write all by itself.

Thanks

ray.wurlod wrote:Flat file is fastest, if for no other reason than that it's easier to parse the data from it based on metadata definitions. You only need to process flat file metadata once (assuming the format/content doesn't change, which is not always the case!). With XML there's the overhead of processing the tags and verifying them, as well as processing the data. For the small added safety it offers in getting things right, I believe that the cost of using XML over text files is unwarranted. If you have the choice, go for flat files every time! Of course, if you don't have the choice, and your data are being delivered in XML format, then DataStage can handle that, too.

ray.wurlod · Post by **ray.wurlod** » Sun May 16, 2004 7:27 pm

DataStage version 7.x has XML reader and XML writer stage types.

Consult on-line help or the relevant manual (these are in your DataStage client install folder, in the Docs sub-folder). The relevant manual in this case is XML PACK Designer Guide (XMLPACK_20_Designer.pdf).

In response to your private message, technically XML files can be transmitted by FTP. They are, after all, still pure text (the tags as well as the data are text). Politically is an entirely different question!

chulett · Post by **chulett** » Mon May 17, 2004 6:44 am

Ray has already pretty much covered this, just wanted to throw another log of opinion onto the fire. Warning, this is all "IMHO" stuff coming.

I know that XML is all the rage among some circles and I've had people shake their heads and mutter about me being a Luddite for not choosing to use XML at every opportunity. That's the key word to me - choice. If you've got one, stick with flat files. When you need to process XML, either as a Source or Target, then go for it. Study the XML docs and make sure you understand the limitations of the different approaches. Otherwise, stick with something that's generally smaller, more flexible and faster to process - the dreaded sequential file.

ray.wurlod · Post by **ray.wurlod** » Mon May 17, 2004 3:51 pm

Ultimately, performance is mainly about the work you can avoid doing.

asadi · Post by **asadi** » Fri May 21, 2004 4:53 pm

I know you guys have already touched on performance, but l would just reiterate that the XML parser in DataStage is very slow. As Ray said stick to flat files.