XM Input Stage

saikir · Post by **saikir** » Mon Apr 13, 2009 2:07 am

Hi,

I am new to using XML capabilities of DataStage. I would like to know whether there is any limit in the number of rows that can be read in terms of count or Size? Say for e.g. there is a 2GB limit on DataSet?

Regards,
Sai

mahadev.v · Post by **mahadev.v** » Mon Apr 13, 2009 3:06 am

The 2GB limit on the DataSet is imposed by the operating system constraints and not a DataStage constraint (mostly for the data files in the resource disk). Number of rows is never a problem, and the size is also not a constraint as long as it is not constrained by the OS.

eostic · Post by **eostic** » Mon Apr 13, 2009 6:29 pm

You are likely to hit in-memory limits in the current version of XMLPack, long before you have issues with ultimate filesize. A "single document" cannot be more than about 300 to 500M in size, depending on the number of nodes and your env....

...however, that is "single document"...you could certainly run a job that is retrieving, parsing, and reading/processing hundreds of such documents in a single run, where such documents are a collection of XML instance documents in a single subdirectory.

Ernie