Page 1 of 1
XM Input Stage
Posted: Mon Apr 13, 2009 2:07 am
by saikir
Hi,
I am new to using XML capabilities of DataStage. I would like to know whether there is any limit in the number of rows that can be read in terms of count or Size? Say for e.g. there is a 2GB limit on DataSet?
Regards,
Sai
Posted: Mon Apr 13, 2009 3:06 am
by mahadev.v
The 2GB limit on the DataSet is imposed by the operating system constraints and not a DataStage constraint (mostly for the data files in the resource disk). Number of rows is never a problem, and the size is also not a constraint as long as it is not constrained by the OS.
Posted: Mon Apr 13, 2009 6:29 pm
by eostic
You are likely to hit in-memory limits in the current version of XMLPack, long before you have issues with ultimate filesize. A "single document" cannot be more than about 300 to 500M in size, depending on the number of nodes and your env....
...however, that is "single document"...you could certainly run a job that is retrieving, parsing, and reading/processing hundreds of such documents in a single run, where such documents are a collection of XML instance documents in a single subdirectory.
Ernie