How much big xml file a XML input stage can process

yogeshhd · Post by **yogeshhd** » Wed Sep 01, 2010 5:11 am

I am using XML input stage in a parallel job. I wanted to know how big xml file the xml input stage can handle? if some one has info on how this stage uses memory that would be helpful for me. thanks

eostic · Post by **eostic** » Wed Sep 01, 2010 5:34 am

Do some searches..this subject has come up often.....you should be fine in the under 200Meg range...there may be some APT_ variables that you will need to set, but that depends more on your approach to reading the xml from disk or message queue.

If you hit such a limit, there are tools and techniques for breaking up xml. How big are your single documents?

Ernie

yogeshhd · Post by **yogeshhd** » Wed Sep 01, 2010 5:59 am

The files are of size 500 MB. I am passing xml file names with directory path to the xml input stage.

eostic · Post by **eostic** » Wed Sep 01, 2010 6:25 am

Chances are you will run into issues. I have done "some" documents of that size, but as noted earlier, it depends on the number and size of the elements, etc.

If you happen to be on Windows, check out a tool called XMLMax ....it has options for breaking apart xml into smaller pieces. I've seen other solutions too, and custom ones from our consulting teams.

Ernie

chulett · Post by **chulett** » Wed Sep 01, 2010 6:43 am

So either you or whomever is building those XML files will need to break them up into smaller deliverables. The stage can process huge numbers of small files but not huge files.