Does anyone know if their is a size constraint for the XML Input stage? I have read several conflicting posts and want to get some opinions. I am trying to make a strategic direction to use XML vs. Flat file and really need some input.
Thanks
David
XML Input Stage File Size Constraint
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 35
- Joined: Thu Apr 19, 2007 12:42 pm
What kind of sizes would you be dealing with? There are definite size limits when using the traditional 'two fields, all content at once' methodolgy with the Folder stage. Cutting that back to just one field to pass the filename and letting the XML Input file read the file directly pretty much removes that limit.
Problem with the size limit is it seems to vary from install to install, O/S to O/S, etc. My limit was around 50MB from what I recall. And I say 'pretty much' on the second methodology as I've been able to parse XML files in the 500MB range but I've read about others having issues with GB sized files. To me, XML files that big are just insane, but sometimes we have to live in an insane world.
Hope that helps.
Problem with the size limit is it seems to vary from install to install, O/S to O/S, etc. My limit was around 50MB from what I recall. And I say 'pretty much' on the second methodology as I've been able to parse XML files in the 500MB range but I've read about others having issues with GB sized files. To me, XML files that big are just insane, but sometimes we have to live in an insane world.
Hope that helps.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
How would this affect on size limit, i mean why would using traditional 'two fields, all content at once' methodology with the folder stage' impose file limit, where as other doesnt.chulett : What kind of sizes would you be dealing with? There are definite size limits when using the traditional 'two fields, all content at once' methodolgy with the Folder stage. Cutting that back to just one field to pass the filename and letting the XML Input file read the file directly pretty much removes that limit.
Problem with the size limit is it seems to vary from install to install, O/S to O/S, etc. My limit was around 50MB from what I recall. And I say 'pretty much' on the second methodology as I've been able to parse XML files in the 500MB range but I've read about others having issues with GB sized files. To me, XML files that big are just insane, but sometimes we have to live in an insane world.
Do you recall any of those issues faced with GB sized files?
The 'two field' way the Folder stage brings the entire XML file into memory and then passes it as one record / field to the XML Input stage. In the other, the XML stage does its own dirty work. It's better at it.
In all cases, the 'issues' are the same - your job falls over dead. Nothing more than that, bang abby-normal termination dead.
In all cases, the 'issues' are the same - your job falls over dead. Nothing more than that, bang abby-normal termination dead.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Because with the Sequential stage you are reading multiple "rows" in the file.... ie.. read a row (till CRLF, for example) and then send that down the link........
...with the folder stage, it's going to "read the whole file as a single row/column" and try to send that down the link.
Presumably, with your flat file, it has many rows, each far smaller than 400G. With the Folder stage, that's it --- one row, one column actually, per whole file.
Ernie
...with the folder stage, it's going to "read the whole file as a single row/column" and try to send that down the link.
Presumably, with your flat file, it has many rows, each far smaller than 400G. With the Folder stage, that's it --- one row, one column actually, per whole file.
Ernie
a) When such kind of error occurs, is there a way to deal with it.your job falls over dead. Nothing more than that, bang abby-normal termination dead.
b) If complete XML file (data) can't be accomodated in physical memory, then doesn't it access with hard disk through mechanism of paging. The time of response would be more. But I guess the job shouldnt be dead for that reason, isnt it?