DSXchange

Hi everyone, I am using a xml stage to parse a xml which has more than 2 million characters, so I defined a varchar field with a length of 9999999 (9.9 million ) and I got the follwing error message

row too big for inter stage rowbuffer

does anyone know how to deal with this?

your help will be greatly appreciated! Thanks.

Whats your buffer size and I presume your SQLType is LongVarChar-9999999

I defined it as varchar of 9999999

supernova2005 wrote:Hi everyone, I am using a xml stage to parse a xml which has more than 2 million characters, so I defined a varchar field with a length of 9999999 (9.9 million ) and I got the follwing error message

row too big for inter stage rowbuffer

does anyone know how to deal with this?

your help will be greatly appreciated! Thanks.

And what are your buffer sizes, particularly the maximum buffer size? These are set by environment variables.

A single 'row' or 'element' in this XML file is 2M characters or the entire file is? I'm curious exactly how you have set up the parsing. Are you using the XML Input stage? What methodology is it using to read the file - is the XML source column option set to XML document or URL/File path?

If you are using the former, I'd suggest switching to the latter. Just set the stage to 'URL/File path' and pass in the full path of the XML file, not the entire contents of the file in one large field. That should solve your issue, I would think.

The entire file has more than 2 million characters. I use an xml input stage, and the xml source is 'xml document'. I saw your message earlier this morning , so I changed the source to 'url/file path', But I am still getting the same error message.

This is the error message:
FBuilder_DTD_test.#0.getXMLfilesC1.lnkGetXMLFile-Output.lnkXMLout: ds_ipcput() - row too big for inter stage rowbuffer

my buffer size is 314572800000.

chulett wrote:A single 'row' or 'element' in this XML file is 2M characters or the entire file is? I'm curious exactly how you have set up the parsing. Are you using the XML Input stage? What methodology is it using to read the file - is the XML source column option set to XML document or URL/File path?

If you are using the former, I'd suggest switching to the latter. Just set the stage to 'URL/File path' and pass in the full path of the XML file, not the entire contents of the file in one large field. That should solve your issue, I would think.

Two million characters is only 2MB, give or take. I've used this mechanism to process 150MB files without issue, so not sure what's going on here. Perhaps it's just a matter of turning off row buffering? How big is any given flattened row?