Hi everyone, I am using a xml stage to parse a xml which has more than 2 million characters, so I defined a varchar field with a length of 9999999 (9.9 million ) and I got the follwing error message
row too big for inter stage rowbuffer
does anyone know how to deal with this?
your help will be greatly appreciated! Thanks.
row too big for inter stage rowbuffer
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 37
- Joined: Mon Jan 24, 2005 10:12 am
-
- Participant
- Posts: 37
- Joined: Mon Jan 24, 2005 10:12 am
Re: row too big for inter stage rowbuffer
I defined it as varchar of 9999999
supernova2005 wrote:Hi everyone, I am using a xml stage to parse a xml which has more than 2 million characters, so I defined a varchar field with a length of 9999999 (9.9 million ) and I got the follwing error message
row too big for inter stage rowbuffer
does anyone know how to deal with this?
your help will be greatly appreciated! Thanks.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
A single 'row' or 'element' in this XML file is 2M characters or the entire file is? I'm curious exactly how you have set up the parsing. Are you using the XML Input stage? What methodology is it using to read the file - is the XML source column option set to XML document or URL/File path?
If you are using the former, I'd suggest switching to the latter. Just set the stage to 'URL/File path' and pass in the full path of the XML file, not the entire contents of the file in one large field. That should solve your issue, I would think.
If you are using the former, I'd suggest switching to the latter. Just set the stage to 'URL/File path' and pass in the full path of the XML file, not the entire contents of the file in one large field. That should solve your issue, I would think.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 37
- Joined: Mon Jan 24, 2005 10:12 am
The entire file has more than 2 million characters. I use an xml input stage, and the xml source is 'xml document'. I saw your message earlier this morning , so I changed the source to 'url/file path', But I am still getting the same error message.
This is the error message:
FBuilder_DTD_test.#0.getXMLfilesC1.lnkGetXMLFile-Output.lnkXMLout: ds_ipcput() - row too big for inter stage rowbuffer
my buffer size is 314572800000.
This is the error message:
FBuilder_DTD_test.#0.getXMLfilesC1.lnkGetXMLFile-Output.lnkXMLout: ds_ipcput() - row too big for inter stage rowbuffer
my buffer size is 314572800000.
chulett wrote:A single 'row' or 'element' in this XML file is 2M characters or the entire file is? I'm curious exactly how you have set up the parsing. Are you using the XML Input stage? What methodology is it using to read the file - is the XML source column option set to XML document or URL/File path?
If you are using the former, I'd suggest switching to the latter. Just set the stage to 'URL/File path' and pass in the full path of the XML file, not the entire contents of the file in one large field. That should solve your issue, I would think.
Two million characters is only 2MB, give or take. I've used this mechanism to process 150MB files without issue, so not sure what's going on here. Perhaps it's just a matter of turning off row buffering? How big is any given flattened row?
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers