is there any way to increase the inter stag rowbuffer?

waklook · Post by **waklook** » Wed May 02, 2007 2:03 am

Hi,

i have this error while activating inter process rowbuffer,
" row too big for inter stage rowbuffer ",
i searched the forum but it did not solved my problem.

my ques is how can i increase the roubuffer size to 20mb, bcz my row size is like 20mb, is there any setting or configration i can play with?

thanks in advance.

kumar_s · Post by **kumar_s** » Wed May 02, 2007 3:45 am

Performance tab in Job properties. Or at Admistrator client for project defaults.

ray.wurlod · Post by **ray.wurlod** » Wed May 02, 2007 6:15 am

20MB per ROW ?!!!

Maybe some kind of re-design is in order.

That is an utterly ridiculous row size. What's in a row? Do you really need to move it all together? Can you move part rows? Or even not move some parts through DataStage at all? Anything that doesn't need transformation can be moved outside of DataStage, whether or not under DataStage control.

waklook · Post by **waklook** » Wed May 02, 2007 6:36 am

Hi again

Ray

say it's 5mb how can i do that?

chulett · Post by **chulett** » Wed May 02, 2007 6:42 am

So... you're just making these sizes up?

Kumar has already answered your question as asked. Ray has attempted to bring some sanity to it.

waklook · Post by **waklook** » Wed May 02, 2007 8:58 am

Sorry guys

i'm not making these sizes up?

i have realy 17,296kb file size and many files that are more than 15,000kb
and the size increases litle bit every load.(monthly).

i know about Kumar answer , the max size availble is 1024kb , i'm only asking if there any other solution.

sorry if i did or said something wrong.

chulett · Post by **chulett** » Wed May 02, 2007 9:04 am

File sizes are one thing, you were talking about 20MB ROW sizes. That's a completely different animal. What kind of average row size are you dealing with?

mctny · Post by **mctny** » Wed May 02, 2007 9:56 am

Hi Waklook,

are your files one line file, or are you trying to read them in one row? maybe there is a problem with your EOL characters in your file

ray.wurlod · Post by **ray.wurlod** » Wed May 02, 2007 4:47 pm

The reason there's a 1MB upper limit on row buffers is that no-one figured anyone would need anywhere near that. The default limit for ODBC connections, for example, is 8KB per row.

kumar_s · Post by **kumar_s** » Wed May 02, 2007 7:11 pm

Or is it any kind of 'large objects' involved?

waklook · Post by **waklook** » Sat May 05, 2007 2:09 am

Hi,

thanks to all of you guys for your replys,

i have xml files with max size around 20mb,
when i use the link partitioner and link collector i have to activate the inter process rowbuffer, at this point it will move every file as one record and bcz the file is too big(row) i have that error,may be i'm wrong understanding it that way.
i have tryed using the file path but i failed. this month i have > 700,000 xml file and it takes >40H to finish loading. i have some other development going on, at the same time i'm trying to improve the performance for this one and i failed,i'm asking you guys to help me on that by asking these questions.

thanks any way to all u.

kumar_s · Post by **kumar_s** » Sat May 05, 2007 3:01 am

Dont you have any transformation involved in the XML file?
If not, why can you move it outside of Datastage, like FTP etc.,
Other options is to parse the XML file using XML readers with sepecific metadata.

chulett · Post by **chulett** » Sat May 05, 2007 7:43 am

It always helps to explain what you are doing rather than ask a single targeted question with nothing to give it context.

The size of the XML file shouldn't be an issue. I've processed XML files that are hundreds of megabytes in size without issue and without buffering or IPC stages or any other hooey like that. I've also done up to 500,000 'at one time' as well.

Describe your job design in detail, please. The first thing you should be doing is parsing the XML and flattening it into rows, not passing it whole down your job. For example:

Folder -> XML Input -> everything else

The folder stage will want to move the file as one large record by default, but that's simple to override. Only create one field in the Folder to return the filename and then set the XML Input stage to use the URL/File path column content option on the XML Source tab. Then it reads the XML file directly rather than taking it as one huge record from the Folder stage and the 'everything else' part of your job simply passes normal rows downstream.

ray.wurlod · Post by **ray.wurlod** » Sat May 05, 2007 3:21 pm

Had you mentioned XML in the original post, you would have received that advice much earlier.

chulett · Post by **chulett** » Sat May 05, 2007 3:46 pm

Exactly.