is there any way to increase the inter stag rowbuffer?

[waklook]

Hi;

thanks Ray and Craig and sorry for short,

i posted this problem before with the job desc, and what i'm trying to do is according to Ernie advise (thanks Ernie), and i have exactly this error,
so i sked according to that error.
Craig: i have tryed the URL/File path but it does't work with me, and if so can you please explain how cat it improve the job perforance.
the job design is like this:
FolderStage --> transformerStage --> XML inputStage --> transformerStage --> 2 SequentialFile + other stuff including pivot stage, >90% of the job time is between
FolderStage --> transformerStage --> XML inputStage --> transformerStage --> 2SequentialFile,
and yes i'm going down to the /text() for all my columns and they are around 255 columns.

"the job is running on monthly basis"

this is the link for my previous post on this issue.

viewtopic.php?p=219830&highlight=&sid=b ... 25f#219830

[chulett]

i have tryed the URL/File path but it does't work with me

What exactly "doesn't work" about it?

the job design is like this:
FolderStage --> transformerStage --> XML inputStage --> transformerStage --> 2 SequentialFile + other stuff including pivot stage

Remove the first transformer, it can't be providing any value, only a 'chokepoint' in the processing. Go directly from the Folder stage to the XML Input stage and see how that helps. Make sure you are using the URL / File path option.

[waklook]

Hi,

Craig:
for using the URL / File path option i missed to choose YES for fully qualified option in folder stage, now it's working,
regarding your advise i'm going to do that and i will let you all know the result.

thanks guys.

[chulett]

for using the URL / File path option i missed to choose YES for fully qualified option in folder stage, now it's working

Good point and something I'll remember to mention from now on in this regard.

[waklook]

Hi guys;


Craig:

this is what happen with me when i run the job for only 10,000 xml file:

using the old job using 'XML DOCUMENT' option and
using your advise 'URL/FILE PATH' the load time was very close (arount 52Min),so i tryed
using LinkPart&LinkColl with 8 shared Containers(containing only the xml input stage) plus using your advise 'URL/FILE PATH' the load time was around 14Min and it seems huge difference, i will go for the second one unless one of you guys have better idea.

i used 8 shared containers bcz we have 8CPUs.

Many thanks for all of you guys specialy for those who gave me the advises.