dsintbuf_getrow() - row has N columns when N expected

Nick_6789 · Post by **Nick_6789** » Wed Jan 11, 2006 7:27 pm

Greetings everybody. I too have encountered similar problem to what yinyin has encounted...

My solution was... in between the active stages, (Mine I had a sorter to aggregator to transformers) I've placed a sequential file. With that the failure rate dropped dramatically! I have also turned off row buffer for my jobs.

What does the sequential file play in this? In doing so I have a strong impression that this whole error has something to do with buffering or memory allocation or caching or whatever for the unix server I am currently hosting the ds project.

Does anyone have a take in why this is so?

jzparad · Post by **jzparad** » Wed Jan 11, 2006 8:18 pm

The sequential file would act as a buffer between the two stages but I'm guessing that unlike an IPC stage, the second process would not begin until the first process has completed. You therefore loose the parallel processing advantage.

ray.wurlod · Post by **ray.wurlod** » Wed Jan 11, 2006 8:38 pm

Welcome aboard. :D

Jim's answer is correct.

There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.

Nick_6789 · Post by **Nick_6789** » Wed Jan 11, 2006 8:52 pm

ray.wurlod wrote:Welcome aboard. :D

Jim's answer is correct.

There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.

I undestand now... thanks guys!

I think it's gotta do with the server I am hosting the project in I guess. I talked to one of my team mates and he did stress out that resources are somewhat lower than my other test servers.

Reckon either I get that up and running or I have to stick with a silly design with sequential files in between...

I know I lose parallel processing advantage but with the current condition... I don't have much choice.

Thanks for enlightening guys!