Page 1 of 1
dsintbuf_getrow() - row has N columns when N expected
Posted: Wed Jan 11, 2006 7:27 pm
by Nick_6789
Greetings everybody. I too have encountered similar problem to what yinyin has encounted...
My solution was... in between the active stages, (Mine I had a sorter to aggregator to transformers) I've placed a sequential file. With that the failure rate dropped dramatically! I have also turned off row buffer for my jobs.
What does the sequential file play in this? In doing so I have a strong impression that this whole error has something to do with buffering or memory allocation or caching or whatever for the unix server I am currently hosting the ds project.
Does anyone have a take in why this is so?
Posted: Wed Jan 11, 2006 8:18 pm
by jzparad
The sequential file would act as a buffer between the two stages but I'm guessing that unlike an IPC stage, the second process would not begin until the first process has completed. You therefore loose the parallel processing advantage.
Posted: Wed Jan 11, 2006 8:38 pm
by ray.wurlod
Welcome aboard. :D
Jim's answer is correct.
There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.
dsintbuf_getrow() - row has N columns when N expected
Posted: Wed Jan 11, 2006 8:52 pm
by Nick_6789
ray.wurlod wrote:Welcome aboard. :D
Jim's answer is correct.
There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.
I undestand now... thanks guys!
I think it's gotta do with the server I am hosting the project in I guess. I talked to one of my team mates and he did stress out that resources are somewhat lower than my other test servers.
Reckon either I get that up and running or I have to stick with a silly design with sequential files in between...
I know I lose parallel processing advantage but with the current condition... I don't have much choice.
Thanks for enlightening guys!
