Greetings everybody. I too have encountered similar problem to what yinyin has encounted...
My solution was... in between the active stages, (Mine I had a sorter to aggregator to transformers) I've placed a sequential file. With that the failure rate dropped dramatically! I have also turned off row buffer for my jobs.
What does the sequential file play in this? In doing so I have a strong impression that this whole error has something to do with buffering or memory allocation or caching or whatever for the unix server I am currently hosting the ds project.
Does anyone have a take in why this is so?
dsintbuf_getrow() - row has N columns when N expected
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Welcome aboard. :D
Jim's answer is correct.
There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.
Jim's answer is correct.
There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
dsintbuf_getrow() - row has N columns when N expected
I undestand now... thanks guys!ray.wurlod wrote:Welcome aboard. :D
Jim's answer is correct.
There seems to be a bug in the metadata handling when buffering is enabled, whether explicitly or implicitly. An intermediate sequential file defeats the purpose of buffering by forcing data to be touched down to disk (at "disk speed" rather than at "memory speed)". But the Sequential File stage does not use inter-process buffering, so can not manifest the symptoms reported in the subject line.
I think it's gotta do with the server I am hosting the project in I guess. I talked to one of my team mates and he did stress out that resources are somewhat lower than my other test servers.
Reckon either I get that up and running or I have to stick with a silly design with sequential files in between...
I know I lose parallel processing advantage but with the current condition... I don't have much choice.
Thanks for enlightening guys!
![Laughing :lol:](./images/smilies/icon_lol.gif)