PX Jobs on N Nodes: Inconsistent Output

[Inquisitive]

I am facing the following problem in one of the parallel jobs that I run using more than 1 node:
(The job contains only Input File sets, Output Seq files, Join Stages and Basic Transformer stages)

1) When the input to a job is changed after successfully running it several times with the previous input, the output for the run with the new input contains some data of the previous input !! (A new output file gets created every time the job is run as the stage is in overwrite mode)

2) As these jobs include a basic transformer, I tried increasing the value of environment variable DSIPC_OPEN_TIMEOUT. There was no change in the behaviour.

Here's what happens: Basic Transformer ---> Output Seq file. The monitor in director shows 68 output records from Transformer but the i/p to seq file shows 125 records.

Did u face something similar ?

SOS
Ravi Mohan


The above problem seems to be a memory related one. Any solutions/suggestions ?

[ray.wurlod]

The row count discrepancy might be accounted for if there were internal end-of-line characters in the rows output from the Transfomer stage. These rows would, properly, be seen as multiple lines in a text file.
Just a thought.