Data inconistency with Transfomer n paralle
Posted: Wed May 19, 2010 11:13 pm
Hi,
I am generating a unique identifcation number (UID) in a transfomer.
The job design like
Dataset---->Sort ---> Transformer---> Target.
I am sorting on columns A,B and generating keychange indicator.
In the transformer,When the keychange indicator is 1 , the UID s incremented else the previous value is generated using stage variables.
When I run the transfomer in parallel, duplicates UID are created, while when I run the transformer in sequential, unique UIDS are generated.
The transfomer,sort stage are partitioned on keys A,B.
Running the transfomer in sequentail will hamper the performance, any suggestin on this.
I am generating a unique identifcation number (UID) in a transfomer.
The job design like
Dataset---->Sort ---> Transformer---> Target.
I am sorting on columns A,B and generating keychange indicator.
In the transformer,When the keychange indicator is 1 , the UID s incremented else the previous value is generated using stage variables.
When I run the transfomer in parallel, duplicates UID are created, while when I run the transformer in sequential, unique UIDS are generated.
The transfomer,sort stage are partitioned on keys A,B.
Running the transfomer in sequentail will hamper the performance, any suggestin on this.