Avoid Dupicates in Transformer stage

[psreepa123]

Hello all,
I have an existing job as

dataset stage --> transformer stage --> 5 different dataset stages

I need to filter out the dupicate recs from one of the 5 output links in transformer stage to dataset stage. Can I do it without having remove duplicates stage between transformer stage and dataset stage.

Thanks
Psrrepa

[Sainath.Srinivasan]

You can include a link sort to remove duplicates. But that will not be advisable.

Why do you NOT need a remove duplicate stage ?

[ArjunK]

What other way did you have in mind? You can do it either using the stage Or directly on the link.

[psreepa123]

I know we can sort on the link.. but how are we going to remove the duplicates in Transformer stage

[chulett]

Stage variables and Group Change Detection techniques. An exact search here for that phrase should turn up the details.

[ray.wurlod]

Or you can have a unique sort on the input link of the Transformer stage.

That's still "in the Transformer stage".