DSXchange

Hello all,
I have an existing job as

dataset stage --> transformer stage --> 5 different dataset stages

I need to filter out the dupicate recs from one of the 5 output links in transformer stage to dataset stage. Can I do it without having remove duplicates stage between transformer stage and dataset stage.

Thanks
Psrrepa

You can include a link sort to remove duplicates. But that will not be advisable.

Why do you NOT need a remove duplicate stage ?

What other way did you have in mind? You can do it either using the stage Or directly on the link.

I know we can sort on the link.. but how are we going to remove the duplicates in Transformer stage

Stage variables and Group Change Detection techniques. An exact search here for that phrase should turn up the details.

Or you can have a unique sort on the input link of the Transformer stage.

That's still "in the Transformer stage".