Remove duplicates using Transformer Stage
Moderators: chulett, rschirm, roy
Remove duplicates using Transformer Stage
Is it possible to remove duplicates by using the Transformer stage ? If yes then how can we accomplish that ? I was thinking to sort the data using sort stage and write a logic in the transformer stage variable and constraint to stream out the unwanted duplicate data and retain only one of the many duplicate records. Am I correct ?
Regards
Kashif Khan
Kashif Khan
Re: Remove duplicates using Transformer Stage
Why dont you use the "Remove Duplicates" stage?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Generate a Key Change column in the Sort stage and apply a constraint in the Transformer stage that this column has the value 1. Or simply specify a unique sort.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.