Remove duplicates using Transformer Stage

kashif007 · Post by **kashif007** » Tue Sep 30, 2008 8:52 am

Is it possible to remove duplicates by using the Transformer stage ? If yes then how can we accomplish that ? I was thinking to sort the data using sort stage and write a logic in the transformer stage variable and constraint to stream out the unwanted duplicate data and retain only one of the many duplicate records. Am I correct ?

jaybee223 · Post by **jaybee223** » Tue Sep 30, 2008 9:03 am

Why dont you use the "Remove Duplicates" stage?

ArndW · Post by **ArndW** » Tue Sep 30, 2008 9:07 am

If the data is sorted and partitioned on the column with the duplicates, then use two stage variables, "IsDup" and "LastValue" derived via "IF In.ColumnName=LastValue THEN 1 ELSE 0" and "In.ColumnName" respectively, the constraint would be "IsDup=0" to only pass on non-duplicate rows.

ray.wurlod · Post by **ray.wurlod** » Tue Sep 30, 2008 3:41 pm

Generate a Key Change column in the Sort stage and apply a constraint in the Transformer stage that this column has the value 1. Or simply specify a unique sort.

kashif007 · Post by **kashif007** » Tue Sep 30, 2008 4:14 pm

Thanks Everybody.

DSXchange

Remove duplicates using Transformer Stage

Remove duplicates using Transformer Stage

Re: Remove duplicates using Transformer Stage