remove duplicates

sunitha_cts · Post by **sunitha_cts** » Mon Jul 27, 2009 3:27 am

HI,

Instead of using Remove duplicate stage , is they any other way to remove duplicates from a table.

Thanks
sunitha

ArndW · Post by **ArndW** » Mon Jul 27, 2009 3:38 am

Yes, there are other ways. All involve sorting the data. The Sort can remove duplicate rows and you can use stage variables in a transform stage to detect duplicate column values and drop those rows.

algfr · Post by **algfr** » Mon Jul 27, 2009 3:51 am

sunitha_cts wrote:HI,

Instead of using Remove duplicate stage , is they any other way to remove duplicates from a table.

Thanks
sunitha

If you query from a DB, you can also use the DISTINCT keyword in the query sentence. This is helpful to restrict as early as possible the number of rows that are retrieved.

ShaneMuir · Post by **ShaneMuir** » Mon Jul 27, 2009 4:00 am

It will also depend if you want to capture those duplicates for reporting purposes.

asnrece1 · Post by **asnrece1** » Mon Jul 27, 2009 4:07 am

by using the sort stage and putting the Cluster index column properety then apply the filter or transformer then wil send the first records and filter the duplicated records. This will be useful in the middle of the job.

nagarjuna · Post by **nagarjuna** » Mon Jul 27, 2009 4:46 am

you can use sort stage and unique option within it .

sunitha_cts · Post by **sunitha_cts** » Mon Jul 27, 2009 5:22 am

'unique' in which stage property

nagarjuna · Post by **nagarjuna** » Mon Jul 27, 2009 5:35 am

i think we have an option unique in sort stage

sunitha_cts · Post by **sunitha_cts** » Mon Jul 27, 2009 5:38 am

No I guess

chrisjones · Post by **chrisjones** » Mon Jul 27, 2009 5:50 am

Hi Sunita,

In the sort stage keep ALLOW DUPLICATES as False this will help you to remove duplicates.

Thanks,
Chris

sunitha_cts · Post by **sunitha_cts** » Mon Jul 27, 2009 5:55 am

Thanks Cris

aladap12 · Post by **aladap12** » Tue Jul 28, 2009 9:44 am

sunitha_cts wrote:Thanks Cris

Yes, we can remove duplicates using sort stage and also every output link..but make sure that your partition type is other that auto and select perform sort and unique.. that will give you unique records..

hope this makes sense..
krishna

ray.wurlod · Post by **ray.wurlod** » Tue Jul 28, 2009 4:37 pm

Sorting is specified on input links, not on output links.

SBSA_DW · Post by **SBSA_DW** » Tue Aug 11, 2009 5:45 am

aladap12 wrote:
sunitha_cts wrote:Thanks Cris
Yes, we can remove duplicates using sort stage and also every output link..but make sure that your partition type is other that auto and select perform sort and unique.. that will give you unique records..

hope this makes sense..
krishna

Thanks for the info , but do you trap those duplicates records from transformer stage.

DSXchange