Remove Duplicates

[kumar66]

Hi All,

I have to remove the duplicates and also need to capture the duplicate records.

I thought of doing with remove duplicate stage. But it didnt worked out.
Please advise how to do it.


Thanks & Regards,
Kumar66

[BugFree]

Do a search on "remove and capture the duplicate records" (Search for all terms). This has been answered a lot of times here.

[mcs_suman]

use sort stage with the option key change column which gives the output value as '0' for non-duplicates and '1' for duplicates.And then use filter stage to filter these records separately based on the key value.




source------>sort--->filter-------->target

Do a search on "remove and capture the duplicate records" (Search for all terms). This has been answered a lot of times here.

[kumar66]

Hi mcs_suman,

Thanks for your response.

I tried in a diffrent way .

source----->copy stage---->Lookup-----> Transformer-----> Target
|
v
Aggrator


I made a lookup on the row count. If the row count is more than 1 , I will reject the row.

Thanks & regards,
Kumar66