how to load in to target files

4friends · Post by **4friends** » Thu Nov 22, 2007 12:42 am

i have a reguirement like in a source file data is having duplicates.

i need to store unique data to one target file and duplicates to another file.

how can we implement?

Maveric · Post by **Maveric** » Thu Nov 22, 2007 12:57 am

Investigate "CreateKeyChageColumn" in Sort stage. It gives a 1 for the first record and 0 for the duplicates. Use a filter on this field to get unique and duplicates in separate files.

Mayur Dongaonkar · Post by **Mayur Dongaonkar** » Thu Nov 22, 2007 1:01 am

Source File ----> Copy Stage -----> Sort Stage ---> Remove dulicate ---> Target File1
|
|
-----------> Sort Stage ---> Agg Stage ---> Filter ( Count > 1 ) ---> Target File12