duplicates

laxmi_etl · Post by **laxmi_etl** » Thu Apr 26, 2007 3:38 pm

Hi,

I need a help on writing duplicate records to a file.

I know we have some duplicates in the source file,
when I use remove duplicate stage I found some duplicates are there in the source file.

But my question is there any way we can write those to a file.

Thanks

csrazdan · Post by **csrazdan** » Thu Apr 26, 2007 3:46 pm

You can also perform this activity using SORT stage. Add a SORT stage to you job design. and sort it based on your key. Add SORT stage property Create Key Change Column. This property will add a column to your output link. The value of this column is 1 for the first record in the sort record group else the value is 0.

Hope it helps........

sud · Post by **sud** » Thu Apr 26, 2007 3:49 pm

csrazdan wrote:You can also perform this activity using SORT stage. Add a SORT stage to you job design. and sort it based on your key. Add SORT stage property Create Key Change Column. This property will add a column to your output link. The value of this column is 1 for the first record in the sort record group else the value is 0.

Hope it helps........

yes ... the keychange = 0 are the duplicate records which can be filtered using a constraint.

swades · Post by **swades** » Thu Apr 26, 2007 5:45 pm

Give output from Sort Stage to Filter Stage in that you can specify KeyChange=1 in Where Clause and In Option set Output Rejects=True(stretch 1 reject link from Filter Stage) That way you will be collecting duplicates in Rejected Link

DSguru2B · Post by **DSguru2B** » Thu Apr 26, 2007 9:24 pm

...and you could have gotton that answer just by a simple search. A similar post was answered today.