Page 1 of 1

REMOVE DUPLICATES IN TRANSFORMER STAGE

Posted: Wed Feb 02, 2011 1:36 am
by hemanth12
Our requirement is

DB-->TR-->Seq or DB(rest of the records in target)
|
Rejected file(seq)(in this i need duplicates in rejected file rest of the records in target)

plz give me the briefly explanation.if it is posible with screen shots to send me..
my mail-id:hemanthrao.k@gmail.com

Thanks in advance...

Posted: Wed Feb 02, 2011 3:34 am
by ray.wurlod
Set up a unique sort on the input link to the Transformer stage.

But it's a silly requirement to want to do this in the Transformer stage.

REMOVE DUPLICATES IN TRANSFORMER STAGE

Posted: Wed Feb 02, 2011 4:30 am
by hemanth12
thanks for reply....

but my requirement is
Don't use any of the part ions technic and sort ...unique

I need duplication records in one file and remaining records in another file .using only for transformer stage..don't use any remove duplicate .in transformer don't use any of the partion technic ....


Regards
Hemanth

ray.wurlod wrote:Set up a unique sort on the input link to the Transformer stage.

But it's a silly requirement to want to do this in the Transformer stage. ...

Posted: Wed Feb 02, 2011 4:55 am
by ray.wurlod
Resist stupid requirements. You can't do it without sorted data. And that implies properly partitioned data.

Posted: Wed Feb 02, 2011 8:49 am
by Vidyut
Ray is right without sorted data u cant remove duplicates using transformer.
What you can do is sort the data on key and then use two stage variables to remove the duplicates.
one having the last value and one current and by comparing both u can remove the duplicates