Page 1 of 1

how remove duplicate in transformer

Posted: Mon Oct 18, 2010 4:48 am
by sureshbabu
hi

can any body sugg me how do romoveduplecate in transformar stage useing stage varible


regards
sure

Posted: Mon Oct 18, 2010 6:01 am
by ArndW
2 Stage Variables:

Code: Select all

svDuplicate          IF In.Key=svLastKey THEN 1 ELSE 0
svLastKey          In.Key
constraint "svDuplicate=0"

Posted: Mon Oct 18, 2010 6:26 am
by chulett
Or, exactly the same but my preference would be:

Code: Select all

svDuplicate        In.Key=svLastKey
svLastKey          In.Key
Constraint "Not(svDuplicate)"

Posted: Mon Oct 18, 2010 6:51 am
by ArndW
Craig, I agree with you - I would program it that way but expanded the logic to make the statement more understandable.

Posted: Mon Oct 18, 2010 6:52 am
by mk_ds09
You can use the following logic to remove duplicates using the Transformer Stage, however for that you need to have data sorted.

Before using the transformer Stage, use Sort Stage to sort the data.

In Transformer Stage, take the

FirstRecordVar=SecondRecordVar
IncomingLnk.Value=FirstRecordVar

Then in constraint you can put constraint like

FirstRecordVar<>SecondRecordVar

This will ensure that duplicates records are not passed to the next stage.

Hope this helps !

Posted: Mon Oct 18, 2010 6:53 am
by chulett
ArndW wrote:Craig, I agree with you - I would program it that way but expanded the logic to make the statement more understandable.
Of course and I figured as much... just couldn't resist the perfect setup. :wink:

Posted: Mon Oct 18, 2010 9:35 am
by agpt
mk_ds09 wrote:You can use the following logic to remove duplicates using the Transformer Stage, however for that you need to have data sorted.

Before using the transformer Stage, use Sort Stage to sort the data.

In Transformer Stage, take the

FirstRecordVar=SecondRecordVar
IncomingLnk.Value=FirstRecordVar

Then in constraint you can put constraint like

FirstRecordVar<>SecondRecordVar

This will ensure that duplicates records are not passed to the next stage.

Hope this helps !
Hi Sorry to interrupt but how do you achieve :

FirstRecordVar=SecondRecordVar
?

What exact syntax is used for the same?

Re: how remove duplicate in transformer

Posted: Mon Oct 18, 2010 9:20 pm
by galmeida
sureshbabu wrote:hi

can any body sugg me how do romoveduplecate in transformar stage useing stage varible


regards
sure
Why not just use the de-duplicate stage?