Page 1 of 1
how remove duplicate in transformer
Posted: Mon Oct 18, 2010 4:48 am
by sureshbabu
hi
can any body sugg me how do romoveduplecate in transformar stage useing stage varible
regards
sure
Posted: Mon Oct 18, 2010 6:01 am
by ArndW
2 Stage Variables:
Code: Select all
svDuplicate IF In.Key=svLastKey THEN 1 ELSE 0
svLastKey In.Key
constraint "svDuplicate=0"
Posted: Mon Oct 18, 2010 6:26 am
by chulett
Or, exactly the same but my preference would be:
Code: Select all
svDuplicate In.Key=svLastKey
svLastKey In.Key
Constraint "Not(svDuplicate)"
Posted: Mon Oct 18, 2010 6:51 am
by ArndW
Craig, I agree with you - I would program it that way but expanded the logic to make the statement more understandable.
Posted: Mon Oct 18, 2010 6:52 am
by mk_ds09
You can use the following logic to remove duplicates using the Transformer Stage, however for that you need to have data sorted.
Before using the transformer Stage, use Sort Stage to sort the data.
In Transformer Stage, take the
FirstRecordVar=SecondRecordVar
IncomingLnk.Value=FirstRecordVar
Then in constraint you can put constraint like
FirstRecordVar<>SecondRecordVar
This will ensure that duplicates records are not passed to the next stage.
Hope this helps !
Posted: Mon Oct 18, 2010 6:53 am
by chulett
ArndW wrote:Craig, I agree with you - I would program it that way but expanded the logic to make the statement more understandable.
Of course and I figured as much... just couldn't resist the perfect setup.
Posted: Mon Oct 18, 2010 9:35 am
by agpt
mk_ds09 wrote:You can use the following logic to remove duplicates using the Transformer Stage, however for that you need to have data sorted.
Before using the transformer Stage, use Sort Stage to sort the data.
In Transformer Stage, take the
FirstRecordVar=SecondRecordVar
IncomingLnk.Value=FirstRecordVar
Then in constraint you can put constraint like
FirstRecordVar<>SecondRecordVar
This will ensure that duplicates records are not passed to the next stage.
Hope this helps !
Hi Sorry to interrupt but how do you achieve :
FirstRecordVar=SecondRecordVar
?
What exact syntax is used for the same?
Re: how remove duplicate in transformer
Posted: Mon Oct 18, 2010 9:20 pm
by galmeida
sureshbabu wrote:hi
can any body sugg me how do romoveduplecate in transformar stage useing stage varible
regards
sure
Why not just use the de-duplicate stage?