Remove duplicates using Transformer

siva7143 · Post by **siva7143** » Tue Sep 02, 2008 1:08 am

Hi ,

how to remove duplicate rows using transformer stage?

Thanks in advance,
Siva Kumar N

mahadev.v · Post by **mahadev.v** » Tue Sep 02, 2008 1:16 am

You have a stage called remove duplicates to do this for you. Any specific reason for wanting to using transformer?

ray.wurlod · Post by **ray.wurlod** » Tue Sep 02, 2008 1:47 am

If you search the forum you will find that this question has been answered before. Searching means that you can get your answer faster than posting and waiting for someone to answer.

singhald · Post by **singhald** » Tue Sep 02, 2008 2:42 am

hello Mahadev,

it is very tricy to remove duplcicate records using transformer stage,

to do that , you need to use two stages,

1) Sort Stage : you need to sort your indatcoming data based on key attributes on which you want to remove duplicate records,
also enable KeyChange options inside the sort stage.

2) in transformer: use the KeyChange coulmn in the constraint.

KeyChange =1

if you use this , in output you will find all unique records.

Please mark Resolved this post if you able to remove duplicate records using transformer.

Regards,

fareeda_b · Post by **fareeda_b** » Wed Dec 03, 2008 4:42 am

Hi Deepak,

In sort stage will have key change column right but we don't have option as key change=1 in transformer so how will remove dupilcates in transformer .
my logic is
take 3 stagevaraibles
st1 =st3
st2= if stg1=stg3 then duplictae else Not-duplicate
st3= column name

if i'm wrong this please correct me .

thanks in advance
fareeda

fareeda_b · Post by **fareeda_b** » Wed Dec 03, 2008 4:44 am

Hi Deepak,

In sort stage will have key change column right but we don't have option as key change=1 in transformer so how will remove dupilcates in transformer .
my logic is
take 3 stagevaraibles
st1 =st3
st2= if stg1=stg3 then duplictae else Not-duplicate
st3= column name

if i'm wrong this please correct me .

thanks in advance
fareeda

battaliou · Post by **battaliou** » Wed Dec 03, 2008 6:09 am

Your logic is fine

singhald · Post by **singhald** » Thu Dec 04, 2008 4:17 am

i have a question, why can't you use KeyChange=1 in transformer instead of using stage variable. is there any problem.

ray.wurlod · Post by **ray.wurlod** » Thu Dec 04, 2008 1:47 pm

Only that there was no mention of a Sort stage in the original question. Without a Sort stage you don't get a KeyChange column. But it would be a viable solution.

DSXchange

Remove duplicates using Transformer

Remove duplicates using Transformer

Need clarification in Transformer

Need clarification in Transformer

Re: Need clarification in Transformer