Remove duplicates using Transformer
Moderators: chulett, rschirm, roy
Remove duplicates using Transformer
Hi ,
how to remove duplicate rows using transformer stage?
Thanks in advance,
Siva Kumar N
how to remove duplicate rows using transformer stage?
Thanks in advance,
Siva Kumar N
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If you search the forum you will find that this question has been answered before. Searching means that you can get your answer faster than posting and waiting for someone to answer.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
hello Mahadev,
it is very tricy to remove duplcicate records using transformer stage,
to do that , you need to use two stages,
1) Sort Stage : you need to sort your indatcoming data based on key attributes on which you want to remove duplicate records,
also enable KeyChange options inside the sort stage.
2) in transformer: use the KeyChange coulmn in the constraint.
KeyChange =1
if you use this , in output you will find all unique records.
Please mark Resolved this post if you able to remove duplicate records using transformer.
Regards,
it is very tricy to remove duplcicate records using transformer stage,
to do that , you need to use two stages,
1) Sort Stage : you need to sort your indatcoming data based on key attributes on which you want to remove duplicate records,
also enable KeyChange options inside the sort stage.
2) in transformer: use the KeyChange coulmn in the constraint.
KeyChange =1
if you use this , in output you will find all unique records.
Please mark Resolved this post if you able to remove duplicate records using transformer.
Regards,
Regards,
Deepak Singhal
Everything is okay in the end. If it's not okay, then it's not the end.
Deepak Singhal
Everything is okay in the end. If it's not okay, then it's not the end.
Need clarification in Transformer
Hi Deepak,
In sort stage will have key change column right but we don't have option as key change=1 in transformer so how will remove dupilcates in transformer .
my logic is
take 3 stagevaraibles
st1 =st3
st2= if stg1=stg3 then duplictae else Not-duplicate
st3= column name
if i'm wrong this please correct me .
thanks in advance
fareeda
In sort stage will have key change column right but we don't have option as key change=1 in transformer so how will remove dupilcates in transformer .
my logic is
take 3 stagevaraibles
st1 =st3
st2= if stg1=stg3 then duplictae else Not-duplicate
st3= column name
if i'm wrong this please correct me .
thanks in advance
fareeda
Thanks
Need clarification in Transformer
Hi Deepak,
In sort stage will have key change column right but we don't have option as key change=1 in transformer so how will remove dupilcates in transformer .
my logic is
take 3 stagevaraibles
st1 =st3
st2= if stg1=stg3 then duplictae else Not-duplicate
st3= column name
if i'm wrong this please correct me .
thanks in advance
fareeda
In sort stage will have key change column right but we don't have option as key change=1 in transformer so how will remove dupilcates in transformer .
my logic is
take 3 stagevaraibles
st1 =st3
st2= if stg1=stg3 then duplictae else Not-duplicate
st3= column name
if i'm wrong this please correct me .
thanks in advance
fareeda
Thanks
Re: Need clarification in Transformer
Your logic is fine
3NF: Every non-key attribute must provide a fact about the key, the whole key, and nothing but the key. So help me Codd.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Only that there was no mention of a Sort stage in the original question. Without a Sort stage you don't get a KeyChange column. But it would be a viable solution.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.