remove duplicates

[vamsipx]

hi all,

i am knew to this environment and i want to mknow how i can remove duplicates in transformer stage.

[Maveric]

It can be done in transformer stage using stage variables. But is there any specific reason for you to do this in the transformer stage when you have a Remove Duplicates stage?

In both the transformer or remove duplicates cases you will have to hash partition on the fields on which you are removing duplicates and also sort the data on the same fields.

[DSRajesh]

hi all,

i am knew to this environment and i want to mknow how i can remove duplicates in transformer stage.

You can eliminate duplicates using stage variables in transformer stage .
Write logic tochk equality of rows.

Let me know if any queries

Regards
Rajesh Devabhaktuni

[Minhajuddin]

As Maveric says, Using a Remove Duplicates stage is the easiest.
You don't even have to change the partitioning(Set them to Auto which is the default and Datastage takes care of partitioning).

If there is a specific reason like capturing rejects or something else, you can always remove duplicates by using Sort(with create key change to true) and a transformer where you reject if the key change is 0.

IHTH