How to remove duplicates in Datastage server edition 8

[sujaoschin]

How to remove the duplicate records in Datastage server edition jobs? Which stage I need to use? Is there any stage like 'remove duplicates' like in parallel extender?

[ray.wurlod]

Write them to a hashed file using the relevant keys. Last one in wins. The Hashed File stage implements destructive overwrite of keys.

Or use a Transformer stage with stage variables to detect change (or lack of) in sorted input.

[arshi]

Hi,
In Serveredition 8 no special stage for remove duplicates.
To remove the duplicates on column by using the stage variables by,
SvCurr=SvPre
link1.col1=SvCurr
If SvPre=SvCurr then @FALST else @TRUE .

Put this result in one more stage variable.
and Place this stage variable in the constraint.

Before this you need to sort the data bases on the column by using Sort stage.

Regards,
Arshi

[david_sol_llaven]

How to remove the duplicate records in Datastage server edition jobs? Which stage I need to use? Is there any stage like 'remove duplicates' like in parallel extender?

As previously written, write to a Hash File. Or you can use an Aggregator Stage.

HTH

[rameshrr3]

Yu can use RANK /ROW_NUMBER functions if you are extracting from Oracle database, refer Oracle SQL Guide- freely downloadable from OTN. Similar database functions may exist for others.

[sujaoschin]

I have used the Hash file and removed the duplicates.
Thank you very much