Remove Duplicates from target File

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
monaz
Participant
Posts: 98
Joined: Sat Aug 23, 2008 3:14 am

Remove Duplicates from target File

Post by monaz »

I need to remove duplicates for example

I am gettin this as the output
aaa bbb ccc
aaa bbb ccc
aaa ddd fff
aaa ddd fff

Actuall output file should look like
aaa bbb ccc
aaa ddd fff

I have used the following stages
database1->Hash file
-
-
-
database2-> Sorter-> Transformer-> OUTPUT File..

Still i am not able to remove duplicate. Please advice me on this.
As i am new to datastage and i have saw some post specifies use Use Stage variable.

But how to work on stagevariable on this. or Advice some other solution
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Method 1 - write to hashed file with all columns as key then read from that hashed file.

Method 2 - sort data then set up a pair of stage variables for each column, one to detect changes and one, executed after the first, to "remember" the value from the previous row.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply