How to Select distinct records from Files

[anil kumar]

Hi thanks for ur quick response...
but still i have one doubt that ..in that scenario if u have duplicate records it will work..as u explained in the example what happens to first record itself..the condition will not satisfy..right....for the second record it is ok...

please clarify me ....

thanks

[DataStageCnu]

Hi thanks for ur quick response...
but still i have one doubt that ..in that scenario if u have duplicate records it will work..as u explained in the example what happens to first record itself..the condition will not satisfy..right....for the second record it is ok...

please clarify me ....

thanks

You can do it by using Hash stage or.. by loading into any database stage and then select distince.

Is it make sence...Let me know, if you have any questions

[Tasneem]

If u r sorting the records using the Sort stage,use the cluster key change feature to remove duplicates.
This works on the same logic as the StgVariables.

[ravij]

Hello Kumar

Where can difine the Sort command?


thanks in advance
RK

[ray.wurlod]

Anywhere you like, but typically in the Input Values field when using ExecSH as a before-job or before-stage subroutine.

[yaminids]

Anil,

When you define the Stage variables assign them an initial value which wont appear in the input data

Yamini

[anil kumar]

thank you very much...

it is working fine.

[sumeet]

I agree with this logic that Stage variable can be used to remove the duplicate files but stage variable stores the previous field value. so its better to have sort the data and club the duplicate recs together and then use the stage variable concept.