How to read DISTINCT from flat file ?

swades · Post by **swades** » Wed Jan 31, 2007 1:48 pm

Hi,

How to read DISTINCT sring from flat file ?
Can any one help ? Please ?

DSguru2B · Post by **DSguru2B** » Wed Jan 31, 2007 1:50 pm

Pass it through a hashed file, the last duplicate will be retained. Or if you want the first one, pass it through aggregator and retain the first record, grouped on the column you want distinct on.

swades · Post by **swades** » Wed Jan 31, 2007 1:59 pm

Thanks for Response

I am using PX so can you please tell me what properties to set in Agrregator Stage ?

Thank

DSguru2B · Post by **DSguru2B** » Wed Jan 31, 2007 2:01 pm

Ouch, completely overlooked that. Sorry about that. Look into the Remove Duplicate Stage. That will get you what you want.

swades · Post by **swades** » Wed Jan 31, 2007 2:09 pm

Thanks a lot

ray.wurlod · Post by **ray.wurlod** » Wed Jan 31, 2007 4:21 pm

Remove Duplicates requires sorted input. In increasing order of inefficiency, try the following techniques.

Sort within DataStage, specifying "Unique"

Filter through sort -u

Remove Duplicates stage

Access through ODBC and specify DISTINCT in the SQL statement.