Remove duplicates-Unique only

[Ush]

Hi

Is it possible to use remove duplicates stage to extract unique records.I am requirement analysis phase.I do not have DS parallel.

ex:
Emp id
1
1
2
3
3

only 2 should go.In the manual i could find only first and last options

Thanks

[AmeyJoshi14]

I think by remove duplicate you can not get the desired result...
You can achieve this by using Aggregator + Filter stage.

source ----> aggregator ( on Emp_id + Aggregation Type--Count Rows ) ---> filter ( count = 1 ) ---> sequencial(target)

[ray.wurlod]

Remove Duplicates stage will remove duplicates without needing any other stage.

You do, however, need to ensure that the data are key-partitioned and sorted on the keys being used to identify duplicates.