How to delete all occurences of duplicates from seq file

[kmsekhar]

Hi All,
I need to delete all occurences of duplicates from Sequential File as Input...

Input.txt
RNO|NAME|GRP|AGE
1|S|10|52
2|X|10|52
3|Y|20|52
1|Z|10|52
4|A|30|52

My Desired Output should be:

RNO|NAME|GRP|AGE
2|X|10|52
3|Y|20|52
4|A|30|52

I acheived using Shell Script & Loading into DB & by using having count(1)> 1..

still is there any better we can achieve this using any of the parallel stages..

Thanks in Advance

Regards,
Sekhar

[vinothkumar]

Use remove duplicates stage with GRP and AGE as Key fields.

[anbu]

Use aggregator to find count on RNO and remove rows having count greater than 1. Join this result with your input to remove the duplicates

[siauchun84]

I assume that you take the highest RNO number as you survival record if duplication occur in GPR and AGE.

In this case, I will sort the records by GPR, AGE then follow by RNO. After that, use the remove duplicate stage with GPR and AGE as key and retain first/last (depends on you sort the RNO asc or desc).

[kmsekhar]

I want to delete all occurences....

[ray.wurlod]

This is a classic fork join design as others have indicated. Downstream of the Join stage use a Filter stage or Transformer stage to allow past only those records that have a count of 1.