In the Sort output I need get record with minimum time diff

[kollurianu]

Hi All,

I 've the data in following way

field 5 is time difference in seconds.

I've sorted and hash partioned the output with fields 2,3,4,5 and would be as below


x1, a,b,c, 10

x2, a,b,c,10

x3, a,b,c,30

x4, a,b,c,40

x5, a1,b1,c1,5

x6,a1,b1,c1,10

x6,a1,b1,c1,15

From the above output I need get the following records with minimum time time difference and key fields being 2,3,4.

x1, a,b,c, 10
x5, a1,b1,c1,5

How do I acheive this? Any thoughts greatly appreciated.
Thanks in advance.

[kwwilliams]

Remove duplicate stage, duplicate to retain = first

[kandyshandy]

Hash partition should happen on fields 2,3 and 4 only. Field 5 should be used for sorting (asc) only. Then there are many ways to get the first record based on fields 2,3 and 4.