Filter on Max Date ?

[bond88]

Hi,
I want to select unique records based on max date. Could you please suggest a best approach to achieve the below output?

Input:

ID--Date

1---10/11/2011
1---12/08/2005
1---01/15/2012
2---02/18/2010
2---03/04/2013

Output:

ID---Date

1---01/15/2012
2---03/04/2013

27-30 million records at input.

Thanks,

[bond88]

I am using sort stage and then remove duplicate stage to implement above logic. Is there any better way to get this done? Suggestion please.

[ray.wurlod]

Not really.

You could do the filtering in a Tranformer stage, using last record in group detection, but a Remove Duplicates is entirely adequate. Either approach requires data sorted by ID and by date, and partitioned by ID.

[chulett]

Aggregation. Not saying it's better but it is another method.

And what you have is not a "workaround" it is a resolution and I am marking it as such.