Hi,
I want to select unique records based on max date. Could you please suggest a best approach to achieve the below output?
Input:
ID--Date
1---10/11/2011
1---12/08/2005
1---01/15/2012
2---02/18/2010
2---03/04/2013
Output:
ID---Date
1---01/15/2012
2---03/04/2013
27-30 million records at input.
Thanks,
Filter on Max Date ?
Moderators: chulett, rschirm, roy
Filter on Max Date ?
Bhanu
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Not really.
You could do the filtering in a Tranformer stage, using last record in group detection, but a Remove Duplicates is entirely adequate. Either approach requires data sorted by ID and by date, and partitioned by ID.
You could do the filtering in a Tranformer stage, using last record in group detection, but a Remove Duplicates is entirely adequate. Either approach requires data sorted by ID and by date, and partitioned by ID.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.