i.e.
For this input data...
Key1 DateKey Value
1 2007-01-04 A
1 2007-01-03 B
1 2007-01-02 B
1 2007-01-01 A
I want the following output
Key1 DateKey Value
1 2007-01-04 A
1 2007-01-02 B
1 2007-01-01 A
i.e I've removed the 3rd record from the input set, because its got the same Value as the row next to it (when sorted by Key1 and DateKey).
I've coded this by sorting the data on Key1 and DateKey, and then passing it through a remove duplicates - with the duplicate keys being Key1 and Value.
So, the keys I use in my remove duplicates stage are not the same as the sort keys.
This seems to work ok, but I get the following message
Remove_Duplicates_239: When checking operator: User inserted sort "srt_ForRecEdt" does not fulfill the sort requirements of the downstream operator "Remove_Duplicates_239"
Is there a way to stop it producing this warning? Can you set something to make the remove duplicates think its sorted as it wants?
I have tried to code it in other ways which do not produce this warning, but the other ways seem to be slower (and more complicated).
Thanks