remove duplicates using Transformer
Posted: Thu Sep 10, 2009 11:21 am
Hi,
I am trying to remove duplictes only using Transformer. In the input tab of transformer I am doing hash partitioning and doing perform sort. On one of key I am doing sort partitioning and then on other key I did just sorting Asc and then other column I did sorting Descending.
Key1 (Sorting,partitioning)
Key2(Sorting Asc)
column1(sorting Des)
Then I checked the Stable and Unique check box in the tab expecting to retain the first record when there are duplictes, but I don't see any duplicate records getting dropped.
Could any one please let me know how this stable and unique works because in documentation it is mentioned that if I check both stable and Unique the first duplicte record will be retained. Please let me know if I am missing anything or any other postes regardign this.
Any help would be really appreciated.
I am trying to remove duplictes only using Transformer. In the input tab of transformer I am doing hash partitioning and doing perform sort. On one of key I am doing sort partitioning and then on other key I did just sorting Asc and then other column I did sorting Descending.
Key1 (Sorting,partitioning)
Key2(Sorting Asc)
column1(sorting Des)
Then I checked the Stable and Unique check box in the tab expecting to retain the first record when there are duplictes, but I don't see any duplicate records getting dropped.
Could any one please let me know how this stable and unique works because in documentation it is mentioned that if I check both stable and Unique the first duplicte record will be retained. Please let me know if I am missing anything or any other postes regardign this.
Any help would be really appreciated.