Remove duplicate Error

dspxlearn · Post by **dspxlearn** » Fri Nov 18, 2005 6:56 am

Hi all,

When i am trying to remove duplicates from the iput link of the remove duplicate stage, it is giving an warning as,

Remove_Duplicates_367: When checking operator: User inserted sort "Remove_Duplicates_367.DSLink358_Sort" does not fulfill the sort requirements of the downstream operator "Remove_Duplicates_367"

I was using Hash partitioning and enabled Sorting option as 'Perform Sort'

What might be the possible problem... :D

dspxlearn · Post by **dspxlearn** » Fri Nov 18, 2005 6:58 am

Sorry guys...This message should be given parallel jobs..

vmcburney · Post by **vmcburney** » Fri Nov 18, 2005 3:23 pm

The sort keys on the sort stage need to be the same as the remove duplicate keys. Better yet you can remove duplicates in the sort stage using one of the properties. From memory the only thing the remove duplicates gives you that isn't in the sort stage is a "Last" option.