Remove Duplicate Warning

dspxlearn · Post by **dspxlearn** » Fri Nov 18, 2005 6:59 am

Hi all,

When i am trying to remove duplicates from the iput link of the remove duplicate stage, it is giving an warning as,

Remove_Duplicates_367: When checking operator: User inserted sort "Remove_Duplicates_367.DSLink358_Sort" does not fulfill the sort requirements of the downstream operator "Remove_Duplicates_367"

I was using Hash partitioning and enabled Sorting option as 'Perform Sort'

What might be the possible problem...

bgs · Post by **bgs** » Fri Nov 18, 2005 2:03 pm

when you use different keys for sorting and for removing duplicates you get this warning

kumar_s · Post by **kumar_s** » Fri Nov 18, 2005 9:00 pm

Hi,
Even the sorting order does the mater!!!
The key used in sort and the Remove Dulicates should match exactly.
Also maintain the same partiton in the Remove Duplicates stage.

-kumar

ray.wurlod · Post by **ray.wurlod** » Sat Nov 19, 2005 1:56 am

The reason that partitioning must be the same is to guarantee that any duplicates occur on the same processing node.

The reason that sorting should occur is so that least memory can be consumed - once the sort key value changes, the stage can be certain that there will be no more matches against this value and can quickly discard any rows from the other input that share this key value.