Remove Duplicates not removing duplicates

[pillip]

hi,

I have an issue with the remove duplicates stage in the job. It is not removing the duplicates based on the key, even though identical values are coming in the key.
Could you please let me know why this could happen.



Thanks

[chulett]

Is your data sorted properly?

[kwwilliams]

Two ways I can think of:

1. The data is not hash partitioned in a manner that would have the records on the same partition.
2. The data is not sorted properly. The remove duplicate stage is removing duplicates when they are sorted by the key, essentially removing duplicates that are located one after another.

So is the data sorted and hash paritioned correctly?

[ray.wurlod]

Keith has it. Your data are not partitioned correctly or not sorted correctly or both.

[chulett]

Figured I'd start with sorted and go from there.

[pillip]

Figured I'd start with sorted and go from there.

Should the data be sorted and hash partitioned or just sorted?


Thanks

[chulett]

Unless you are running on a single node, you need both.