remove duplicates

harryhome · Post by **harryhome** » Mon Oct 03, 2011 4:50 pm

I have 5 partitions and trying to get distinct records on a key column using remove duplicate stage. getting different number of record count each time I run job.

SURA · Post by **SURA** » Mon Oct 03, 2011 5:33 pm

SORT it!

What do you mean 5 partition?

DS User

ray.wurlod · Post by **ray.wurlod** » Mon Oct 03, 2011 5:39 pm

Is this the same question as this one?

prakashdasika · Post by **prakashdasika** » Mon Oct 03, 2011 7:12 pm

Do you mean 5 nodes ? link Sort the data in hash partion on the key. ascending or descending depends on the preference.

DSXchange

remove duplicates

remove duplicates

Re: remove duplicates