sort stage followed by remove duplicates stage
Posted: Wed Oct 01, 2014 8:19 pm
Hi,
we are using sort stage followed by remove duplicates stage in a datastage job.
Hash partioning done on col1,col2,col3 and sorting done on col1,col2 col4 in sort stage. Now in remove duplicates stage removing duplicates on col1,col2,col3. Retaining the first row in remove duplicates stage.
Remove duplicates stage is not working fine. Its once selecting the first row or the last row.
The query is, is it mandatory that the rows having duplicates be side by side for remove duplicates to retain the correct row.
Thanks.
we are using sort stage followed by remove duplicates stage in a datastage job.
Hash partioning done on col1,col2,col3 and sorting done on col1,col2 col4 in sort stage. Now in remove duplicates stage removing duplicates on col1,col2,col3. Retaining the first row in remove duplicates stage.
Remove duplicates stage is not working fine. Its once selecting the first row or the last row.
The query is, is it mandatory that the rows having duplicates be side by side for remove duplicates to retain the correct row.
Thanks.