DSXchange

I used a sort stage with hash partionning and after sorting the data , i used remove duplicates stage with same partition.

I am encountering the error as
"User inserted sort "LabResultMem_srt" does not fulfill the sort requirements of the downstream operator "LabResultMem_rdup"

Note:

I have not got the warning when i tried to run this job under 4 partioned nodes, But when i tried running the job with 8 node partitions , i am getting this error.

I searched under this topic,but i have not got answers in relation with no of partitions.is there any problem with no of partitions...?

No of records processed - 300 million records.

prematram wrote:I used a sort stage with hash partionning and after sorting the data , i used remove duplicates stage with same partition.

I am encountering the error as
"User inserted sort "LabResultMem_srt" does not fulfill the sort requirements of the downstream operator "LabResultMem_rdup"

Note:

I have not got the warning when i tried to run this job under 4 partioned nodes, But when i tried running the job with 8 node partitions , i am getting this error.

I searched under this topic,but i have not got answers in relation with no of partitions.is there any problem with no of partitions...?

No of records processed - 300 million records.

First of all why you need to use remove duplicate stage .. you can remove duplicates in sort stage it self ( by setting Allow Duplicates to FALSE)

2nd if you have used join and merge stage Set the environment variable APT_NO_SORT_INSERTION to true ...

Hope it will work !!!

Hi,

Thanks for your response.

But same job when i tried running with config referring 4 nodes, it hasa run successfully. But the problem araised only when we tried running with config file pointing to 8 nodes.

The prevailing problem is anyway related to that...??

prematram wrote:Hi,

Thanks for your response.

But same job when i tried running with config referring 4 nodes, it hasa run successfully. But the problem araised only when we tried running with config file pointing to 8 nodes.

The prevailing problem is anyway related to that...??

Well.. the above warning is coming for which stage...
also provide your design in detail?

I know it's obvious but you ARE sorting and de-duping on the same keys, aren't you?

What partitioning is specified on the input (a) of the Sort stage and (b) of the Remove Duplicates stage? Are there any stages between these two?

What are your specified Sort keys? What are your specified Remove Duplicate keys?