I used a sort stage with hash partionning and after sorting the data , i used remove duplicates stage with same partition.
I am encountering the error as
"User inserted sort "LabResultMem_srt" does not fulfill the sort requirements of the downstream operator "LabResultMem_rdup"
Note:
I have not got the warning when i tried to run this job under 4 partioned nodes, But when i tried running the job with 8 node partitions , i am getting this error.
I searched under this topic,but i have not got answers in relation with no of partitions.is there any problem with no of partitions...?
No of records processed - 300 million records.
User inserted sort...
Moderators: chulett, rschirm, roy
User inserted sort...
Prem R.
-
- Participant
- Posts: 135
- Joined: Tue Aug 14, 2007 4:27 am
- Location: Mumbai
Re: User inserted sort...
prematram wrote:I used a sort stage with hash partionning and after sorting the data , i used remove duplicates stage with same partition.
I am encountering the error as
"User inserted sort "LabResultMem_srt" does not fulfill the sort requirements of the downstream operator "LabResultMem_rdup"
Note:
I have not got the warning when i tried to run this job under 4 partioned nodes, But when i tried running the job with 8 node partitions , i am getting this error.
I searched under this topic,but i have not got answers in relation with no of partitions.is there any problem with no of partitions...?
No of records processed - 300 million records.
First of all why you need to use remove duplicate stage .. you can remove duplicates in sort stage it self ( by setting Allow Duplicates to FALSE)
2nd if you have used join and merge stage Set the environment variable APT_NO_SORT_INSERTION to true ...
Hope it will work !!!
Thanks
Swapnil
"Whenever you find whole world against you just turn around and Lead the world"
Swapnil
"Whenever you find whole world against you just turn around and Lead the world"
Still having one doubt
Hi,
Thanks for your response.
But same job when i tried running with config referring 4 nodes, it hasa run successfully. But the problem araised only when we tried running with config file pointing to 8 nodes.
The prevailing problem is anyway related to that...??
Thanks for your response.
But same job when i tried running with config referring 4 nodes, it hasa run successfully. But the problem araised only when we tried running with config file pointing to 8 nodes.
The prevailing problem is anyway related to that...??
Prem R.
-
- Participant
- Posts: 135
- Joined: Tue Aug 14, 2007 4:27 am
- Location: Mumbai
Re: Still having one doubt
prematram wrote:Hi,
Thanks for your response.
But same job when i tried running with config referring 4 nodes, it hasa run successfully. But the problem araised only when we tried running with config file pointing to 8 nodes.
The prevailing problem is anyway related to that...??
Well.. the above warning is coming for which stage...
also provide your design in detail?
Thanks
Swapnil
"Whenever you find whole world against you just turn around and Lead the world"
Swapnil
"Whenever you find whole world against you just turn around and Lead the world"
-
- Premium Member
- Posts: 301
- Joined: Thu Jul 14, 2005 10:27 am
- Location: Melbourne, Australia
- Contact:
I know it's obvious but you ARE sorting and de-duping on the same keys, aren't you?
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
What partitioning is specified on the input (a) of the Sort stage and (b) of the Remove Duplicates stage? Are there any stages between these two?
What are your specified Sort keys? What are your specified Remove Duplicate keys?
What are your specified Sort keys? What are your specified Remove Duplicate keys?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.