Hi,
I have a Datastage job design, like that or something like that :
INPUT --> SORT+HASH PARTITION --> TRANSFORMER(partition : SAME, no sort) --> LOOKUP (partition : SAME, no sort) --> AGREGATION (partition SAME, no sort)--> OUTPUT
First, I partition and sort data, and I want to keep the partition and the sort all along the job.
So, I used SAME partitionning and no sort in all stages.
and for the reference lookup i use entire partitionning.
But,
Because I use same partitionning, I have the following warnings :
In the transformer :"Transformer_128: Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."
In the lookup : "Lookup: Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."
So, I read post in the forum, and for the transformer warning it's a Datastage bug due to the use of the lookup logic in transformer. This bug is fix in the 7.6 version -> ok
but, it seem that the lookup warnings is not a bug. So my questions (finally) are :
- Does this warning is a problem for the logic of my job ?
- What does "disabling memory sharing" mean ? what is the consequence for the performance ?
- perhaps it's also a bug because Datastage check for all lookup inputs to be in entire partionning while only reference input must be in entire, isn't it ?
thanks,
Martin.
warning in lookup stage
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It's not a bug. And there is no 7.6 version.
It's an alert. Only Entire partitioning can make use of shared memory for exchange of keys, or even needs to. All other partitioning algorithms don't need to share keys.
That you have chosen not to use Entire means that the use of shared memory for exchanging keys is disabled, and if your algorithm requires re-partitioning to occur then this will be done not via shared memory but via TCP sockets.
The alert message is warning you that such is the case.
It's an alert. Only Entire partitioning can make use of shared memory for exchange of keys, or even needs to. All other partitioning algorithms don't need to share keys.
That you have chosen not to use Entire means that the use of shared memory for exchanging keys is disabled, and if your algorithm requires re-partitioning to occur then this will be done not via shared memory but via TCP sockets.
The alert message is warning you that such is the case.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ok, so I try to use entire partitionning in input of the lookup and performance decrease, so I will keep the same partitionning and the warning.
ps : for the DS 7.6 version, the last post of this topic mentionned it (viewtopic.php?t=101149). Certainly, not PX.
ps : for the DS 7.6 version, the last post of this topic mentionned it (viewtopic.php?t=101149). Certainly, not PX.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: