warning in lookup stage
Posted: Tue Sep 09, 2008 3:37 am
Hi,
I have a Datastage job design, like that or something like that :
INPUT --> SORT+HASH PARTITION --> TRANSFORMER(partition : SAME, no sort) --> LOOKUP (partition : SAME, no sort) --> AGREGATION (partition SAME, no sort)--> OUTPUT
First, I partition and sort data, and I want to keep the partition and the sort all along the job.
So, I used SAME partitionning and no sort in all stages.
and for the reference lookup i use entire partitionning.
But,
Because I use same partitionning, I have the following warnings :
In the transformer :"Transformer_128: Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."
In the lookup : "Lookup: Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."
So, I read post in the forum, and for the transformer warning it's a Datastage bug due to the use of the lookup logic in transformer. This bug is fix in the 7.6 version -> ok
but, it seem that the lookup warnings is not a bug. So my questions (finally) are :
- Does this warning is a problem for the logic of my job ?
- What does "disabling memory sharing" mean ? what is the consequence for the performance ?
- perhaps it's also a bug because Datastage check for all lookup inputs to be in entire partionning while only reference input must be in entire, isn't it ?
thanks,
Martin.
I have a Datastage job design, like that or something like that :
INPUT --> SORT+HASH PARTITION --> TRANSFORMER(partition : SAME, no sort) --> LOOKUP (partition : SAME, no sort) --> AGREGATION (partition SAME, no sort)--> OUTPUT
First, I partition and sort data, and I want to keep the partition and the sort all along the job.
So, I used SAME partitionning and no sort in all stages.
and for the reference lookup i use entire partitionning.
But,
Because I use same partitionning, I have the following warnings :
In the transformer :"Transformer_128: Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."
In the lookup : "Lookup: Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."
So, I read post in the forum, and for the transformer warning it's a Datastage bug due to the use of the lookup logic in transformer. This bug is fix in the 7.6 version -> ok
but, it seem that the lookup warnings is not a bug. So my questions (finally) are :
- Does this warning is a problem for the logic of my job ?
- What does "disabling memory sharing" mean ? what is the consequence for the performance ?
- perhaps it's also a bug because Datastage check for all lookup inputs to be in entire partionning while only reference input must be in entire, isn't it ?
thanks,
Martin.