In lookup stage, I am using hash partition on key colums on the stream and the reference links and I am getting the below warning in director.
When checking operator: Operator of type "APT_LUTCreateOp": will partition despite the preserve-partitioning flag on the data set on input port 0.
I have searched the forum and cleared the Partition on all the links and also on the lookup stage, but the warning is not getting removed.
If the hash partition is replaced by Auto or (Round robin and Entire) then the warning is not coming. Please help.
lookup will partition despite the preserve-partitioning flag
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 40
- Joined: Mon May 11, 2009 12:19 am
- Location: Madurai
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
Lookup expects the data in reference link partitioned using entire and any other partitioning schemes will make it throw this warning. How big is your reference data?
If you are sure of what you are doing then you can ignore this warning. Else try to keep it entire or auto to make all the data in reference link available to all nodes.
If you are sure of what you are doing then you can ignore this warning. Else try to keep it entire or auto to make all the data in reference link available to all nodes.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.![Wink :wink:](./images/smilies/icon_wink.gif)
Genius may have its limitations, but stupidity is not thus handicapped.
![Wink :wink:](./images/smilies/icon_wink.gif)
Arnd's suggestion is the standard response for this kind of warning; but if I remember correctly, it doesn't work in this case.
The lookup stage actually results in two operators, and the warning is generated between the lookup stage's operators, so going to a different stage to set the partitioning to clear doesn't alleviate it.
I don't remember exactly what my workaround was before... perhaps partitioning in the stage before the lookup and then setting the lookup to use SAME partitioning.
Mike
The lookup stage actually results in two operators, and the warning is generated between the lookup stage's operators, so going to a different stage to set the partitioning to clear doesn't alleviate it.
I don't remember exactly what my workaround was before... perhaps partitioning in the stage before the lookup and then setting the lookup to use SAME partitioning.
Mike
-
- Participant
- Posts: 40
- Joined: Mon May 11, 2009 12:19 am
- Location: Madurai
lookup warning
Thank you all for your replies. I have tried partitioned the data in the before stage and used same partition on lookup links. But the warning is not getting removed.