lookup will partition despite the preserve-partitioning flag

[priyadharsini]

In lookup stage, I am using hash partition on key colums on the stream and the reference links and I am getting the below warning in director.

When checking operator: Operator of type "APT_LUTCreateOp": will partition despite the preserve-partitioning flag on the data set on input port 0.

I have searched the forum and cleared the Partition on all the links and also on the lookup stage, but the warning is not getting removed.
If the hash partition is replaced by Auto or (Round robin and Entire) then the warning is not coming. Please help.

[asorrell]

Is the lookup data source a dataset? The partitioning scheme is stored in a dataset when it is created.

[priyadarshikunal]

Lookup expects the data in reference link partitioned using entire and any other partitioning schemes will make it throw this warning. How big is your reference data?

If you are sure of what you are doing then you can ignore this warning. Else try to keep it entire or auto to make all the data in reference link available to all nodes.

[ArndW]

Since it is Input port 0 it is your stream data, you should go to the stage BEFORE the lookup and ensure that it sets "clear" partitioning.

[Mike]

Arnd's suggestion is the standard response for this kind of warning; but if I remember correctly, it doesn't work in this case.

The lookup stage actually results in two operators, and the warning is generated between the lookup stage's operators, so going to a different stage to set the partitioning to clear doesn't alleviate it.

I don't remember exactly what my workaround was before... perhaps partitioning in the stage before the lookup and then setting the lookup to use SAME partitioning.

Mike

[priyadharsini]

Thank you all for your replies. I have tried partitioned the data in the before stage and used same partition on lookup links. But the warning is not getting removed.