partition despite preserve-partitioning flag

[just4geeks]

I am trying to eliminate warnings from the Director Log. Does anyone have any clue on how can I remove the following warning? It is related to a Look up stage with all partitioning options set to default. (preserve-partitioning flag is propagate)

LKUP: When checking operator: Operator of type "APT_LUTCreateOp": will partition despite the preserve-partitioning flag on the data set on input port 0.

Thanks for your time..

[Havoc]

In the Lookup stage goto Stage->Advanced->Set Preserve Partitioning Flag to 'Clear'

[just4geeks]

In the Lookup stage goto Stage->Advanced->Set Preserve Partitioning Flag to 'Clear'

I did that and I get the following error now.

LKUP: When checking operator: Operator of type "APT_LUTCreateOp": will partition despite the preserve-partitioning flag on the data set on input port 1.

Also, I should have mentioned in my earlier post that the subsequent warning is as follows. Not sure if it helps in finding the cause of the warning.

buffer(2): When checking operator: Operator of type "APT_BufferOperator": will partition despite the preserve-partitioning flag on the data set on input port 0.

[ray.wurlod]

Set the preserve partitioning flag to Clear upstream of these stages.

That is, clear the preserve partitioning flag on the output link of the prior stage on each input link to the Lookup stage.

[abc123]

Ray, I had the same problem and your solution works but can you tell me why? I would think that, if we set the preserve-partitioning flag to Default(Propagate), it should not give the warning but it does.

The message I get is "When checking operator: Operator of type "APT_TSortOperator": will partition despite the preserve-partitioning flag on the data set on input port 0".

I have an earlier stage where Preserve-Partitioning is set to Set. Then, ongoing stages have it to Default(propagate). The last stage that causes the warning has Set by default. I think, the error is probably because partitioning is being set while it was already set but the message isn't clear.

Also, can you tell me why it says "input port 0"? Thanks.

[ray.wurlod]

Inputs (input ports) are numbered from 0. So if you have a stage with three inputs, they are #0, #1 and #2. Outputs are also numbered from 0.

The alert is generated whenever the downstream operator can not use the same partitioning as the upstream operator. This may be, for example, because the downstream operator executes sequentially or in a different node pool than the upstream operator.

Note that I have deliberately used the term "operator" rather than "stage" here. Your job has a tsort operator which may be one you designed in or may be one that the Framework inserted. You need to check the score to see what is actually being executed.

In this case the tsort operator is overriding the upstream partitioning because input to this operator must use a key-based partitioning algorithm.

[jatinrheen]

Thanks