Warning message when changing lookup stage to a join stage

[miwinter]

Having changed a lookup stage to a join stage (for performance improvement), we now see the warning below in the log - is there any explanation for this?

MapResult: When checking operator: A sequential operator cannot preserve the partitioning
of the parallel data set on input port 0.

[ray.wurlod]

You've got a Sequential File stage downstream of the Join stage. For the Join stage you have had to partition the data based on key value. The message is essentially telling you that the Sequential File stage does not operate in parallel, so cannot preserve the partitioning specified for the Join stage.

[miwinter]

Thank you, I'll check it out

[kumar_s]

As explained in the other post, Clear the partition on the output table for the Preserve Partion option.

[dsusr]

First of all why have you removing lookup stage to a join stage for performance improvement because lookup always give better performance to join.

dsusr

[kumar_s]

First of all why have you removing lookup stage to a join stage for performance improvement because lookup always give better performance to join.

dsusr

Not in all cases. Untill the lookup data is small enough to handled in lookup memory, lookup approach is better. Once it is beyond certain limit, it is advisable to switch join, which doesnt pre load the data into lookup memeory.

[dsusr]

Not in all cases. Untill the lookup data is small enough to handled in lookup memory, lookup approach is better. Once it is beyond certain limit, it is advisable to switch join, which doesnt pre load the data into lookup memeory.

Yes Kumar you are right that it depends on the data but performance of join will always be better and if we use the lookup when we have large amount of data then possibly due to high consumption of memory jobs may start getting aborted.

Also I had a chat with Senior IBM engineer and as per him we can use lookup till the data is less than 1 gb.