DSXchange

Hi

I am getting the following warnings

1)SEQL_Tgt_User: When checking operator: A sequential operator cannot
preserve the partitioning of the parallel data set on input port 0.

2)LKP_Master_1: When checking operator: Operator of type
"APT_LUTCreateOp": will partition despite the preserve-partitioning
flag on the data set on input port 0.


I am using a lookup to a dataset in this job and is not even complex partitioning logic. I am preserving the partition while carrying forward from previous job which creates the input as well as the lookup dataset. What can cause these warnings?

Insufficient information. What is your source? How are these data partitioned?

Clearly a Sequential File stage (which operates in sequential mode) can not preserve partitioning. You should not be expecting it to. Exactly why the Lookup stage is choosing to override your "preserve partitioning" setting can not be determined on the information you've provided.

ray.wurlod wrote:Insufficient information. What is your source? How are these data partitioned?

Clearly a Sequential File stage (which operates in sequential mode) can not preserve partitioning. You should not be expecting it to. Exactly why the Lookup stage is choosing to override your "preserve partitioning" setting can not be determined on the information you've provided.

Sorry

I missed it in a hurry. I have the sequential file as input then doing some transformation (using transformer) and doing lookup on three datasets (lookup master key files ) before loading final dataset.

I have put entire partitioning in all my lookup master key files. And also the link from transformer I am partitioning by hash. Then I am preserving that partition in my lookup.

Ray could you please tell me where I went wrong in determining the partition logic.

I have removed the preserve partitioning on the Sequential stage. But can we read data parallely in some way. I know of mentioning "Readers per node" but never used it. Please suggest.

If you used Entire when loading Lookup File Sets, then you must specify Entire when using them. Or (Auto). Or get the warning.

ray.wurlod wrote:If you used Entire when loading Lookup File Sets, then you must specify Entire when using them. Or (Auto). Or get the warning.

Thanks a lot Ray...we decided to use message handler for that.....

DO you think it is a good decision??

I would suggest, you can 'handle' the first one but not the second.

It's always better to know the cause of warnings and to eliminate them where possible. Sometimes time pressure to get a result means that you take a short cut, such as implementing a message handler. Don't forget to promote the message handler into test and production environments if it is your choice to use it.

Warning 2 can be handled in the job.
Is your preserve parition flag from your transforer is ser?
If so 'clear' it.