will partition despite the preserve-partitioning flag

[gowrishankar_h]

Hi Guys,

I have a senario to join 2 dataset using join stage.I put hash partition and sort on link to join stage since 2 dataset preserve with different partition.

its executing fine but iam getting follwoing warning.Can you please guide to takeaway this warning in log?.

When checking operator: Operator of type "APT_TSortOperator": will partition despite the preserve-partitioning flag on the data set on input port 0

Thanks
Gowri

[ArndW]

The warning refers to the stage at the end of the link leading to the left side of the join. In the advanced tab of that stage you will need to put "clear" instead of the stage's default to get rid of this warning.

[gowrishankar_h]

I changed the the preserve partition into clear but still iam getting this warning..We will get this waring because we use link on sort instead of using explict sort stage before join?..

[ray.wurlod]

Did you set Preserve Partitioning to Clear in the preceding stage in the job?

[gowrishankar_h]

Yes Ray,

We are joining 2 dataset in join stage which was set as preserve partition.
We cant change in dataset since it is used in many other job with that partition.

[ray.wurlod]

Then, yes, the warning is because you are using a link sort. Use an explicit Sort stage to eliminate the warning (keeping the Preserve Partitioning = "Clear" setting in the stage upstream of that).

Or you could demote the warning with a message handler since you know it doesn't affect what you're seeking to accomplish.

[gowrishankar_h]

Thanks..When i use explicit sort stage before join it works...But i want to know why its behaving warning when sorting with link?

[ray.wurlod]

Because the composer (conductor) inserts a tsort operator with all default settings (over which you have no control apart from memory allocation*) when you use a link sort. An explicit Sort stage gives you that control, via the stage Advanced properties.

* via APT_TSORT_STRESS_BLOCKSIZE environment variable