Transformer warning

just4u_sharath · Post by **just4u_sharath** » Mon Jan 28, 2008 1:25 pm

When i selected Partition type of SAME for the transformer (transformer name is FtrGrpFields), this is the INFO i get. This is not a warning nor an error. But I want to know what is this
"FtrGrpFileds:Input dataset 0 has a partitioning method other than entire specified; disabling memory sharing."

What is Disabling Memory Sharing

Teej · Post by **Teej** » Mon Jan 28, 2008 6:56 pm

Good question.

If you would look in the advanced tab within each individual stage for DataStage, you would see several options, such as "Execution mode", "Combinability mode" and "Preserve partition".

The first option tells that specific stage whether it should run in parallel or sequential mode. Sometimes you do not have an option to change it. This is great if you have a specific reason for sequentially reading data.

The combinability mode refers to the ability for DataStage to dynamically optimize the job by combining this stage with other stages if possible into one big stage (and one big memory allocation instead of multiple). I have never found a need to use this option (preferring to use $APT_DISABLE_COMBINATION to determine the guilty stage that is throwing warnings and errors.)

The third option, "Preserve Partition," is dedicated to determining if the partition should be kept as is for the output link for this stage. This tells the later stages, "Hey, keep the data like this, please."

However, there are stages that have to say, "I'm sorry, but I have to re-partition this data." Sort stage is one. Manually set the partition option is another. Hence, you get a warning message saying, "Unlike what you want, I must repartition it."

This is a great way for the developer to know, "Okay, do we NEED to partition the data here?" If the answer is, "Yes," then you can go to the previous stage from where this warning message is showing up, and change the "Preserve Partition" option to "Clear" instead of "Propagate".

* * *

To succiently answer your question -- Your warning message is caused by forcing partition (either by choice or by specific insertion done by DataStage), and this goes against a previous stage's desire to propagate/preserve the partition. Tell the previous stage to clear the partition, and the warning message will disappear.

However, be absolutely sure you DO want to partition the data, as it incur some processing expenses.

just4u_sharath · Post by **just4u_sharath** » Mon Jan 28, 2008 7:22 pm

Teej wrote:Good question.

If you would look in the advanced tab within each individual stage for DataStage, you would see several options, such as "Execution mode", "Combinability mode" and "Preserve partition".

The first option tells that specific stage whether it should run in parallel or sequential mode. Sometimes you do not have an option to change it. This is great if you have a specific reason for sequentially reading data.

The combinability mode refers to the ability for DataStage to dynamically optimize the job by combining this stage with other stages if possible into one big stage (and one big memory allocation instead of multiple). I have never found a need to use this option (preferring to use $APT_DISABLE_COMBINATION to determine the guilty stage that is throwing warnings and errors.)

The third option, "Preserve Partition," is dedicated to determining if the partition should be kept as is for the output link for this stage. This tells the later stages, "Hey, keep the data like this, please."

However, there are stages that have to say, "I'm sorry, but I have to re-partition this data." Sort stage is one. Manually set the partition option is another. Hence, you get a warning message saying, "Unlike what you want, I must repartition it."

This is a great way for the developer to know, "Okay, do we NEED to partition the data here?" If the answer is, "Yes," then you can go to the previous stage from where this warning message is showing up, and change the "Preserve Partition" option to "Clear" instead of "Propagate".

* * *

To succiently answer your question -- Your warning message is caused by forcing partition (either by choice or by specific insertion done by DataStage), and this goes against a previous stage's desire to propagate/preserve the partition. Tell the previous stage to clear the partition, and the warning message will disappear.

However, be absolutely sure you DO want to partition the data, as it incur some processing expenses.

In My Case, theres no need for me to partition the data. Thats why i put SAME partitioning which is fastest (i believe) . And i guess when we have a partitioning type SAME for a stage, and optionDefault(Propogate) for its previous stage, this should not matter. Propogate is just equal to SAME. Before the transformer there are 3 other stages which all has a partitioing type SAME.

Teej · Post by **Teej** » Mon Jan 28, 2008 7:52 pm

There is most likely a stage within your job that REQUIRES sort and partitioning. Join stage is one. Aggregate stage is another. There are more, and you should read the Developer Guide for that detail. Many of them would automatically add this partitioning and sorting for you.

There is also output sequential stages that REQUIRES partitioning. After all, from many nodes to one node = re-partition the data. DataStage will automatically add the partitioning for you.

Also... please do not force "Same" partitioning. Use Auto unless you have a specific reason to chose a partition option.

throbinson · Post by **throbinson** » Tue Jan 29, 2008 6:20 am

This is a bug (at least on AIX). There is an e-case and patch from IBM.

From the README...

PATCH REPORT FOR ECASE: 61672
PRODUCT: DSEE
VERSION: 7.5.2

PROBLEM REPORTED:
Warning message appears in the transformeer for migration from v6 to v7.5.

CAUSE OF PROBLEM:
This a side affect of the integration of lookup functionality into the
transform operator.

FIX SUMMARY:
The warning message is meaningless if lookup functionality isn't used in the
transform and can be ignored. This is disabled in the transform.

ray.wurlod · Post by **ray.wurlod** » Tue Jan 29, 2008 9:28 pm

Entire partitioning uses shared memory on SMP environments/configurations. Other partitioning algorithms do not (because they do not need to share anything).

If you choose a different partitioning algorithm than Entire (or Auto) for a reference input, then this warning occurs.

It's not a "bad" warning, it's an "alert" warning. Anything that is unusual is logged by DataStage as a warning.

In this case, since I knew I'd made a deliberate choice about partitioning on the reference input, I'd create or attach a message handler that demotes this alert to informational.

just4u_sharath · Post by **just4u_sharath** » Tue Jan 29, 2008 11:41 pm

ray.wurlod wrote:Entire partitioning uses shared memory on SMP environments/configurations. Other partitioning algorithms do not (because they do not need to share anything).

If you choose a different partitioning algorithm than Entire (or Auto) for a reference input, then this warning occurs.

It's not a "bad" warning, it's an "alert" warning. Anything that is unusual is logged by DataStage as a warning.

In this case, since I knew I'd made a deliberate choice about partitioning on the reference input, I'd create or attach a message handler that demotes this alert to informational.

No. its not a reference link.
Output of Join stage is fed to transformer stage and output of transformer stage is fed to dataset. In the output of join stage, i have selected "Default(propagate) in the advanced tab and in the input properties of transformer i have selected SAME as the partitioning type. Again in the advances tab of transformer selected "default propagate" and Same partitioning in the input properties of Dataset.
When i do this "other partioning is used other than entire specified, disabling memory sharing". This Info( not a warning, not a error) is for transformer stage only. there is not reference links in this job.

just4u_sharath · Post by **just4u_sharath** » Wed Jan 30, 2008 12:09 am

ray.wurlod wrote:Entire partitioning uses shared memory on SMP environments/configurations. Other partitioning algorithms do not (because they do not need to share anything).

If you choose a different partitioning algorithm than Entire (or Auto) for a reference input, then this warning occurs.

It's not a "bad" warning, it's an "alert" warning. Anything that is unusual is logged by DataStage as a warning.

In this case, since I knew I'd made a deliberate choice about partitioning on the reference input, I'd create or attach a message handler that demotes this alert to informational.

No. its not a reference link.
Output of Join stage is fed to transformer stage and output of transformer stage is fed to dataset. In the output of join stage, i have selected "Default(propagate) in the advanced tab and in the input properties of transformer i have selected SAME as the partitioning type. Again in the advances tab of transformer selected "default propagate" and Same partitioning in the input properties of Dataset.
When i do this "other partioning is used other than entire specified, disabling memory sharing". This Info( not a warning, not a error) is for transformer stage only. there are no reference links in this job.

ray.wurlod · Post by **ray.wurlod** » Wed Jan 30, 2008 1:15 am

None of which negates anything I posted. It appears that other circumstances - as well as reference links - can generate this warning. It's still just an alert that something unusual, or non-standard, exists in the design.

throbinson · Post by **throbinson** » Wed Jan 30, 2008 6:27 am

It sounds like Ray is saying choosing a partitioning method other then Entire or Auto for a Transformer input stream link is unusual to DataStage hence the warning.
Ray, can you explain why this is the case?

just4u_sharath · Post by **just4u_sharath** » Wed Jan 30, 2008 12:25 pm

ray.wurlod wrote:None of which negates anything I posted. It appears that other circumstances - as well as reference links - can generate this warning. It's still just an alert that something unusual, or non-standard, exists in the design.

It sounds like Ray is saying choosing a partitioning method other then Entire or Auto for a Transformer input stream link is unusual to DataStage hence the warning.
Ray, can you explain why this is the case?

Is this true. If it is the case can you please explain this if you dont mind. It will be a great help if you .