Preserve Partitioning option when using multiple stages
Posted: Wed May 06, 2009 6:21 pm
I have a Sequential File stage as input that is connected with a Transformer stage.
From the Transformer stage, I need to connect to three output links.
The first one connects to a Lookup stage, the second one connects to a Funnel stage and the third one connects to a Filter stage.
The Funnel stage and Filter stage needs input data to be partitioned by the same set of keys and so I have partitioned the input data on the Transformer stage using these keys. This allows me to use "Same" parititioning for both Funnel and Filter stages.
However, the Lookup stage needs the input data to be partitioned on a different set of keys. Ideally, I would set the "Preserve Partitioning" flag to "Clear" in the Transformer stage so that the Lookup stage does not receive partitioned data. By doing so, the partitioning is cleared for Funnel and Filter stages also. This is not what I want since I cannot use "Same" partitioning for these two stages.
This is probably a naive question. Is there a way to selectively clear the partition for specific output links?
Thanks in advance!
From the Transformer stage, I need to connect to three output links.
The first one connects to a Lookup stage, the second one connects to a Funnel stage and the third one connects to a Filter stage.
The Funnel stage and Filter stage needs input data to be partitioned by the same set of keys and so I have partitioned the input data on the Transformer stage using these keys. This allows me to use "Same" parititioning for both Funnel and Filter stages.
However, the Lookup stage needs the input data to be partitioned on a different set of keys. Ideally, I would set the "Preserve Partitioning" flag to "Clear" in the Transformer stage so that the Lookup stage does not receive partitioned data. By doing so, the partitioning is cleared for Funnel and Filter stages also. This is not what I want since I cannot use "Same" partitioning for these two stages.
This is probably a naive question. Is there a way to selectively clear the partition for specific output links?
Thanks in advance!