I have a Sequential File stage as input that is connected with a Transformer stage.
From the Transformer stage, I need to connect to three output links.
The first one connects to a Lookup stage, the second one connects to a Funnel stage and the third one connects to a Filter stage.
The Funnel stage and Filter stage needs input data to be partitioned by the same set of keys and so I have partitioned the input data on the Transformer stage using these keys. This allows me to use "Same" parititioning for both Funnel and Filter stages.
However, the Lookup stage needs the input data to be partitioned on a different set of keys. Ideally, I would set the "Preserve Partitioning" flag to "Clear" in the Transformer stage so that the Lookup stage does not receive partitioned data. By doing so, the partitioning is cleared for Funnel and Filter stages also. This is not what I want since I cannot use "Same" partitioning for these two stages.
This is probably a naive question. Is there a way to selectively clear the partition for specific output links?
Thanks in advance!
Preserve Partitioning option when using multiple stages
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 67
- Joined: Thu Aug 09, 2007 7:51 pm
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It's a good question, naive or not. The answer is no. Therefore you are going to wear the warning, and might choose to incorporate a message handler to demote it.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 6
- Joined: Tue Jan 29, 2008 3:48 am
- Location: Chennai
You could choose to use copy stage in between Transformer and other stagesray.wurlod wrote:It's a good question, naive or not. The answer is no. Therefore you are going to wear the warning, and might choose to incorporate a message handler to demote it. ...
Change your partition type in copy stage between Transformer and Lookup, in other copy stage set enforce option to false
Santhosh
-
- Premium Member
- Posts: 67
- Joined: Thu Aug 09, 2007 7:51 pm
-
- Premium Member
- Posts: 67
- Joined: Thu Aug 09, 2007 7:51 pm
-
- Charter Member
- Posts: 299
- Joined: Wed Nov 13, 2002 5:38 pm
- Location: USA
-
- Premium Member
- Posts: 67
- Joined: Thu Aug 09, 2007 7:51 pm
Thanks throbinson!
From the transformer, there are two links that are sent out to the Lookup stage. One of them is a primary link and the other reference link. Since the input data to the transformer is partitioned using a set of keys, I am using "Same" partitioning for both. Since you mentioned about "Entire" I looked into "Parallel Job Developer Guide" and found the following:
"You need to ensure that the data being looked up in the lookup table is in the same partition as the input data referencing it. One way of doing this is to partition the lookup tables using the Entire method. Another way is to partition it in the same way as the input data (although this implies sorting of the data)."
So I guess I am exempt from using "Entire" for the look-up stage
From the transformer, there are two links that are sent out to the Lookup stage. One of them is a primary link and the other reference link. Since the input data to the transformer is partitioned using a set of keys, I am using "Same" partitioning for both. Since you mentioned about "Entire" I looked into "Parallel Job Developer Guide" and found the following:
"You need to ensure that the data being looked up in the lookup table is in the same partition as the input data referencing it. One way of doing this is to partition the lookup tables using the Entire method. Another way is to partition it in the same way as the input data (although this implies sorting of the data)."
So I guess I am exempt from using "Entire" for the look-up stage