why lookup reference link requires entire partition method?
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 57
- Joined: Mon Dec 03, 2007 12:43 am
- Location: hyderabad
why lookup reference link requires entire partition method?
why lookup reference link requires entire partition method and the master link ,hash partitioning method?
ramesh.n.
The parallel jobs leverage the paralellism by using the pipelining and partitining mechanism. During partitioning the Reference records might be distributed accross the nodes. But if the input records are looked up to the wrong node then the output might be problemetic. Entire partition copy the complete set of records on each and every node. Which ensures the proper lookup(Does not matter on which node input records are) and proper output.
Hope this makes sense.
Hope this makes sense.
The parallel jobs leverage the paralellism by using the pipelining and partitining mechanism. During partitioning the Reference records might be distributed accross the nodes. But if the input records are looked up to the wrong node then the output might be problemetic. Entire partition copy the complete set of records on each and every node. Which ensures the proper lookup(Does not matter on which node input records are) and proper output.
Hope this makes sense.
Hope this makes sense.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Re: why lookup reference link requires entire partition meth
They don't. These are merely the defaults, which are guaranteed to give correct results.ramesh_inform wrote:why lookup reference link requires entire partition method and the master link ,hash partitioning method?
Another possibility is that the lookup key is a single integer. In that case the stream input could use the Modulus partitioning algorithm, if that gives more even spread of rows over nodes.
Similarly, it is legitimate (and even desirable in multiple machine configurations) to partition the reference input identically to the stream input, to avoid distributing all rows to all nodes.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 57
- Joined: Mon Dec 03, 2007 12:43 am
- Location: hyderabad
-
- Participant
- Posts: 49
- Joined: Fri May 11, 2007 12:24 am
- Location: kolkata
could u please confirm weather the lookup data get copied to all the partitions or data from all the partition are poll to memory for lookup.bkumar103 wrote:The parallel jobs leverage the paralellism by using the pipelining and partitining mechanism. During partitioning the Reference records might be distributed accross the nodes. But if the input records are looked up to the wrong node then the output might be problemetic. Entire partition copy the complete set of records on each and every node. Which ensures the proper lookup(Does not matter on which node input records are) and proper output.
Hope this makes sense.
regards
sunayan
sunayan
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 49
- Joined: Fri May 11, 2007 12:24 am
- Location: kolkata
could u please confirm weather the lookup data get copied to all the partitions or data from all the partition are poll to memory for lookup.bkumar103 wrote:The parallel jobs leverage the paralellism by using the pipelining and partitining mechanism. During partitioning the Reference records might be distributed accross the nodes. But if the input records are looked up to the wrong node then the output might be problemetic. Entire partition copy the complete set of records on each and every node. Which ensures the proper lookup(Does not matter on which node input records are) and proper output.
Hope this makes sense.
regards
sunayan
sunayan
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 49
- Joined: Fri May 11, 2007 12:24 am
- Location: kolkata
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: