Node And partiioning clarification

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
myukassign
Premium Member
Premium Member
Posts: 238
Joined: Fri Jul 25, 2008 8:55 am

Node And partiioning clarification

Post by myukassign »

I need a help to get my concept validated... I read many articles and I have a mixed understanding about partitioning and node.

Q1. If I have 4 nodes configured in my config file. Any parallel job that running in my server will make exactly 4 copies of my job. Is that right?

Q2. If I hash partitioned my stages and I have only 2 records coming. At the max it will process the data through two nodes and the remaining copies of job in the other two nodes will become void.

Please correct me if my understanding is wrong gurus. Many thanks for your time reading and replying in advance.
tbharathkumar
Participant
Posts: 26
Joined: Mon Aug 27, 2007 6:27 am
Location: Des Moines

Re: Node And partiioning clarification

Post by tbharathkumar »

Q1: Yes, Four instances will create
Q2:Yes, I guess you are correct
Regards,
Bharath Tipirisetty
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

1. Assuming that four nodes are in your default node pool and no node pools are constraining which nodes stages execute on and provided no stage executes in sequential mode, yes, stages will execute using four nodes. But this is not the same as "four copies of your job". One copy (the "score") is made, and distributed to four processing nodes.

2. At least two nodes will process zero rows. There is no concept of "become void". The two rows may be executed on two nodes or may be executed on one node, depending on the partitioning algorithm, node pools and whether the stages are executing in sequential mode.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply