For using Datastage to take benefit of parallel processing, we need to configure multiple nodes in config file, is all the nodes configured are separate physical machines or they are just logical machines?
If they are physical machines do we need to install some Datastage components on all the nodes?
Node configuration
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 48
- Joined: Tue Nov 25, 2008 11:10 pm
- Location: Des Moines,IA
Re: Node configuration
can you please elaborate what you meant by that?bunty wrote:, is all the nodes configured are separate physical machines or they are just logical machines?
"It's easier to go down a hill than up it but the view is much better at the top"
-Bennet,Arnold
-Bennet,Arnold
Re: Node configuration
Suppose we have our configuration file is configured to use 4 different nodes and we have given different fast names and other settings for the each node and I have one parallel job, which read records from flat file and we have one aggregation stage for doing sum and finally storing the aggregates to some flat file.datskosaraju wrote:can you please elaborate what you meant by that?bunty wrote:, is all the nodes configured are separate physical machines or they are just logical machines?
we have partitioning logic as round robin, so my question is, is the data is going to be moved to different nodes or where the aggregation will take place means at which node, if the data moves to different nodes based on round robin then which process at that node will do the aggregation like sum.
bunty
Re: Node configuration
If I say ,in general, about parallel processing in Datastage,it depends on the environment that you use like SMP,Cluster or MPP .One physical node can be divided across multiple logical nodes which you build accordingly in config file of DS.For the scenario asked above,it was already explained in ParallelJob developers guide about different partitionings and methodologies.
Kiran Vaduguri
As soon as the fear approaches near, attack and destroy it.
As soon as the fear approaches near, attack and destroy it.
-
- Participant
- Posts: 2
- Joined: Wed Mar 26, 2008 4:41 am
- Location: Bangalore
Re: Node configuration
Aggregation would happen at every node , not at any particular node.
In config files node names are logical, node number does not necessarily correspond to number of CPUs. There can be more than one node per CPU depends on situation where tasks are not CPU intensive.
In config files node names are logical, node number does not necessarily correspond to number of CPUs. There can be more than one node per CPU depends on situation where tasks are not CPU intensive.
i dont believe in god but I am afraid of Him.