Node vs Process

[karthi_gana]

All,

I have desgined a simple job.

Code: Select all

SeqFile Stage (only one file) --> Filter Stage --> Dataset Stage

I just changed the "Number of Readers Per Node" value to 2,3,4,5 and monitored the job score.

main_program: This step has 3 datasets:
ds0: {op0[3p] (parallel Sequential_File_0)
eAny=>eCollectAny
op1[3p] (parallel Filter_26)}
ds1: {op2[3p] (parallel delete data files in delete /bis_data/datasets/dataset_test.ds)
>>eCollectAny
op3[1p] (sequential delete descriptor file in delete /bis_data/datasets/dataset_test.ds)}
ds2: {op1[3p] (parallel Filter_26)
=>
/bis_data/datasets/dataset_test.ds}
It has 4 operators:
op0[3p] {(parallel Sequential_File_0)
on nodes (
node1[op0,p0]
node1[op0,p1]
node1[op0,p2]
)}
op1[3p] {(parallel Filter_26)
on nodes (
node1[op1,p0]
node2[op1,p1]
node3[op1,p2]
)}
op2[3p] {(parallel delete data files in delete /bis_data/datasets/dataset_test.ds)
on nodes (
node1[op2,p0]
node2[op2,p1]
node3[op2,p2]
)}
op3[1p] {(sequential delete descriptor file in delete /bis_data/datasets/dataset_test.ds)
on nodes (
node1[op3,p0]
)}
It runs 10 processes on 3 nodes.

Try #1:

Rumber of Readers Per Node = 3

It runs 10 processes on 3 nodes.

Try #2:

Rumber of Readers Per Node = 4

It runs 11 processes on 3 nodes.

Try #3:

Rumber of Readers Per Node = 5

It runs 12 processes on 3 nodes.

I really don't understand what is the relation ship with Process and Node? what is mean by Process? what is mean by Node? Can anybody explain it little bit in detail for me?

[jwiles]

Nodes: Logical nodes in your configuration file

Processes: running processes on the server (ps -ef in Linux/Unix to get a list of executing processes)

[qt_ky]

The Parallel Job Developer's Guide chapter "The Parallel engine configuration file" will give you details about the config file and nodes.

[ray.wurlod]

The Sequential File stage runs on one node (because the file only exists on one node). Each increase of 1 in number of readers per node will cause an increase of 1 in the number of processes executing on this node - which you should have seen if you were monitoring the job score thoroughly.