job running on single node

balajisr · Post by **balajisr** » Mon Nov 30, 2009 4:47 am

Force row generator stage to run in parallel and verify changes.

vinsashi · Post by **vinsashi** » Mon Nov 30, 2009 4:50 am

Hi Balaji,
I have changed default exection mode seq to parallel for rowgenreartor but its not running on 2nodes.

Thanks
vin

balajisr · Post by **balajisr** » Mon Nov 30, 2009 4:53 am

Can you please post the score generated?

Do you find this scenario in other jobs as well?

vinsashi · Post by **vinsashi** » Mon Nov 30, 2009 4:57 am

Code: Select all

main_program: This step has 4 datasets:
ds0: {op0[1p] (parallel delete data files in delete D:/test/sas.ds)
      ->eCollectAny
      op1[1p] (sequential delete descriptor file in delete D:/test/sas.ds)}
ds1: {op2[1p] (parallel APT_CombinedOperatorController(1):APT_LUTCreateOp in Lookup_2)
      eEntire->eCollectAny
      op3[1p] (parallel APT_CombinedOperatorController(0):APT_LUTProcessOp in Lookup_2)}
ds2: {op2[1p] (parallel APT_CombinedOperatorController(1):APT_LUTCreateOp in Lookup_2)
      eAny->eCollectAny
      op3[1p] (parallel APT_CombinedOperatorController(0):APT_LUTProcessOp in Lookup_2)}
ds3: {op3[1p] (parallel APT_CombinedOperatorController(0):APT_LUTProcessOp in Lookup_2)
      [pp] ->
      D:/test/sas.ds}
It has 4 operators:
op0[1p] {(parallel delete data files in delete D:/test/sas.ds)
    on nodes (
      node1[op0,p0]
    )}
op1[1p] {(sequential delete descriptor file in delete D:/test/sas.ds)
    on nodes (
      node1[op1,p0]
    )}
op2[1p] {(parallel APT_CombinedOperatorController(1):
      (Row_Generator_1)
      (APT_LUTCreateOp in Lookup_2)
    ) on nodes (
      node1[op2,p0]
    )}
op3[1p] {(parallel APT_CombinedOperatorController(0):
      (Row_Generator_0)
      (Sort_11)
      (APT_LUTProcessOp in Lookup_2)
    ) on nodes (
      node1[op3,p0]
    )}
It runs 4 processes on 1 node.

Thanks
vin..[/code]

balajisr · Post by **balajisr** » Mon Nov 30, 2009 5:01 am

Can you set APT_DISABLE_COMBINATION=true and try executing it again.
Hope you had modified the execution mode of source row generator "Row_Generator_0" and not the one which connects to lookup.

Do you get any warnings when you run the job?

vinsashi · Post by **vinsashi** » Mon Nov 30, 2009 5:10 am

Code: Select all

main_program: This step has 7 datasets:
ds0: {op0[1p] (parallel Row_Generator_0)
      eOther(APT_HashPartitioner { key={ value=empid, 
        subArgs={ asc }
      }
})->eCollectAny
      op2[1p] (parallel Sort_11)}
ds1: {op1[1p] (parallel Row_Generator_1)
      eEntire->eCollectAny
      op3[1p] (parallel APT_LUTCreateOp in Lookup_2)}
ds2: {op2[1p] (parallel Sort_11)
      [pp] eSame->eCollectAny
      op4[1p] (parallel APT_LUTProcessOp in Lookup_2)}
ds3: {op3[1p] (parallel APT_LUTCreateOp in Lookup_2)
      eEntire->eCollectAny
      op4[1p] (parallel APT_LUTProcessOp in Lookup_2)}
ds4: {op3[1p] (parallel APT_LUTCreateOp in Lookup_2)
      eAny->eCollectAny
      op4[1p] (parallel APT_LUTProcessOp in Lookup_2)}
ds5: {op5[1p] (parallel delete data files in delete D:/test/sas.ds)
      ->eCollectAny
      op6[1p] (sequential delete descriptor file in delete D:/test/sas.ds)}
ds6: {op4[1p] (parallel APT_LUTProcessOp in Lookup_2)
      [pp] ->
      D:/test/sas.ds}
It has 7 operators:
op0[1p] {(parallel Row_Generator_0)
    on nodes (
      node1[op0,p0]
    )}
op1[1p] {(parallel Row_Generator_1)
    on nodes (
      node1[op1,p0]
    )}
op2[1p] {(parallel Sort_11)
    on nodes (
      node1[op2,p0]
    )}
op3[1p] {(parallel APT_LUTCreateOp in Lookup_2)
    on nodes (
      node1[op3,p0]
    )}
op4[1p] {(parallel APT_LUTProcessOp in Lookup_2)
    on nodes (
      node1[op4,p0]
    )}
op5[1p] {(parallel delete data files in delete D:/test/sas.ds)
    on nodes (
      node1[op5,p0]
    )}
op6[1p] {(sequential delete descriptor file in delete D:/test/sas.ds)
    on nodes (
      node1[op6,p0]
    )}
It runs 7 processes on 1 node.

i got this one after putting true for apt_disable_combination.
i have changed execution mode for two rowgenerators.i didnt get any warnings..
Thanks
vin...

balajisr · Post by **balajisr** » Mon Nov 30, 2009 5:17 am

What about other jobs. Do you see same behaviour in other jobs as well?

vinsashi · Post by **vinsashi** » Mon Nov 30, 2009 5:25 am

Hi,
I have changed my config file to 2node in my system.but i have to implement paralligm. so i am trying to run job in parallel mode but it not working.

balajisr · Post by **balajisr** » Mon Nov 30, 2009 5:27 am

vinsashi wrote:Hi,
I have changed my config file to 2node in my system.but i have to implement paralligm. so i am trying to run job in parallel mode but it not working.

That's ok. Do you get same issues in your other jobs as well?

vinsashi · Post by **vinsashi** » Mon Nov 30, 2009 5:48 am

Hi,
For other jobs also i am getting same problem.one job score check it this one plz.

Code: Select all

main_program: This step has 10 datasets:
ds0: {op0[1p] (parallel Row_Generator_0)
      eAny->eCollectAny
      op1[1p] (parallel APT_TransformOperatorImplV0S2_test001_Transformer_2 in Transformer_2)}
ds1: {op1[1p] (parallel APT_TransformOperatorImplV0S2_test001_Transformer_2 in Transformer_2)
      eOther(APT_HashPartitioner { key={ value=ename },
  key={ value=mid }
})->eCollectAny
      op2[1p] (parallel APT_HashedGroup2Operator in Aggregator_6)}
ds2: {op1[1p] (parallel APT_TransformOperatorImplV0S2_test001_Transformer_2 in Transformer_2)
      eOther(APT_HashPartitioner { key={ value=ename }
})->eCollectAny
      op4[1p] (parallel inserted tsort operator {key={value=ename, subArgs={asc, nulls={value=first}, cs}}}(1) in Join_9)}
ds3: {op2[1p] (parallel APT_HashedGroup2Operator in Aggregator_6)
      [pp] eSame->eCollectAny
      op3[1p] (parallel inserted tsort operator {key={value=ename, subArgs={asc, cs}}}(0) in Join_9)}
ds4: {op3[1p] (parallel inserted tsort operator {key={value=ename, subArgs={asc, cs}}}(0) in Join_9)
      [pp] eSame->eCollectAny
      op5[1p] (parallel buffer(0))}
ds5: {op4[1p] (parallel inserted tsort operator {key={value=ename, subArgs={asc, nulls={value=first}, cs}}}(1) in Join_9)
      [pp] eSame->eCollectAny
      op6[1p] (parallel buffer(1))}
ds6: {op5[1p] (parallel buffer(0))
      [pp] eSame->eCollectAny
      op7[1p] (parallel APT_JoinSubOperatorNC in Join_9)}
ds7: {op6[1p] (parallel buffer(1))
      [pp] eSame->eCollectAny
      op7[1p] (parallel APT_JoinSubOperatorNC in Join_9)}
ds8: {op7[1p] (parallel APT_JoinSubOperatorNC in Join_9)
      [pp] eSame->eCollectAny
      op8[1p] (parallel APT_TransformOperatorImplV0S12_test001_Transformer_12 in Transformer_12)}
ds9: {op8[1p] (parallel APT_TransformOperatorImplV0S12_test001_Transformer_12 in Transformer_12)
      [pp] ->eCollectAny
      op9[1p] (sequential APT_RealFileExportOperator in Sequential_File_1)}
It has 10 operators:
op0[1p] {(parallel Row_Generator_0)
    on nodes (
      node1[op0,p0]
    )}
op1[1p] {(parallel APT_TransformOperatorImplV0S2_test001_Transformer_2 in Transformer_2)
    on nodes (
      node1[op1,p0]
    )}
op2[1p] {(parallel APT_HashedGroup2Operator in Aggregator_6)
    on nodes (
      node1[op2,p0]
    )}
op3[1p] {(parallel inserted tsort operator {key={value=ename, subArgs={asc, cs}}}(0) in Join_9)
    on nodes (
      node1[op3,p0]
    )}
op4[1p] {(parallel inserted tsort operator {key={value=ename, subArgs={asc, nulls={value=first}, cs}}}(1) in Join_9)
    on nodes (
      node1[op4,p0]
    )}
op5[1p] {(parallel buffer(0))
    on nodes (
      node1[op5,p0]
    )}
op6[1p] {(parallel buffer(1))
    on nodes (
      node1[op6,p0]
    )}
op7[1p] {(parallel APT_JoinSubOperatorNC in Join_9)
    on nodes (
      node1[op7,p0]
    )}
op8[1p] {(parallel APT_TransformOperatorImplV0S12_test001_Transformer_12 in Transformer_12)
    on nodes (
      node1[op8,p0]
    )}
op9[1p] {(sequential APT_RealFileExportOperator in Sequential_File_1)
    on nodes (
      node1[op9,p0]
    )}
It runs 10 processes on 1 node.

Thanks
vin...

varaprasad · Post by **varaprasad** » Mon Nov 30, 2009 6:55 am

Please check if that ID has proper access permissions on the second node. And also check the correctness of the path.

varaprasad · Post by **varaprasad** » Mon Nov 30, 2009 6:56 am

Please check if that ID has proper access permissions on the second node. And also check the correctness of the path.

sachinnilose · Post by **sachinnilose** » Mon Nov 30, 2009 7:38 am

Please try the below config file. I think you placed brackets at wring location so it is reading only upto first node and ignoring node 2.

Changed Configuration File

{
node "node1"
{
fastname "TOMEDW"
pools ""
resource disk "C:/IBM/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
node "node2"
{
fastname "TOMEDW"
pools ""
resource disk "E:/datastage/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
}

sachinnilose · Post by **sachinnilose** » Mon Nov 30, 2009 7:39 am

Please try the below config file. I think you placed brackets at wrong location so it is reading only upto first node and ignoring node 2.

Changed Configuration File

{
node "node1"
{
fastname "TOMEDW"
pools ""
resource disk "C:/IBM/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
node "node2"
{
fastname "TOMEDW"
pools ""
resource disk "E:/datastage/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
}

sachinnilose · Post by **sachinnilose** » Mon Nov 30, 2009 7:40 am

Please try the below config file. I think you placed brackets at wring location so it is reading only upto first node and ignoring node 2.

Changed Configuration File

{
node "node1"
{
fastname "TOMEDW"
pools ""
resource disk "C:/IBM/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
node "node2"
{
fastname "TOMEDW"
pools ""
resource disk "E:/datastage/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
}