DB2 EE Stage - The Section Leader on node has terminated

pras · Post by **pras** » Wed Sep 19, 2012 1:34 pm

Hi -

We are migrating our Jobs from 8.0 to 8.7 in Linux. After installation and configuring the dsenv file, When we run the Orchadmin check we are getting the below error

##I IIS-DSEE-TFCN-00001 15:25:19(000) <main_program>
IBM InfoSphere DataStage Enterprise Edition 8.7.0.6419
Copyright (c) 2001, 2005-2011 IBM Corporation. All rights reserved

##I IIS-DSEE-TUTL-00031 15:25:19(001) <main_program> The open files limit is 8192; raising to 65535.
##I IIS-DSEE-TFCN-00006 15:25:19(002) <main_program> conductor uname: -s=Linux; -r=2.6.18-308.4.1.el5; -v=#1 SMP Wed Mar 28 01:54:56 EDT 2012; -n=dwapp2w129m3; -m=x86_64
##I IIS-DSEE-TCOA-00067 15:25:19(003) <main_program> OS charset: ISO-8859-1.
##I IIS-DSEE-TCOA-00068 15:25:19(004) <main_program> Input charset: UTF-8.
##I IIS-DSEE-TFSC-00001 15:25:19(005) <main_program> APT configuration file: /opt/IBM/prd/InformationServer/Server/DSEngine/../Configurations/default.apt
##E IIS-DSEE-TFPM-00330 15:25:19(006) <main_program> The Section Leader on node db2_server3 has terminated unexpectedly.
##E IIS-DSEE-TFPM-00330 15:25:19(007) <main_program> The Section Leader on node db2_server1 has terminated unexpectedly.
##E IIS-DSEE-TFPM-00330 15:25:19(008) <main_program> The Section Leader on node db2_server2 has terminated unexpectedly.
##E IIS-DSEE-TFPM-00330 15:25:19(009) <main_program> The Section Leader on node db2_server5 has terminated unexpectedly.
##E IIS-DSEE-TFPM-00330 15:25:19(010) <main_program> The Section Leader on node db2_server4 has terminated unexpectedly.
##F IIS-DSEE-TFPM-00113 15:30:53(000) <APT_CheckConfigOperator,0> Fatal Error: Unable to start ORCHESTRATE network connection on node node1(dwapp2w129m3): COMPLETEWAIT failed: parallel APT_CheckConfigOperator(0,0)
##F IIS-DSEE-TFPM-00114 15:30:53(000) <APT_RealFileExportOperator1 in APT_FileExportOperator,0> Fatal Error: Unable to start ORCHESTRATE network connection on node node1 (dwapp2w129m3): APT_PMConnectionSetup:: operator 1(sequential APT_RealFileExportOperator1 in APT_FileExportOperator)timed out with 5 incomplete incoming connections.
##E IIS-DSEE-TFPM-00192 15:30:53(000) <node_node1> Player 1 terminated unexpectedly.
##E IIS-DSEE-TFPM-00338 15:30:53(000) <main_program> APT_PMsectionLeader(1, node1), player 1 - Unexpected exit status 1.
##E IIS-DSEE-TFPM-00192 15:30:53(001) <node_node1> Player 2 terminated unexpectedly.
##E IIS-DSEE-TFPM-00338 15:30:53(001) <main_program> APT_PMsectionLeader(1, node1), player 2 - Unexpected exit status 1.
##W IIS-DSEE-TFPM-00091 15:30:58(000) <main_program> APT_PMpollUntilZero: WARNING: called with counter = 0
##E IIS-DSEE-TFSC-00011 15:31:03(000) <main_program> Step execution finished with status = FAILED.
##E IIS-DSEE-TCOA-00069 15:31:03(001) <main_program> ERROR: check configuration file failed.

1) We are able to do rsh to all the db2 nodes from the Application server
2) We are able to connect to the DB2 from commandline
3) Connector stage connects to DB2.

Please help me to resolve this.

Thanks,
Prasanna

lstsaur · Post by **lstsaur** » Wed Sep 19, 2012 4:24 pm

Did you set APT_CONFIG_FILE correctly? From reading the error message, it seemed like that your default config.apt file cannot access all the disks defined in this file from all the processing nodes.

ArndW · Post by **ArndW** » Thu Sep 20, 2012 1:35 am

Can you post your APT config file plus what your new system name is ("hostname" from the command line)?

pras · Post by **pras** » Thu Sep 20, 2012 8:48 am

Host Name:dwapp2w129m3

Code: Select all

APT_CONFIG_FILE:

 {
        node "node1"
        {
                fastname "dwapp2w129m3"
                pools ""
                resource disk "/etrade/IBM/prd/InformationServer/Server/Datasets" {pools ""}
                resource scratchdisk "/etrade/IBM/prd/InformationServer/Server/Scratch" {pools ""}
        }

        node "db2_server1"
        {
                fastname "dw1w129m3"
                pools "db2"
                resource disk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Datasets" {pools ""}
                resource scratchdisk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Scratch" {pools ""}
        }

        node "db2_server2"
        {
                fastname "dw2w129m3"
                pools "db2"
                resource disk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Datasets" {pools ""}
                resource scratchdisk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Scratch" {pools ""}
        }

        node "db2_server3"
        {
                fastname "dw3w129m3"
                pools "db2"
                resource disk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Datasets" {pools ""}
                resource scratchdisk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Scratch" {pools ""}
        }

        node "db2_server4"
        {
                fastname "dw4w129m3"
                pools "db2"
                resource disk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Datasets" {pools ""}
                resource scratchdisk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Scratch" {pools ""}
        }

        node "db2_server5"
 {
                fastname "dw5w129m3"
                pools "db2"
                resource disk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Datasets" {pools ""}
                resource scratchdisk "/etrade/prd/edw/dbs/edwinst1/etlreject/ETL/Scratch" {pools ""}
        }




}

ArndW · Post by **ArndW** » Thu Sep 20, 2012 9:12 am

Are you in clustered environment and the other fastnames in your config file are "ping"able systems with the requisite directories? If not, then replace the fastnames with "dwapp2w129m3" in order to make the config file acceptable.

jwiles · Post by **jwiles** » Thu Sep 20, 2012 10:32 am

You mention DB2 EE stage in the thread title, so...

Have the DS 8.7 engine executables been correctly copied or mounted to the db2 nodes? If not, this could be the issue as the 8.0 and 8.7 engines are incompatible. If you are migrating the jobs to the DB2 Connector, you do not need this and can remove the node from the config file. If you are still using DB2 EE stage, you will need to ensure that the 8.7 engine executables are available on the DB2 nodes.

Regards,

pras · Post by **pras** » Mon Oct 08, 2012 9:12 am

Thanks & Apologies for not responding for long time..The issue was resolved.

The installation in the DB2 nodes were wrong, Once the installation in DB2 nodes were corrected , the DB2 EE stage is working fine and also the "Orchadmin Check"

Thanks,
Prasanna