The set of available nodes for op1 is empty

[DS_TM]

Hi,
In our development server I get the below error for one of the jobs which was running fine all these days without any issues.

main_program: Fatal Error: The set of available nodes for op1 (parallel DSInput in Copy_of_Pre_CleaneData_Trf).
is empty. This set is influenced by calls to addNodeConstraint(),
addResourceConstraint() and setAvailableNodes(). If none of these
functions have been called on this operator, then the default node
pool must be empty.
This step has 5 datasets:

I used several configuration files to make this job run without any success. The configuration file is as below.

/opt/IBM/IS/InformationServer/Server/Configurations/default.apt
{
node "node1"
{
fastname "unix0489"
pools ""
resource disk "/opt/IBM/IS/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "/opt/IBM/IS/InformationServer/Server/Scratch" {pools ""}
}
node "node2"
{
fastname "unix0489"
pools ""
resource disk "/opt/IBM/IS/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "/opt/IBM/IS/InformationServer/Server/Scratch" {pools ""}
}

node "node3"
{
fastname "unix0489"
pools ""
resource disk "/opt/IBM/IS/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "/opt/IBM/IS/InformationServer/Server/Scratch" {pools ""}
}
node "node4"
{
fastname "unix0489"
pools ""
resource disk "/opt/IBM/IS/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "/opt/IBM/IS/InformationServer/Server/Scratch" {pools ""}
}

}

Your valuable inputs are very much appreciated!

[ray.wurlod]

Examine the score. Which operator is op1? Does it specify any node pool or disk pool constraint?

[DS_TM]

Examine the score. Which operator is op1? Does it specify any node pool or disk pool constraint?

Hi Ray,
Interestingly score shows node pool constraint which is not set in job design though. Not sure how it gets added compile time. When I manually removed the nodepool("conductor") from OshScript.osh, job runs fine. The issue is seen only when a BASIC transformer is used in parallel job. Please advice.

-O0 'px_VIP_26000_ETL2T_Job_working.[&"DSJobInvocationId"].Copy_of_Pre_CleaneData_Trf.AllRows_read-Output'
## General options
[ident('Copy_of_Pre_CleaneData_Trf'); jobmon_ident('Copy_of_Pre_CleaneData_Trf'); nodepool("conductor")]
## Inputs
0< [] 'Data_Set_164:DSLink162.v'
## Outputs
0> [modify (
keep
DATA_DATE,TI_GRP,TI_SEQ,DA,
SEQNO,Record_Type,COMPANY_CODE,ACCOUNTING_DATE,
TRANSACTION_DAY,CURRENCY_UNIT,ACCRUAL_REVERSAL_INDICATOR,EXT_INT_INDICATOR,
ON_OFF_BALANCE_INDICATOR,AGGREGATION_INDICATOR,ACCOUNTING_UNIT_COMMON,CP_COMPANY_CODE,
GL_ACCOUNT,GL_CUSTOMER_GROUP,DATA_DELIVERY_ID,DEBIT_CREDIT_INDICATOR,
AMOUNT_TRANSACTION_CURRENCY,ITEM_TEXT,CP_ACC_UNIT,ASSIGNMENT_CD_NO,
PROJECT_CODE,VAT_CODE,Filer;
nowarn;
)] 'Copy_of_Pre_CleaneData_Trf:AllRows_read.v'

[ray.wurlod]

Looks like this can happen when you use "CopyOf..." stage names (? and sequential mode specified). Not totally sure how.

[DS_TM]

Ray,
I also created a new parallel job row generator->basic transformer->peek which throws similar error. But when I use Node Map Constraint property to "node1" job runs fine. Your expert advice is very much appreciated!!!

Node Map Constraint is not set:
OSH script
# OSH / orchestrate script for Job Untitled12 compiled at 07:51:28 29 DEC 2010
#################################################################
#### STAGE: Row_Generator_7
## Operator
generator
## Operator options
-schema record
(
Age:int32;
)
-records 10
## General options
[ident('Row_Generator_7'); jobmon_ident('Row_Generator_7')]
## Outputs
0> [] 'Row_Generator_7:DSLink10.v'
;
#################################################################
#### STAGE: BASIC_Transformer_8
## Operator
dsjobsh
## Operator options
-p '[&"DSProjectName"]'
-h '[&"DSHostName"]:[&"DSTCPPort"]'
-e '[&"DSInstallTag"]'
-j 'Untitled12(BASIC_Transformer_8).[&"DSJobInvocationId"]'
-i record
(
Age:int32;
)
-I 'Untitled12.[&"DSJobInvocationId"].BASIC_Transformer_8.DSLink10-Input'
-o0 record
(
Age:int32;
)
-O0 'Untitled12.[&"DSJobInvocationId"].BASIC_Transformer_8.DSLink11-Output'
## General options
[ident('BASIC_Transformer_8'); jobmon_ident('BASIC_Transformer_8'); nodepool("conductor")]
## Inputs
0< [] 'Row_Generator_7:DSLink10.v'
## Outputs
0> [] 'BASIC_Transformer_8:DSLink11.v'
;
#################################################################
#### STAGE: Peek_9
## Operator
peek
## Operator options
-nrecs 10
-name
## General options
[ident('Peek_9'); jobmon_ident('Peek_9')]
## Inputs
0< [] 'BASIC_Transformer_8:DSLink11.v'
;
# End of OSH code

ERROR:
main_program: Fatal Error: The set of available nodes for op1 (parallel DSInput in BASIC_Transformer_8).
is empty. This set is influenced by calls to addNodeConstraint(),
addResourceConstraint() and setAvailableNodes(). If none of these
functions have been called on this operator, then the default node
pool must be empty.
This step has 4 datasets:
ds0: {op0[] (sequential Row_Generator_7)
eAny>eCollectAny
op1[] (parallel DSInput in BASIC_Transformer_8)}
ds1: {op1[] (parallel DSInput in BASIC_Transformer_8)
eSame>eCollectAny
op2[] (parallel DSJobRun in BASIC_Transformer_8)}
ds2: {op2[] (parallel DSJobRun in BASIC_Transformer_8)
eSame>eCollectAny
op3[] (parallel DSOutput in BASIC_Transformer_8)}
ds3: {op3[] (parallel DSOutput in BASIC_Transformer_8)
eAny>eCollectAny
op4[] (parallel Peek_9)}
It has 5 operators:
op0[] {(sequential Row_Generator_7)
}
op1[] {(parallel DSInput in BASIC_Transformer_8)
}
op2[] {(parallel DSJobRun in BASIC_Transformer_8)
}
op3[] {(parallel DSOutput in BASIC_Transformer_8)
}
op4[] {(parallel Peek_9)
}

Node Map Constraint set to node1:

OSH script
# OSH / orchestrate script for Job Untitled12 compiled at 07:54:42 29 DEC 2010
#################################################################
#### STAGE: Row_Generator_7
## Operator
generator
## Operator options
-schema record
(
Age:int32;
)
-records 10
## General options
[ident('Row_Generator_7'); jobmon_ident('Row_Generator_7')]
## Outputs
0> [] 'Row_Generator_7:DSLink10.v'
;
#################################################################
#### STAGE: BASIC_Transformer_8
## Operator
dsjobsh
## Operator options
-p '[&"DSProjectName"]'
-h '[&"DSHostName"]:[&"DSTCPPort"]'
-e '[&"DSInstallTag"]'
-j 'Untitled12(BASIC_Transformer_8).[&"DSJobInvocationId"]'
-i record
(
Age:int32;
)
-I 'Untitled12.[&"DSJobInvocationId"].BASIC_Transformer_8.DSLink10-Input'
-o0 record
(
Age:int32;
)
-O0 'Untitled12.[&"DSJobInvocationId"].BASIC_Transformer_8.DSLink11-Output'
## General options
[ident('BASIC_Transformer_8'); jobmon_ident('BASIC_Transformer_8'); nodemap ( node1 ) ]
## Inputs
0< [] 'Row_Generator_7:DSLink10.v'
## Outputs
0> [] 'BASIC_Transformer_8:DSLink11.v'
;
#################################################################
#### STAGE: Peek_9
## Operator
peek
## Operator options
-nrecs 10
-name
## General options
[ident('Peek_9'); jobmon_ident('Peek_9')]
## Inputs
0< [] 'BASIC_Transformer_8:DSLink11.v'
;
# End of OSH code

[DS_TM]

Hi Ray,
I think the issue is with the DSParams file which got corrupted for some reason. Jobs are running fine when I replaced corrupted DSParams with the working copy from a different project on the same server. I have to investigate corrupted DSParams to figure out the root cause though.

[ray.wurlod]

Curious to learn the results of your deliberation. Probably the first place to look would be the APT_CONDUCTOR_NODE environment variable definition.

[DS_TM]

Curious to learn the results of your deliberation. Probably the first place to look would be the APT_CONDUCTOR_NODE environment variable definition.

DSParams investigation revealed that PXGridGUIEnabled is set to true which generates the nodepool("conductor") text when compiling on grid-enabled systems.