irreconcilable constraints on the number of partitions

laxman.ds · Post by **laxman.ds** » Mon Jun 01, 2009 12:23 am

Dear All,
Job is aborted since there is change in config file from 3 node to 5 node.
Job has been aborted with the below error message. can any body help us to resolve this.the same job has been executed successfully with 3 node.

The number of partitions is already constrained to 5 --is this about no .of nodes in the config file?
Hist Dataset
|
JOB design : Dataset--->Change capture-->transformer----->ORACLE
|
|Reject link
|
xfm
|
ORACLE Stage R

ORACLE Stage for reject stage name is:(StgRejExpCodes)

Log of aborted job
***************************************************
Message: main_program: APT configuration file: /dspx00/Ascential/DataStage/Configurations/config5node.apt
{
node "node1_hpus51"
{
fastname "hpus51"
pools ""
resource disk "/dsxeone/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxetwo/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxethree/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefour/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefive/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxesix/Ascential/DataStage/Datasets" {pools ""}
resource scratchdisk "/dsxeone/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxetwo/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxethree/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefour/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefive/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxesix/Ascential/DataStage/Scratch" {pools ""}
}
node "node2_hpus51"
{
fastname "hpus51"
pools ""
resource disk "/dsxetwo/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxethree/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefour/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefive/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxesix/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxeone/Ascential/DataStage/Datasets" {pools ""}
resource scratchdisk "/dsxetwo/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxethree/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefour/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefive/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxesix/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxeone/Ascential/DataStage/Scratch" {pools ""}
}
node "node3_hpus51"
{
fastname "hpus51"
pools ""
resource disk "/dsxethree/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefour/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefive/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxesix/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxeone/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxetwo/Ascential/DataStage/Datasets" {pools ""}
resource scratchdisk "/dsxethree/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefour/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefive/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxesix/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxeone/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxetwo/Ascential/DataStage/Scratch" {pools ""}
}
node "node4_hpus51"
{
fastname "hpus51"
pools ""
resource disk "/dsxefour/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefive/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxesix/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxeone/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxetwo/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxethree/Ascential/DataStage/Datasets" {pools ""}
resource scratchdisk "/dsxefour/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefive/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxesix/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxeone/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxetwo/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxethree/Ascential/DataStage/Scratch" {pools ""}
}
node "node5_hpus51"
{
fastname "hpus51"
pools ""
resource disk "/dsxefive/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxesix/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxeone/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxetwo/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxethree/Ascential/DataStage/Datasets" {pools ""}
resource disk "/dsxefour/Ascential/DataStage/Datasets" {pools ""}
resource scratchdisk "/dsxefive/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxesix/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxeone/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxetwo/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxethree/Ascential/DataStage/Scratch" {pools ""}
resource scratchdisk "/dsxefour/Ascential/DataStage/Scratch" {pools ""}
}
}

Item #: 903
Event ID: 8833
Timestamp: 2009-05-31 21:29:46
Type: Fatal
User Name: dsadm
Message: main_program: Fatal Error: There are irreconcilable constraints on the number of
partitions of an operator: parallel APT_DBExportOperator in StgRejExpCodes.
The number of partitions is already constrained to 5,
but an eSame partitioned input virtual dataset produced by
parallel APT_TransformOperatorImplV8S1_StgCstm_XfmReject in XfmReject has 3.
This step has 13 datasets:
ds0: {/dspxfile/DataStage/Staging/HistCstm.ds
[pp] eSame=>eCollectAny
op0[3p] (parallel HistCstmDs)}
ds1: {/dspxfile/DataStage/Staging/Cstm.ds
eAny->eCollectAny
op1[] (parallel CstmDs)}
ds2: {op0[3p] (parallel HistCstmDs)
[pp] eSame#>eCollectAny
op2[3p] (parallel APT_TransformOperatorImplV0S22_StgCstm_XfmLimlitValue in XfmLimlitValue)}
ds3: {op1[] (parallel CstmDs)
eOther(APT_HashPartitioner { key={ value=CSTM_CUST_NB,
subArgs={ asc }
}
})>eCollectAny
op4[3p] (parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}})}
ds4: {op2[3p] (parallel APT_TransformOperatorImplV0S22_StgCstm_XfmLimlitValue in XfmLimlitValue)
[pp] eSame#>eCollectAny
op3[3p] (parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}}(0))}
ds5: {op3[3p] (parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}}(0))
[pp] eSame#>eCollectAny
op5[3p] (parallel FindDelta)}
ds6: {op4[3p] (parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}})
[pp] eSame#>eCollectAny
op5[3p] (parallel FindDelta)}
ds7: {op5[3p] (parallel FindDelta)
[pp] eSame#>eCollectAny
op6[3p] (parallel APT_TransformOperatorImplV0S2_StgCstm_XfmTransform in XfmTransform)}
ds8: {op6[3p] (parallel APT_TransformOperatorImplV0S2_StgCstm_XfmTransform in XfmTransform)
[pp] eSame#>eCollectAny
op7[3p] (parallel InsStgCstm)}
ds9: {op7[3p] (parallel InsStgCstm)
[pp] eSame#>eCollectAny
op8[3p] (parallel APT_TransformOperatorImplV8S1_StgCstm_XfmReject in XfmReject)}
ds10: {op8[3p] (parallel APT_TransformOperatorImplV8S1_StgCstm_XfmReject in XfmReject)
[pp] eSame#>eCollectAny
op9[5p] (parallel APT_DBExportOperator in StgRejExpCodes)}
ds11: {op9[5p] (parallel APT_DBExportOperator in StgRejExpCodes)
eAny#>eCollectAny
op10[5p] (parallel APT_OraWriteSubOperator in StgRejExpCodes)}
ds12: {op6[3p] (parallel APT_TransformOperatorImplV0S2_StgCstm_XfmTransform in XfmTransform)
[pp] =>
/dspxfile/DataStage/Staging/TempHistCstm.ds}
It has 11 operators:
op0[3p] {(parallel HistCstmDs)
}
op1[] {(parallel CstmDs)
}
op2[3p] {(parallel APT_TransformOperatorImplV0S22_StgCstm_XfmLimlitValue in XfmLimlitValue)
}
op3[3p] {(parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}}(0))
}
op4[3p] {(parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}})
}
op5[3p] {(parallel FindDelta)
}
op6[3p] {(parallel APT_TransformOperatorImplV0S2_StgCstm_XfmTransform in XfmTransform)
}
op7[3p] {(parallel InsStgCstm)
}
op8[3p] {(parallel APT_TransformOperatorImplV8S1_StgCstm_XfmReject in XfmReject)
}
op9[5p] {(parallel APT_DBExportOperator in StgRejExpCodes)
}
op10[5p] {(parallel APT_OraWriteSubOperator in StgRejExpCodes)
}

regards

ray.wurlod · Post by **ray.wurlod** » Mon Jun 01, 2009 12:47 am

Something in Transformer stage XfmTransform, or in a routine called from it, is explicitly specifying three-way partitioning.

preetiv · Post by **preetiv** » Mon Jun 01, 2009 1:34 am

Hi,

It might be something related to the "preserve partitioning" in Transformer. Try other options for it.

Thanks,
Preeti

miwinter · Post by **miwinter** » Mon Jun 01, 2009 3:11 am

Are node pools defined and in use?

ray.wurlod · Post by **ray.wurlod** » Mon Jun 01, 2009 6:06 am

Unless the job was created under a different configuration file no, because there are no pools mentioned in the given configuration file.

miwinter · Post by **miwinter** » Mon Jun 01, 2009 7:30 am

Good call Mr. W, schoolboy error in my hasty reply there

Looking at the score it shows all but the DB side processes running against only three nodes...

op0[3p] {(parallel HistCstmDs)
}
op1[] {(parallel CstmDs)
}
op2[3p] {(parallel APT_TransformOperatorImplV0S22_StgCstm_XfmLimlitValue in XfmLimlitValue)
}
op3[3p] {(parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}}(0))
}
op4[3p] {(parallel inserted tsort operator {key={value=CSTM_CUST_NB, subArgs={asc, cs}}})
}
op5[3p] {(parallel FindDelta)
}
op6[3p] {(parallel APT_TransformOperatorImplV0S2_StgCstm_XfmTransform in XfmTransform)
}
op7[3p] {(parallel InsStgCstm)
}
op8[3p] {(parallel APT_TransformOperatorImplV8S1_StgCstm_XfmReject in XfmReject)
}
op9[5p] {(parallel APT_DBExportOperator in StgRejExpCodes)
}
op10[5p] {(parallel APT_OraWriteSubOperator in StgRejExpCodes)

laxman.ds · Post by **laxman.ds** » Mon Jun 01, 2009 2:54 pm

Thank you all for sending quick response and suggestions

Design of my job:Before DS(3n)
DataSet(5n)----Change Capture---(XfmTransform)---Oracle

The issue has been resolved in below two ways
1. As before Data set for change capture stage created in previous day run with 3 node config file, recreated before dataset (0 records) with 5 node config file .the job went fine with out having any issues.
2. Made XfmTransform stage preserve portioning to clear

Almost all Delta identification jobs which are using datasets as before/after inputs got aborted in production. We made these jobs to execute with 3 node config file only as of now and recovered the data in production.

Please provide your comments on above resolutions, what will be the pre requisites to avoid this type of incidents if there is change in no.of nodes

Reg,

ray.wurlod · Post by **ray.wurlod** » Mon Jun 01, 2009 4:32 pm

In a phrase, "better management".

miwinter · Post by **miwinter** » Tue Jun 02, 2009 2:58 am

ray.wurlod wrote:In a phrase, "better management". ...

A-men... the holy grail of nothing ever going wrong again... ever (maybe) :D