I have searched the forum and did maximum possible thing from my understanding to solve the issue. But I couldn't. Please have a look.
We have the following warning message in the lookup stage
Code: Select all
ReferAttributes: When checking operator: Operator of type "APT_LUTProcessOp": will partition despite the
preserve-partitioning flag on the data set on input port 1.
1) Made the up stream stage to clear the preserve partition flag
2) Cleared the preserve partition flag of the stage
3) Tried to set up the down stream stage to propagate partition.
Nothing has helped to take them out. So I designed a simple job to read the data from two Oracle Enterprise stage and partition the data based on the same key
Code: Select all
Oracle______
|
|
|
----------
| LKP |-------------- Dataset
-----------
|
|
|
Oracle _____|
Please find the Job Score
Code: Select all
main_program: This step has 7 datasets:
ds0: {op0[1p] (sequential C_DYN_PART_CONFIG)
eOther(APT_HashPartitioner { key={ value=ASSMBL_CD,
subArgs={ cs }
},
key={ value=IPC_REF_CD,
subArgs={ cs }
}
})<>eCollectAny
op3[4p] (parallel buffer(0))}
ds1: {op1[1p] (sequential C_PART_CONFIG)
eOther(APT_HashPartitioner { key={ value=ASSMBL_CD,
subArgs={ cs }
},
key={ value=IPC_REF_CD,
subArgs={ cs }
}
})<>eCollectAny
op2[4p] (parallel APT_LUTCreateOp in ReferAttributes)}
ds2: {op2[4p] (parallel APT_LUTCreateOp in ReferAttributes)
[pp] eEntire#>eCollectAny
op6[4p] (parallel APT_CombinedOperatorController:APT_LUTProcessOp in ReferAttributes)}
ds3: {op2[4p] (parallel APT_LUTCreateOp in ReferAttributes)
[pp] eSame=>eCollectAny
op6[4p] (parallel APT_CombinedOperatorController:APT_LUTProcessOp in ReferAttributes)}
ds4: {op3[4p] (parallel buffer(0))
[pp] eSame=>eCollectAny
op6[4p] (parallel APT_CombinedOperatorController:APT_LUTProcessOp in ReferAttributes)}
ds5: {op4[4p] (parallel delete data files in delete test33.ds)
>>eCollectAny
op5[1p] (sequential delete descriptor file in delete test33.ds)}
ds6: {op6[4p] (parallel APT_CombinedOperatorController:Data_Set_3)
=>
test33.ds}
It has 7 operators:
op0[1p] {(sequential C_DYN_PART_CONFIG)
on nodes (
node1[op0,p0]
)}
op1[1p] {(sequential C_PART_CONFIG)
on nodes (
node2[op1,p0]
)}
op2[4p] {(parallel APT_LUTCreateOp in ReferAttributes)
on nodes (
node1[op2,p0]
node2[op2,p1]
node3[op2,p2]
node4[op2,p3]
)}
op3[4p] {(parallel buffer(0))
on nodes (
node1[op3,p0]
node2[op3,p1]
node3[op3,p2]
node4[op3,p3]
)}
op4[4p] {(parallel delete data files in delete test33.ds)
on nodes (
node1[op4,p0]
node2[op4,p1]
node3[op4,p2]
node4[op4,p3]
)}
op5[1p] {(sequential delete descriptor file in delete test33.ds)
on nodes (
node1[op5,p0]
)}
op6[4p] {(parallel APT_CombinedOperatorController:
(APT_LUTProcessOp in ReferAttributes)
(Data_Set_3)
) on nodes (
node1[op6,p0]
node2[op6,p1]
node3[op6,p2]
node4[op6,p3]
)}
It runs 19 processes on 4 nodes.
Code: Select all
main_program: Echo:
oraread
-dboptions '{user=DM,password=******}'
-query 'SELECT ASSMBL_CD, IPC_REF_CD, PART_NO_OEM, MANUFACT_REF, PART_NO_SDESC, APPLICABILITY_DESC, INTERCHANGE_ORD, PART_BASELINE_CD FROM C_DYN_PART_CONFIG_SB_747'
-server 'MXIDM'
[ident('C_DYN_PART_CONFIG'); jobmon_ident('C_DYN_PART_CONFIG')]
0> [modify (
ASSMBL_CD:nullable string[max=8]=ASSMBL_CD;
IPC_REF_CD:nullable string[max=50]=IPC_REF_CD;
PART_NO_OEM:nullable string[max=40]=PART_NO_OEM;
MANUFACT_REF:nullable string[max=16]=MANUFACT_REF;
PART_NO_SDESC:nullable string[max=80]=PART_NO_SDESC;
APPLICABILITY_DESC:nullable string[max=4000]=APPLICABILITY_DESC;
INTERCHANGE_ORD:nullable decimal[4,0]=INTERCHANGE_ORD;
PART_BASELINE_CD:nullable string[max=80]=PART_BASELINE_CD;
keep
ASSMBL_CD,IPC_REF_CD,PART_NO_OEM,MANUFACT_REF,
PART_NO_SDESC,APPLICABILITY_DESC,INTERCHANGE_ORD,PART_BASELINE_CD;
)] 'C_DYN_PART_CONFIG:GetDynParts.v'
;
oraread
-dboptions '{user=DM,password=******}'
-query 'SELECT ASSMBL_CD, IPC_REF_CD, IPC_REF_NAME, ATA_CD, INV_CLASS_CD, ASSMBL_BOM_ZONE_CD, POS_CT, POS_NAME_LIST, NH_IPC_REF_CD, PART_GRP_APPLICABILITY, LRU_BOOL FROM C_PART_CONFIG_SB_747'
-server 'MXIDM'
[ident('C_PART_CONFIG'); jobmon_ident('C_PART_CONFIG')]
0> [modify (
ASSMBL_CD:nullable string[max=8]=ASSMBL_CD;
IPC_REF_CD:nullable string[max=50]=IPC_REF_CD;
IPC_REF_NAME:nullable string[max=100]=IPC_REF_NAME;
ATA_CD:nullable string[max=50]=ATA_CD;
INV_CLASS_CD:nullable string[max=8]=INV_CLASS_CD;
ASSMBL_BOM_ZONE_CD:nullable string[max=40]=ASSMBL_BOM_ZONE_CD;
POS_CT:nullable decimal[10,0]=POS_CT;
POS_NAME_LIST:nullable string[max=4000]=POS_NAME_LIST;
NH_IPC_REF_CD:nullable string[max=50]=NH_IPC_REF_CD;
PART_GRP_APPLICABILITY:nullable string[max=4000]=PART_GRP_APPLICABILITY;
LRU_BOOL:nullable string[max=1]=LRU_BOOL;
keep
ASSMBL_CD,IPC_REF_CD,IPC_REF_NAME,ATA_CD,
INV_CLASS_CD,ASSMBL_BOM_ZONE_CD,POS_CT,POS_NAME_LIST,
NH_IPC_REF_CD,PART_GRP_APPLICABILITY,LRU_BOOL;
)] 'C_PART_CONFIG:GetPartsData.v'
;
hash -key ASSMBL_CD -cs -key IPC_REF_CD -cs
[ident('ReferAttributes.GetDynParts_Part')]
0< [] 'C_DYN_PART_CONFIG:GetDynParts.v'
0> [] 'C_DYN_PART_CONFIG:GetDynParts_Part.v'
;
hash -key ASSMBL_CD -cs -key IPC_REF_CD -cs
[ident('ReferAttributes.GetPartsData_Part')]
0< [] 'C_PART_CONFIG:GetPartsData.v'
0> [] 'C_PART_CONFIG:GetPartsData_Part.v'
;
lookup
-table
-key ASSMBL_CD
-key IPC_REF_CD
-ifNotFound fail
[ident('ReferAttributes'); jobmon_ident('ReferAttributes')]
0< [] 'C_DYN_PART_CONFIG:GetDynParts_Part.v'
1< [] 'C_PART_CONFIG:GetPartsData_Part.v'
0> [-pp; modify (
keep
ASSMBL_CD,IPC_REF_CD,IPC_REF_NAME,ATA_CD,
INV_CLASS_CD,ASSMBL_BOM_ZONE_CD,POS_CT,POS_NAME_LIST,
NH_IPC_REF_CD,PART_GRP_APPLICABILITY,LRU_BOOL,PART_NO_OEM,
MANUFACT_REF,PART_NO_SDESC,APPLICABILITY_DESC,INTERCHANGE_ORD,
PART_BASELINE_CD;)] 'ReferAttributes:SendDynParts.v'
;
same
[ident('Data_Set_3.SendDynParts_Part')]
0< [] 'ReferAttributes:SendDynParts.v'
0> [] 'ReferAttributes:SendDynParts_Part.v'
;
copy
[ident('Data_Set_3')]
0< [] 'ReferAttributes:SendDynParts_Part.v'
0>| [ds] 'test33.ds'
;
While reading the other posts I feel this shouldn't happen? Am I missing some thing? Please advise/help