main_program: Fatal Error: There are irreconcilable constraints on the number of
partitions of an operator: parallel Sur_Key_For_Same_Partition.
The number of partitions is already constrained to 2,
but an eSame partitioned input virtual dataset produced by
sequential Src_SeqFile has 1.
This step has 2 datasets:
ds0: {op0[1p] (sequential Src_SeqFile)
eSame<>eCollectAny
op1[2p] (parallel Sur_Key_For_Same_Partition)}
ds1: {op1[2p] (parallel Sur_Key_For_Same_Partition)
>>eCollectAny
op2[1p] (sequential APT_RealFileExportOperator in Tgt_SeqFile)}
It has 3 operators:
op0[1p] {(sequential Src_SeqFile)
}
op1[2p] {(parallel Sur_Key_For_Same_Partition)
}
op2[1p] {(sequential APT_RealFileExportOperator in Tgt_SeqFile)
If its the case how can I achieve the Parallelism with Same Partition?
Any answer can be appreciated.
Thanks in advance.
Please post the configuration file. Without this it's difficult to judge. For example, what exactly do you mean by "two nodes in the same partition"?
Further, be aware that Sequential File stages can only sustain parallel operation under very particular circumstances. This is not the same as "multiple readers per node".
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
There are irreconcilable constraints on the number of
partitions of an operator: parallel Sur_Key_For_Same_Partition.
The number of partitions is already constrained to 2,
but an eSame partitioned input virtual dataset produced by
sequential Src_SeqFile has 1
A Sequential file reading a single file runs Sequentially(1 partition only). After the dataset is read, it is partitioned ( to a 2 node config in your case).
Since the "same" partitioning method is employed just after the sequential file in this job, no partitioning can take place. Hence the error.
In the stage after the sequential file, partition the data; after this stage, the method can be kept "same".
Amey Vaidya<i>
I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.</i>
<i>- Douglas Adams</i>
OK, I now see that you mean "two identically defined processing nodes". Though unusual, and not optimally efficient, this is legal and not the source of your problem.
It appears that the virtual Data Set on the link between the srcSeqFile stage and the Surrogate Key Generator stage contains some kind of incompatibility with what the latter expects.
Can you post the generated OSH? This will help to confirm this theory or otherwise.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Pls let me know what is the poit/benifit of mantaing single node as two virutal node.
Ultimatly the single node is going to process for the both partition
It's obliging two processes to be created for each stage operating in parallel mode. Presumably to force the use of two CPUs, even though they're sharing not only memory but also disk and scratch disk resources.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.