Same partition with Two Nodes

[ravij]

Hi ,

When I define 2 nodes in Same Partition the Job is getting aborted n with single node its working fine.

The error log I got is

Code: Select all

main_program: Fatal Error: There are irreconcilable constraints on the number of
partitions of an operator: parallel Sur_Key_For_Same_Partition.
The number of partitions is already constrained to 2,
but an eSame partitioned input virtual dataset produced by
 sequential Src_SeqFile has 1.
This step has 2 datasets:
ds0: {op0[1p] (sequential Src_SeqFile)
      eSame<>eCollectAny
      op1[2p] (parallel Sur_Key_For_Same_Partition)}
ds1: {op1[2p] (parallel Sur_Key_For_Same_Partition)
      >>eCollectAny
      op2[1p] (sequential APT_RealFileExportOperator in Tgt_SeqFile)}
It has 3 operators:
op0[1p] {(sequential Src_SeqFile)
    }
op1[2p] {(parallel Sur_Key_For_Same_Partition)
    }
op2[1p] {(sequential APT_RealFileExportOperator in Tgt_SeqFile)

If its the case how can I achieve the Parallelism with Same Partition?
Any answer can be appreciated.
Thanks in advance.

[ray.wurlod]

Please post the configuration file. Without this it's difficult to judge. For example, what exactly do you mean by "two nodes in the same partition"?

Further, be aware that Sequential File stages can only sustain parallel operation under very particular circumstances. This is not the same as "multiple readers per node".

[ameyvaidya]

Hi Ravij,
this part of the posted error message has a clue:

Code: Select all

There are irreconcilable constraints on the number of
partitions of an operator: parallel Sur_Key_For_Same_Partition.
The number of partitions is already constrained to 2,
but an eSame partitioned input virtual dataset produced by
 sequential Src_SeqFile has 1

A Sequential file reading a single file runs Sequentially(1 partition only). After the dataset is read, it is partitioned ( to a 2 node config in your case).

Since the "same" partitioning method is employed just after the sequential file in this job, no partitioning can take place. Hence the error.

In the stage after the sequential file, partition the data; after this stage, the method can be kept "same".

[ravij]

Hi Ray,

This is my Configuration file which contains 2 nodes.

Code: Select all

 {
	node "node2"
	{
		fastname "PROXY-1"
		pools "DefPool" 
		resource disk "C:/Ascential/DataStage/Datasets" { pools "" "pool1" }
		resource scratchdisk "C:/Ascential/DataStage/Scratch" { pools "" "pool2" }
	}
node "node3"
	{
		fastname "PROXY-1"
		pools ""
		resource disk "C:/Ascential/DataStage/Datasets" { pools "" "pool1" }
		resource scratchdisk "C:/Ascential/DataStage/Scratch" { pools "" "pool1" "pool2"}
	}


}

Any answer can be appreciated.
thanks in advance.

[kumar_s]

HI,
Is that you have played with No of Readers per Node option in Sequential file?

-Kumar

[ray.wurlod]

OK, I now see that you mean "two identically defined processing nodes". Though unusual, and not optimally efficient, this is legal and not the source of your problem.
It appears that the virtual Data Set on the link between the srcSeqFile stage and the Surrogate Key Generator stage contains some kind of incompatibility with what the latter expects.
Can you post the generated OSH? This will help to confirm this theory or otherwise.

[kumar_s]

Hi,

Pls let me know what is the poit/benifit of mantaing single node as two virutal node.
Ultimatly the single node is going to process for the both partition

-Kumar

[ray.wurlod]

It's obliging two processes to be created for each stage operating in parallel mode. Presumably to force the use of two CPUs, even though they're sharing not only memory but also disk and scratch disk resources.