surrogate key logic not working on 4 node

rakeshvayu · Post by **rakeshvayu** » Mon Aug 02, 2010 4:24 pm

Hi Guys

I want to generate sequential numbers in a parallel job with configuration file using 4 nodes. I have done the following in the transformer stage.

1) Created a stage variable for the counter named Skdelta.
with the value as "@PARTITIONNUM - @NUMPARTITIONS + 1".
2) Set the derivation of the stage variable named SVRow to "Skdelta + @NUMPARTITIONS".

its working fiine in single node.

Each instance will start at a different number, eg. -1, -2, -3, -4. When the counter is incremented each instance is increment by the number of partitions, eg. 4. This gives us a sequence in instance 1 of 1, 5, 9, 13... Instance 2 is 2, 6, 10, 14... etc

can anyone suggest me any idea.....how to get the increment like 1,2,3,4 while running on 4 node and also i can't use surrogate stage over here.so, plz help me in this logic.

anbu · Post by **anbu** » Mon Aug 02, 2010 4:30 pm

Code: Select all

@NUMPARTITIONS * ( @INROWNUM - 1) +  @PARTITIONNUM

rakeshvayu · Post by **rakeshvayu** » Mon Aug 02, 2010 9:09 pm

Hi Anbu.........i tried with the logic as you have mentioned......but it is still incrementing by 4 while running on 4 node..........is there any other way possible............anyway thanks for the reply.

chulett · Post by **chulett** » Mon Aug 02, 2010 9:20 pm

What exactly do you want it to do when it runs on multiple nodes? Typically you want the 'round robin' effect that anbu mentioned, i.e.

Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15
Node 4: 4,8,12,16

However, it seems like you want something else?

rakeshvayu · Post by **rakeshvayu** » Mon Aug 02, 2010 9:35 pm

chulett wrote:What exactly do you want it to do when it runs on multiple nodes? Typically you want the 'round robin' effect that anbu mentioned, i.e.

Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15
Node 4: 4,8,12,16

However, it seems like you want something else? :?

Hi Chulett.......i need the data to read exactly as u have metioned over here.......when i tried the logic with 4000 records it worked fine createing records like
Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15

but when i tried with 1 million records.......i am getting records as 1,5,9,13,....so on.........i guess all the records is passing through single node only at this point........is it because .....i have used AUTO method in transformer stage...

kris007 · Post by **kris007** » Mon Aug 02, 2010 9:41 pm

What you are seeing is the data from the first partition/node in your view data result set. Just for the sake of it, after the transformer stage put a sort stage and sort your data on the key and then try viewing the data. That should make you feel better

rakeshvayu · Post by **rakeshvayu** » Mon Aug 02, 2010 9:50 pm

Hi Kris.........ya i did that too..........in my job .......i have used the tail stage too see the max of surrogate key generated.......like if i am expecting max num to be 100.......but i am getting as 400.......which is a mutiple of 4.....
:roll:

priyadarshikunal · Post by **priyadarshikunal** » Tue Aug 03, 2010 4:08 am

check your partitioning scheme before transformer.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Aug 03, 2010 5:12 am

Is there any constraint in the flow that may restrict data from other nodes ?

rakeshvayu · Post by **rakeshvayu** » Tue Aug 03, 2010 2:59 pm

Hello friends

Thanks for all ur valuable suggestion........its really helped me alot to go through my logic..........i just used this logic which is working fine now...

@PARTITIONNUM + ( @NUMPARTITIONS * (@INROWNUM - 1) ) + 1

even on 4 node..........

Thanks,
Rakesh.