surrogate key logic not working on 4 node
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 6
- Joined: Mon Feb 08, 2010 10:57 pm
surrogate key logic not working on 4 node
Hi Guys
I want to generate sequential numbers in a parallel job with configuration file using 4 nodes. I have done the following in the transformer stage.
1) Created a stage variable for the counter named Skdelta.
with the value as "@PARTITIONNUM - @NUMPARTITIONS + 1".
2) Set the derivation of the stage variable named SVRow to "Skdelta + @NUMPARTITIONS".
its working fiine in single node.
Each instance will start at a different number, eg. -1, -2, -3, -4. When the counter is incremented each instance is increment by the number of partitions, eg. 4. This gives us a sequence in instance 1 of 1, 5, 9, 13... Instance 2 is 2, 6, 10, 14... etc
can anyone suggest me any idea.....how to get the increment like 1,2,3,4 while running on 4 node and also i can't use surrogate stage over here.so, plz help me in this logic.
I want to generate sequential numbers in a parallel job with configuration file using 4 nodes. I have done the following in the transformer stage.
1) Created a stage variable for the counter named Skdelta.
with the value as "@PARTITIONNUM - @NUMPARTITIONS + 1".
2) Set the derivation of the stage variable named SVRow to "Skdelta + @NUMPARTITIONS".
its working fiine in single node.
Each instance will start at a different number, eg. -1, -2, -3, -4. When the counter is incremented each instance is increment by the number of partitions, eg. 4. This gives us a sequence in instance 1 of 1, 5, 9, 13... Instance 2 is 2, 6, 10, 14... etc
can anyone suggest me any idea.....how to get the increment like 1,2,3,4 while running on 4 node and also i can't use surrogate stage over here.so, plz help me in this logic.
Code: Select all
@NUMPARTITIONS * ( @INROWNUM - 1) + @PARTITIONNUM
You are the creator of your destiny - Swami Vivekananda
-
- Participant
- Posts: 6
- Joined: Mon Feb 08, 2010 10:57 pm
What exactly do you want it to do when it runs on multiple nodes? Typically you want the 'round robin' effect that anbu mentioned, i.e.
Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15
Node 4: 4,8,12,16
However, it seems like you want something else?![Confused :?](./images/smilies/icon_confused.gif)
Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15
Node 4: 4,8,12,16
However, it seems like you want something else?
![Confused :?](./images/smilies/icon_confused.gif)
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 6
- Joined: Mon Feb 08, 2010 10:57 pm
hi
chulett wrote:What exactly do you want it to do when it runs on multiple nodes? Typically you want the 'round robin' effect that anbu mentioned, i.e.
Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15
Node 4: 4,8,12,16
However, it seems like you want something else? :?
Hi Chulett.......i need the data to read exactly as u have metioned over here.......when i tried the logic with 4000 records it worked fine createing records like
Node 1: 1,5,9,13
Node 2: 2,6,10,14
Node 3: 3,7,11,15
but when i tried with 1 million records.......i am getting records as 1,5,9,13,....so on.........i guess all the records is passing through single node only at this point........is it because .....i have used AUTO method in transformer stage...
-
- Participant
- Posts: 6
- Joined: Mon Feb 08, 2010 10:57 pm
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
-
- Participant
- Posts: 6
- Joined: Mon Feb 08, 2010 10:57 pm