Page 1 of 1

Problem while using Head stage

Posted: Tue Aug 04, 2009 9:50 am
by datisaq
From the source i have 20 records say 1 to 20.And i'm having a 4 node configuration file.I used the round robin partitioning mechanism, so my output should be

node0 node1 node2 node3
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
I given the output to a head stage, i specified the following options:-

No of rows per partition=10
All Partitions=True
I got the proper output as mentioned above.

If i specify the property:-
No of rows per partition=10
All Partitions=False
Partition Number=1

I got the output as:-
5 6 7 8
But the output shouldbe
2 6 10 14 18
since this are the records passed to the node1.

Please let me know if i'm wrong..

Need your valuable inputs...

Posted: Tue Aug 04, 2009 10:25 am
by ArndW
Sounds like you've discovered a potential bug in the stage.

Posted: Tue Aug 04, 2009 10:30 am
by datisaq
Thanks Arndw for your prompt reply.
Could you please check this at your end and confirm this?

Posted: Tue Aug 04, 2009 11:15 am
by ArndW
Sorry, no time at the moment but perhaps someone else might have time. I've been here at work 12 hours and am working on other problems (for which I'm being paid)

Posted: Tue Aug 04, 2009 1:51 pm
by Sainath.Srinivasan
Try hash or modulus partition and check the result.

Posted: Tue Aug 04, 2009 8:25 pm
by datisaq
For Hash and modulus partitioning it's working fine but for round robin method the output is incorrect.
Following are the results:-

For Hash:-

node0 node1 node2 node3
9 5 6 1
18 12 8 2
19 14 7 10
16 4
13 20
3
17
11
15


Output of Head stage(Correct Result):-
5 12 14 16 13 3 17 11 15


For Modulus:-

node0 node1 node2 node3
4 1 2 3
8 5 6 7
12 9 10 11
16 13 14 15
20 17 18 19

Output of Head stage(Correct Result):-
1 5 7 9 13 17

For Round Robin:-
node0 node1 node2 node3
1 2 3 4 ---> Row0
5 6 7 8 ---> Row1
9 10 11 12 ---> Row2
13 14 15 16 ---> Row3
17 18 19 20 ---> Row0

Output of Head stage(InCorrect Result):-
5 6 7 8
Expecting output from node1 that's:- 2 6 10 14 18

If i change the partition number=2 then the output is:-
9 10 11 12
If partition number=0 then the output is:-
1 2 3 4
17 18 19 20

So it's working rowwise(horizontally) for round robin based on the configuration file.

Please be cautious while using round robin method with Head stage.
      [quote][/quote]

      head Stage working fine for Round Robin partitioning

      Posted: Wed Aug 05, 2009 12:41 am
      by ethelvina
      It is working fine.
      I used Rowgenerator-Sortstage(RoundRobin Partition)-Head Stage
      Try this

      Posted: Wed Aug 05, 2009 12:54 am
      by datisaq
      But the stage shlould work for all the scenarios..

      Posted: Wed Aug 05, 2009 1:25 am
      by datisaq
      The problem is resolved now.
      Actually my job design is :

      rowgen---->copy--->Headstage--->peek

      At copy stage i given earlier partioning method as "Auto" since it's a sequential to parallel process from row generator to copy stage and at Head Stage it's "round robin".Because of "Auto" partioning i got in correct output.

      When i changed the partioning method as "Same" at input of copy stage i got correct output.

      Thanks all for your help...