Uneven partition with Surrogate key

[somu_june]

Hi

Please help me in solving this problem. I have a job which has sort stage and remove duplicates stage and surrogate key stage. Iam sorting on validfromdate which is char[8] and removing duplicates using Remove duplicate stage and Iam using hash partition and iam using surrogate key for generating cablenum . Iam using round robin partition for surrogate key and I am sending those records through transformer to dataset. In dataset Iam getting cable number right . I mean Iam getting from 1 to 70 numbers right but after 70. For example Iam getting DTcbl1.... Dtcbl70 and after 70 Iam not getting Dtcbl71 and Dtcbl72 instead Iam getting Dtcbl 77 and Dtcbl81 after Dtcbl73.

There are four partitions
node0 node1 node2 node3

2 3 4 1
6 7 8 5
.. .. .. ..
.. .. ... ..
66 67 68 65
70 69
73
77
81

This is what Iam seeing in dataset file. Iam not getting Dtcbl71 and Dtcbl72 missing and Iam getting 77 and 81 instead of those two.


Thanks,
Srinuraju.

[kwwilliams]

This job sounds familiar

Did you set your remove duplicate stage to clear the partition, before setting the next stage to round robin?

Another question would be is this data that you are going to expose to an end user? If it's not or if they aren't going to care if there are "gaps" don't worry about it. I generally use surrogate keys in dimensional data marts, and if there is a gap I really don't care.

[somu_june]

Hi Williams,

Thanks for your help . I set clear the partition in remove duplicate stage before the surrogate key stage and it worked.



Thanks,
Somaraju