Surrogate key issue

[Devendrudu]

Hi,

In my job i am using the Surrogate_Key_Generator stage when i run first time the sequence values are getting properly like 1,2,3,4 but when i run second time with one more new insert record i am getting 1,2,3,4,6 not as 5.

for every run it is skipping one sequence value.

first run:
source
-------
cid,name
1,xx
2,yy
3,ww
4,bb

out put
----------

seq_num,cid,name
1,1,xx
2,2,yy
3,3,ww
4,4,bb

second run
--------
source
-------
cid,name
1,xx
2,yy
3,ww
4,bb
5,cc

out put
----------

seq_num,cid,name
1,1,xx
2,2,yy
3,3,ww
4,4,bb
6,5,cc

note: Surrogate Source_type = flat file
User specified block size =1
every time i am running in single node only.

please suggest me what options i need to select to get proper sequence numbers.

[priyadarshikunal]

I think we are missing something here as it will have a proper pattern while generating the keys not like missing just 1 value.

Are you generating the sequence first and then filtering out records from the stream?

[ray.wurlod]

How are you partitioning these data? Is each partition processing exactly the same number of rows? If not, expect gaps.