Records missing in the sequential file
Posted: Tue Nov 23, 2010 11:48 pm
Hello Everyone,
I am trying to write two sets of data containg 5 records each to the same sequential file in an APPEND mode after the tranformations in a parallel job. But, my sequential file contains only 5 records after the job runs even though log shows that 5 records are consumed by each of the seq file stages. I also tried changing the job to run in single node configuration and changed all the stages execution mode to sequential without any success.
As far as I can understand this may be due to write lock of the sequential file which gets released only on completion of the job.
Please note that this scenario work perfectly in a server job when we use hashed file stage.
I am trying to write two sets of data containg 5 records each to the same sequential file in an APPEND mode after the tranformations in a parallel job. But, my sequential file contains only 5 records after the job runs even though log shows that 5 records are consumed by each of the seq file stages. I also tried changing the job to run in single node configuration and changed all the stages execution mode to sequential without any success.
As far as I can understand this may be due to write lock of the sequential file which gets released only on completion of the job.
Please note that this scenario work perfectly in a server job when we use hashed file stage.
Code: Select all
Row generator -> Transformer -> Transformer
| |
v v
Seq File Seq File
