SeqFile----LookupStage----Xformer----DB2APIStage
This job is failing because of the error...
We have enough space and memory. We have faced the same issue on similar jobs which I changed the Lookup stage with Join stage andLukup,0: Could not map table file "/datastage/prd/Datasets/lookuptable.20110318.feldurb (size 1134819352 bytes)": Not enough space
Error finalizing / saving table /datastage/tmp/dynLUT3199226b80c2818
they ran fine. But I want to use Bulk Load stage for this for 2 reasons.
1. Bulk load will be faster.
2. If I use too many jobs with Join stgae then it will hurt other running jobs for memory.
I don't want to do sparse lookup as the input will be around 2 million.
Lookup will have around 17 million records. This is for restartability.
We use DB2 sequencers to genrate our surrogate keys and we use the nextval command in our insert statement. but If I use Bulk Load I can't do that. How do I handle this situation ?
I can query the seq max value and then insert that in transformer + @outrownum but now I have to reset the DB2 sequencer after the job is done. so I waw wonderinbg if there is anyother way...
Appreciate any input...
Thanks