Batch process using Oracle connector
Posted: Thu Mar 15, 2012 8:09 pm
Hi,
Currently we are trying to fetch 10 million records from oracle (it may grow) and to creating a flat file. In between, and need to get the description from the master using lookup.
The jobs is running fine without any problem.
But the problem is, the scratch disk is getting full (20 gb) for just 1.9 million record. It is trying to pull all the records and creating multiple datasets. This is occupying most of the scratch disk space.
Is there any option in oracle connector to process the data in batches.
Currently we are trying to fetch 10 million records from oracle (it may grow) and to creating a flat file. In between, and need to get the description from the master using lookup.
The jobs is running fine without any problem.
But the problem is, the scratch disk is getting full (20 gb) for just 1.9 million record. It is trying to pull all the records and creating multiple datasets. This is occupying most of the scratch disk space.
Is there any option in oracle connector to process the data in batches.