Performance of Sequential File Stage
Posted: Thu Jan 16, 2014 2:55 am
Hi,
Job Design : Sequential File --> Transformer --> Oracle Connector
Description : Source File is around 7 GB which is having 9 million records. These records has to be insert in target table.
Issue Description : Job run time is around 4 hrs which shouldn't according to business.
Hint : Job is running on one node and Records are not fixed length.
Trail and Error : In the Sequential file stage, I have tried the option 'Number of Readers per node = 2' and in the Oracle Connector stage, 'Record count = 10000 and Array Size = 20000'. In the original job, the 'Record count = 2000 and Array size = 5000'. But no use because almost same time for original job and after modification.
Please suggest something to reduce the job run time (Performance).
Job Design : Sequential File --> Transformer --> Oracle Connector
Description : Source File is around 7 GB which is having 9 million records. These records has to be insert in target table.
Issue Description : Job run time is around 4 hrs which shouldn't according to business.
Hint : Job is running on one node and Records are not fixed length.
Trail and Error : In the Sequential file stage, I have tried the option 'Number of Readers per node = 2' and in the Oracle Connector stage, 'Record count = 10000 and Array Size = 20000'. In the original job, the 'Record count = 2000 and Array size = 5000'. But no use because almost same time for original job and after modification.
Please suggest something to reduce the job run time (Performance).