As i got information from the forms ODBC stage not good for loading bulk data.
Data stage job running 10 hrs for 30 million records. i want to increase the performance of the job.
Below i have tried the possibility options from the forms to increase the performance but still talking lot of time.
Job name: Customer_job, The job contains 4 jobs.
Note: it is not a sequencer. it is a parallel job. With in the parallel job i have 4 jobs.
Code: Select all
Job design
Sequential file stage1-->ODBC stage1(Table1)
Sequential file stage2-->ODBC stage2(Table2)
Sequential file stage3-->ODBC stage3(Table3)
Sequential file stage4-->ODBC stage4(Table4)
Code: Select all
Seq file stage Properties:
Number of Readers per node -->2
Delimeter -->comma
Quote -->double
ODBC Stage properties
Write method --> write
Write mode --> Truncate
Insert array size -->2000
partition type -->random
My Database-->SQL server. we are using only ODBC stage.
I have tried possibilities to increase the performance. But still taking 10 hours to run the job.
1. Added in the source Number of Readers per node -->2
2. Increased Insert array size is 2000 in the ODBC stage.
3. Added partition type is random
4. I have spitted the job into 4 individual jobs and executed the jobs. It is taking same amout of time.
Any help greatly appreciated.
Thanks
Suri.