Huge Number of Colums -- Performance Issue
Posted: Mon Jan 19, 2009 11:53 pm
My Source is Oracle and my target is also Oracle, but both are in different servers.
In one of the source table, we have around 200 Columns and the data is also huge (around 7 Million). We need to extract the data incrementally, and we dont have timestamp column to identify the newly inserted or modified rows. So we deploy CRC logic to identify the new or modified rows.
As the number of columns and the data is huge, identifying the incremental rows taking so much time.
I did some analysis, and found , we dont have problem while writing to the target, and it is only when fetching from the Source.
Irrespective of Array Size, the number of rows fetched are very less. Now i am working on splitting the job into multiple instances. But it is failing with unknown reasons in some instances, like Communication failure.
Please provide some inputs on tuning the job.
In one of the source table, we have around 200 Columns and the data is also huge (around 7 Million). We need to extract the data incrementally, and we dont have timestamp column to identify the newly inserted or modified rows. So we deploy CRC logic to identify the new or modified rows.
As the number of columns and the data is huge, identifying the incremental rows taking so much time.
I did some analysis, and found , we dont have problem while writing to the target, and it is only when fetching from the Source.
Irrespective of Array Size, the number of rows fetched are very less. Now i am working on splitting the job into multiple instances. But it is failing with unknown reasons in some instances, like Communication failure.
Please provide some inputs on tuning the job.