Datastage Job performance
Posted: Tue Mar 31, 2015 11:45 pm
Hi All,
There is a datastage job we are using to fetch records from a source DB by using Teradata Connector stage (Bulk mode, invoking the Fast Export) and write it to a dataset - the volume is close to 32 million records (fetching data close to 25 GB) and the job takes close to 1 hour and 15 minutes daily. When I see the explain plan, the approximate time shown is less than a minute.
I also tried to run the query via Bteq on the unix command line - the query took less than 3 minutes to execute; but while writing to the data file, it took close to 6 hours for completion.
I even tried enabling Parallel Sync while setting the max sessions as 6, but it hasnt shown any considerable improvement either.
So, if I can take a guess, query is working considerably well. What needs to be looked into is how the writing of data can be optimized.
Is there a way how this job can be further optimized?
With best regards
Sriram
There is a datastage job we are using to fetch records from a source DB by using Teradata Connector stage (Bulk mode, invoking the Fast Export) and write it to a dataset - the volume is close to 32 million records (fetching data close to 25 GB) and the job takes close to 1 hour and 15 minutes daily. When I see the explain plan, the approximate time shown is less than a minute.
I also tried to run the query via Bteq on the unix command line - the query took less than 3 minutes to execute; but while writing to the data file, it took close to 6 hours for completion.
I even tried enabling Parallel Sync while setting the max sessions as 6, but it hasnt shown any considerable improvement either.
So, if I can take a guess, query is working considerably well. What needs to be looked into is how the writing of data can be optimized.
Is there a way how this job can be further optimized?
With best regards
Sriram