Hi All,
There is a datastage job we are using to fetch records from a source DB by using Teradata Connector stage (Bulk mode, invoking the Fast Export) and write it to a dataset - the volume is close to 32 million records (fetching data close to 25 GB) and the job takes close to 1 hour and 15 minutes daily. When I see the explain plan, the approximate time shown is less than a minute.
I also tried to run the query via Bteq on the unix command line - the query took less than 3 minutes to execute; but while writing to the data file, it took close to 6 hours for completion.
I even tried enabling Parallel Sync while setting the max sessions as 6, but it hasnt shown any considerable improvement either.
So, if I can take a guess, query is working considerably well. What needs to be looked into is how the writing of data can be optimized.
Is there a way how this job can be further optimized?
With best regards
Sriram
Datastage Job performance
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 96
- Joined: Fri Apr 17, 2009 5:58 am
- Location: Delhi
-
- Participant
- Posts: 32
- Joined: Fri Nov 07, 2014 11:18 am
In the job that writes to the Data Set stage, experiment with using different numbers of nodes in the config file. How many nodes is it currently running on?
You may also want to check with your server administrator and/or storage team, if you have one. Find out exactly how slow those disks are writing.
You may also want to check with your server administrator and/or storage team, if you have one. Find out exactly how slow those disks are writing.
Choose a job you love, and you will never have to work a day in your life. - Confucius