Can anyone list the best practice in performance tuning of datastage jobs? This is to get knowledge and we can share all the tips we know.
Thank you.
Performance Tuning Tips
Moderators: chulett, rschirm, roy
Tuning is, in my opinion, a two-phase iterative process. For some reason, many think that there are ready-made solutions to tuning, sort of like a magic "take two aspirins and call me in the morning", or "raise MFILES and all your problems will be gone".
1. Measure.
The first step must always be to measure your system and identify your problem. Just stating that "The job runs slow" isn't going to help. Narrow it down as far as possible. Usually it will be either CPU, memory, Disk I/O, or Network I/O; other problems exist (i.e. synchronization, locking, etc.) but the ones listed above account for over 99% of typical problems.
2. Identify.
After you know what your performance bottleneck is, you have to work at identifying the source. This is where "art" comes into play, as well as a good fundamental understanding of how software (including the OS) works. This can be the most difficult phase of performance tuning.
3. Rectify.
Once the problem has been identified it must be rectified. Usually this is the easiest hase, but it can also be the toughest to fix, particularly if the cause is a fundamental design issue.
After phase 3 is finished, you need to iterate through the whole thing again, since the source of the performance bottleneck has in all likelihood changed. These three steps are repeated as often as practicable. I like to use the Pareto rule 20% of the jobs will account for 80% of the resources. Once you've taken care of the 80% your gains won't be as much. The first couple of cycles will usually give a very high net gain, but each subsequent pass will give you less and less return on your investment of time and effort. Knowning when to stop (and perhaps just suggest "more hardware") is also a skill.
1. Measure.
The first step must always be to measure your system and identify your problem. Just stating that "The job runs slow" isn't going to help. Narrow it down as far as possible. Usually it will be either CPU, memory, Disk I/O, or Network I/O; other problems exist (i.e. synchronization, locking, etc.) but the ones listed above account for over 99% of typical problems.
2. Identify.
After you know what your performance bottleneck is, you have to work at identifying the source. This is where "art" comes into play, as well as a good fundamental understanding of how software (including the OS) works. This can be the most difficult phase of performance tuning.
3. Rectify.
Once the problem has been identified it must be rectified. Usually this is the easiest hase, but it can also be the toughest to fix, particularly if the cause is a fundamental design issue.
After phase 3 is finished, you need to iterate through the whole thing again, since the source of the performance bottleneck has in all likelihood changed. These three steps are repeated as often as practicable. I like to use the Pareto rule 20% of the jobs will account for 80% of the resources. Once you've taken care of the 80% your gains won't be as much. The first couple of cycles will usually give a very high net gain, but each subsequent pass will give you less and less return on your investment of time and effort. Knowning when to stop (and perhaps just suggest "more hardware") is also a skill.
</a>-
nirav_2004
- Participant
- Posts: 7
- Joined: Fri Mar 20, 2009 9:09 pm
Re: Performance Tuning Tips
Hi,
Recently we had performance problems for our v8.1 on windows 2k3, jobs were failing with out of processor time and memory, after searching a lot I found following article:-
http://www-01.ibm.com/support/docview.w ... wg21318253
Tips helped us tuning the server.
Recently we had performance problems for our v8.1 on windows 2k3, jobs were failing with out of processor time and memory, after searching a lot I found following article:-
http://www-01.ibm.com/support/docview.w ... wg21318253
Tips helped us tuning the server.