DataStage v6 - Row Buffering
Posted: Tue Nov 12, 2002 12:50 pm
I wanted to pass this on to DataStage v6 users. We have just upgraded from v5 to v6 and the most beneficial feature so far has been Row Buffering. Row Buffering can be set at the project level (on the server) or at the job level (job properties).
We are choosing to do it at the job level for tuning reasons. There are two types of row buffering: In Process and Inter Process. In Process should help improve performance on most jobs. Inter Process Row Buffering takes advantage of SMP architecture and spreads the workload across the different CPU's. I am not up on all of the details of how it works but we are seeing at least 50% improvement :D in run time for many different jobs. Some have gone from 2.5hrs to 32 minutes using In Process Row Buffering. Everyone's environment is different so check it out for yourself.
I detected a bug
when using Inter Process Row Buffering. It fails when a job has a SORT stage in it.
This information is based on DataStage Server running on AIX.
-Kevin
We are choosing to do it at the job level for tuning reasons. There are two types of row buffering: In Process and Inter Process. In Process should help improve performance on most jobs. Inter Process Row Buffering takes advantage of SMP architecture and spreads the workload across the different CPU's. I am not up on all of the details of how it works but we are seeing at least 50% improvement :D in run time for many different jobs. Some have gone from 2.5hrs to 32 minutes using In Process Row Buffering. Everyone's environment is different so check it out for yourself.
I detected a bug
![Crying or Very sad :cry:](./images/smilies/icon_cry.gif)
This information is based on DataStage Server running on AIX.
-Kevin