CPU resource utilization

[senthilmp]

Hi,
Am facing some issue with performance of the job. The CPU resource on the server on which the target table database(Oracle) resides is not been well utilized by the Datastage job. i.e the Server on which the Database resided is alwasy 60% idle but still the Datastage job is not using the idle resources.

Is their any particular setting in datastage to request more CPU resources?

My job design is extract data from a table , some transformation is applied and update/insert is done to target table.

Can you pls help me out on getting more CPU resources?

[chulett]

No, there's no magic setting for that. You may, however, be able to make the job "multi-instance" and run multiple copies, each handling a portion of the data simultaneously.

[senthilmp]

Ok Craig, Thanks

[ArndW]

If the %CPU use on the DataStage system during job run is not above 80% then your job is not CPU bound, but most likely I/O bound.

[chulett]

Right, it's all about the job design and the resources you are leveraging - disk, database, etc.

[muruganr117]

If the %CPU use on the DataStage system during job run is not above 80% then your job is not CPU bound, but most likely I/O bound. ...

Hello,

That was a wonderful piece of information i found after long search.
"Thanks ArndW!!!"
I would like dig further to know the execution process of Server jobs.

Scenario -
> I am working on DS7.1r2 Server , Unix
> Production server has 16physical CPU's,its Sun Solaris M8000 series
> I have a job running on 8 instances , there is no much transformation involved,
I/P HASH file(100fields) looks up from 39 hash files based on KEY and write to O/P seq file(200)length 5000byte per record
> Each instance handles nearly 500,000records

--Item to understand
Even if there are 16CPU's and CPU IDLE time is 60% ~ 70%
each instance is processing with less than 20% CPU as seen in director and process around 400~500 records per second.

I would like to know any method of increasing utilization of resources to improve performance. Any other suggestions with regard to design change of this job is also welcome.

Thank you very much for your time!!!

Really appreciate Chullet, ArndW efforts...

regards

[ArndW]

(I'm depressed, I wrote a long post and it has disappeared, and now I'm out of steam to re-write it...)

Don't touch your job or system until you know where your bottleneck is. You will need to measure and analyse what your system is doing before making any changes; otherwise you might cause everything to slow down instead of speeding up.

With the CPU load described it is clear that your system is not constrained by processing power at present. Memory is probably not an issue, since paging/swapping conditions usually will increase your %CPU significantly.

That leaves Disk I/O or, in the case of a SAN, network I/O. I don't know which flavour of UNIX you have to recommend a measuring tool, but the standard UNIX command iostat will give you a rough idea of I/O rates and distribution.

[muruganr117]

Thank you very much ArndW

We are using Sun solaris 5.10(is this the UNIX flavor ?) , normally i do verify prstat, vmstat which i think more of CPU related, i shall understand & verify iostat and get back

Thank you very much once again!!!

(I'm depressed, I wrote a long post and it has disappeared, and now I'm out of steam to re-write it...)

Don't touch your job or system until you know where your bottleneck is. You will need to measure and analyse what your system is doing before making any changes; otherwise you might cause everything to slow down instead of speeding up.

With the CPU load described it is clear that your system is not constrained by processing power at present. Memory is probably not an issue, since paging/swapping conditions usually will increase your %CPU significantly.

That leaves Disk I/O or, in the case of a SAN, network I/O. I don't know which flavour of UNIX you have to recommend a measuring tool, but the standard UNIX command iostat will give you a rough idea of I/O rates and distribution.