Memory and CPU Usage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Lotus26
Premium Member
Premium Member
Posts: 48
Joined: Tue Jul 13, 2004 2:09 pm

Memory and CPU Usage

Post by Lotus26 »

HI

We are using Datastage Parallel Extender and I am just wondering like is there any way I can determine how much CPU time the jobs are utilizing and how much memory they are using etc things .

I would appreciate any input .

Thanks.
Regards
Lotus26
sud
Premium Member
Premium Member
Posts: 366
Joined: Fri Dec 02, 2005 5:00 am
Location: Here I Am

Re: Memory and CPU Usage

Post by sud »

Please do a search, this topic has been discussed many times and you will find valuable information. To begin with, use the job monitor in datastage and also checkout unix Top command. Other utilities like free also gives usage info if recorded through the span of job run. Usually your unix admin will be able to give you very good info about the utilities they have installed on unix and which one can be used for what purpose.

Also, you can also set the environmental variable APT_PM_PLAYER_TIMING in the job to get the CPU timings per player. Or, set APT_PERFORMANCE_DATA to a folder path where you want the performance data to be written out.
It took me fifteen years to discover I had no talent for ETL, but I couldn't give it up because by that time I was too famous.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

And when you get to version 8 there is a whole suite of monitoring and resource estimation tools available in the product, with which you can exhaust all the toner cartridges in your color printer!
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
dsdeveloper13
Participant
Posts: 10
Joined: Thu Aug 28, 2008 12:40 am

Post by dsdeveloper13 »

Hi All,
I am trying to read the actual CPU usage versus the system clock duration for the jobs and stages. Using the APT_PERFORMANCE_DATA option, the generated information is showing the time elapsed and CPU usage to be the same. ( i would think that the CPU usage would be less than the time elapsed).
Are there any other methods by which the CPU usage can be captured?

Also, does anyone have any pointers on how to capture the Memory usage information generated by the DataStage application? ( besides the unix tools like top etc)

Thanks in Advance..
DSDev13.
AmeyJoshi14
Participant
Posts: 334
Joined: Fri Dec 01, 2006 5:17 am
Location: Texas

Post by AmeyJoshi14 »

You can use APT_PM_PLAYER_MEMORY parameter. :)
Form the Advance Parallel Job Developer Guide:

Code: Select all

APT_PM_PLAYER_MEMORY
This environment variable is available in the DataStage Administrator
under the Parallel --> Reporting branch. Setting this variable causes
each player process to report the process heap memory allocation in
the job log when returning.
http://findingjobsindatastage.blogspot.com/
Theory is when you know all and nothing works. Practice is when all works and nobody knows why. In this case we have put together theory and practice: nothing works. and nobody knows why! (Albert Einstein)
Post Reply