Performance issue during Run

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ravibabu
Participant
Posts: 39
Joined: Tue Feb 13, 2007 12:18 am
Location: vijayawada

Post by ravibabu »

My dear friend ,pls split the job as small as possible.this method is very useful for error fixing in the future...
kcbland
Participant
Posts: 5208
Joined: Wed Jan 15, 2003 8:56 am
Location: Lutz, FL
Contact:

Post by kcbland »

If the Designer response is slow, is it while the job is running? Are you monitoring the node that hosts the DS Engine? (prstat, topas, top, glance, etc).

If that node is heavily burdened, don't you think that your GUI, which is communicating to processes on that node to retrieve design information from the repository which physically resides on that node and uses the same resources as jobs, would be impacted by that processing?
Kenneth Bland

Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
dsedi
Participant
Posts: 220
Joined: Wed Jun 02, 2004 12:38 am

Post by dsedi »

ravibabu wrote:My dear friend ,pls split the job as small as possible.this method is very useful for error fixing in the future...
Thanks my friend!
this design already has 40 + joins - 100 + stages involved so while designing i have splited as 3 piece's already..which i should join in future( our aim is to reduce DISK I/O)
kcbland wrote:If the Designer response is slow, is it while the job is running? Are you monitoring the node that hosts the DS Engine? (prstat, topas, top, glance, etc).
No.Let say when i open the DS connection first time and Edit something the response is fine,during RUn time also it's fine (job is Running at normal optimal response) but once after Run...all the response on my GUI is become very slow.
kcbland wrote: If that node is heavily burdened, don't you think that your GUI, which is communicating to processes on that node to retrieve design information from the repository which physically resides on that node and uses the same resources as jobs, would be impacted by that processing?
I got you.as you said may be becoz of the complexity of the job, after run the node may be heavily burdened and right after the Run when i access design From GUI , it may be slow becoz of resource sharing.
but how to overcome this?
(FYI..I see my config file with 8 nodes)

Thanks,Edi
ravibabu
Participant
Posts: 39
Joined: Tue Feb 13, 2007 12:18 am
Location: vijayawada

Post by ravibabu »

hi,

can you tell me ,You are useing SMP or MPP systems.

I think problem with resource sharing better to use MPP system.
lack of resource problem.
dsedi
Participant
Posts: 220
Joined: Wed Jun 02, 2004 12:38 am

Post by dsedi »

ravibabu wrote:hi,

can you tell me ,You are useing SMP or MPP systems.

I think problem with resource sharing better to use MPP system.
lack of resource problem.
Sorry & Please don't post Offtopic's which will mislead the discussion!

Thanks,Edi
kcbland
Participant
Posts: 5208
Joined: Wed Jan 15, 2003 8:56 am
Location: Lutz, FL
Contact:

Post by kcbland »

You can have 1000 nodes, all that matters to the GUI is the Server node which houses the repository. Since it could also be the coordinator node, it's likely that when this node is heavily utilized you will experience poor GUI response time when opening/closing/designing jobs.

You need to be watching the Server node's resources. If on AIX, run topas and look at the disk and CPU utilization. In fact, leave this running ALL THE TIME YOU ARE WORKING. It is your primary measure of performance. When you notice a slow GUI response time, peep over at topas (glance, prstat, top, etc).

If you see that the disks are heavily utilized, then that is probably the issue. If you see the disks are okay but the cpus are busy, then that's the culprite. Don't assume low cpu usage is good, check the disks!!

If you see low cpu and disk and are having poor GUI experiences, then the only remaining variable is network. If you're running topas and have it refreshing every 1 seconds, check to see if that refresh is happening quickly. I do this all of the time as I use VPN a lot. When I see my prstat session "stall", I know my GUIs are going to be hanging as well. Once I see my telnet sessions clean up their screens I know my GUIs are going to respond.
Kenneth Bland

Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Post Reply