HI All,
The Datastage server is running slow because of a particular design.
Earlier we had a non-multi instance datastage job calling
a multinstance job. Now we changed the application design to
multi-instance datastage job calling another multi-instance job.
This causes the number of dsapi in the unix box to run upto 500
when only four such jobs are running.
Can anyone tell how to solve this by keeping the multi-instance feature.
Regards
Sreenivasulu
Datastage running slow due to multi instance issues
Moderators: chulett, rschirm, roy
-
Sreenivasulu
- Premium Member
- Posts: 892
- Joined: Thu Oct 16, 2003 5:18 am
The only solution you need is to do some better design, this is not a problem with single- or multi-instace jobs but at a lower level. At a very abstract level you might say that it takes 5000 seconds do to a job. If you split this up into 500 jobs that each need only 10 seconds you will get a box flooded with processes and it will still take 5000 wall-clock seconds for these jobs to run. On bigger SMP systems you can reduce this since job can truly run in parallel; but you will still have a lot of processes (this is part of what PX does in order to run so quickly).
A multi-instance job calling another multi-instance still is only 2 job processes. So you must be doing quite a bit more to get so many jobs running at the same time, and slowly.
A multi-instance job calling another multi-instance still is only 2 job processes. So you must be doing quite a bit more to get so many jobs running at the same time, and slowly.
</a>Running more jobs on the same machine without increasing cpus means things will run slower. What's the problem? Are you running any diagnostic tools to show you cpu, memory, and disk utilization?
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Processes dsapi_server and dsapi_slave have nothing whatsoever to do with running jobs; rather these are the agent processes for connected DataStage clients.
Depending on version, server jobs run as uvsh or dssh, while processes in parallel jobs run as lots of osh processes - one conductor, one section leader per processing node, and one for each uncombined operator.
You need to find out why the dsapi processes are not being cleaned up. What exact version of DataStage are you running? Do you auto-start the deadlock daemon?
Depending on version, server jobs run as uvsh or dssh, while processes in parallel jobs run as lots of osh processes - one conductor, one section leader per processing node, and one for each uncombined operator.
You need to find out why the dsapi processes are not being cleaned up. What exact version of DataStage are you running? Do you auto-start the deadlock daemon?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
Sreenivasulu
- Premium Member
- Posts: 892
- Joined: Thu Oct 16, 2003 5:18 am