IPC & Manageing resource contention
Moderators: chulett, rschirm, roy
IPC & Manageing resource contention
We have an 8 node, dual core server. Assuming:
a) no other jobs are running
c) we have a simple passive-active-passive stage structure in our test job
d) IPC is enabled
How many processes will be started for the test job?
Assuming the first test job is running, if a second test job (test2) with the similar stucture is submitted, what is the impact to test2 with regards to server resoruces?
tks
a) no other jobs are running
c) we have a simple passive-active-passive stage structure in our test job
d) IPC is enabled
How many processes will be started for the test job?
Assuming the first test job is running, if a second test job (test2) with the similar stucture is submitted, what is the impact to test2 with regards to server resoruces?
tks
It depends upon how your kernal is set up. Ideally, it should invoke the second processor. To get a better understanding of this, I suggest you sit with your SA and ask him/her at the OS level how processes are managed.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You're operating on an incorrect assumption. The number of processes available is not governed by the number of CPUs - as proof just look at the Processes tab of Task Manager on your PC - there are probable a couple of hundred processes even thought there's only one CPU.pajj wrote:So with 8 dual node cpus, there are 16 processes available?
Will the server dynamically adjust the number of processes when the second job is submitted or will test1 keep all the processes until it is finished?
One of the functions of the operating system is to allocate each process that needs CPU a share of the available time. With most applications (including DataStage) you do not control which CPU gets which task - through its life a particular process may execute on any subset of the available CPUs - though only on one at a time unless it has multiple threads.
The whole thing is automatic and you don't need to worry about a thing until the point when there is too much demand for CPU (and other resources), at which point you back off a little on the demand side.
Operating system tuning is a complex art, and one that takes considerable time to learn to do well. It is definitely not something that can be taught through a DataStage forum.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
In our project, we are trying to include IPC stage in each job. is it a good approach? Assuming i have 40 jobs, will there be any resource contention if all the 40 jobs are running in parallel? and if the volume of data in a job is just say 10,000, should we include IPC or a job without transformers is fine?
Aim high
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Putting an IPC stage in every job will indeed create extra processes - that is its function. As for "kill the CPU" that depends on total load, but other posts from you and from your site suggest that you are doing that already.
In short - everything in moderation. I suspect someone went on a class or read in a manual that "IPC stage will improve performance" and made a decision to include them everywhere. Garfield once famously asserted "diet kitty munchies? 20 packets of them and I should be thin as a rake!".
In short - everything in moderation. I suspect someone went on a class or read in a manual that "IPC stage will improve performance" and made a decision to include them everywhere. Garfield once famously asserted "diet kitty munchies? 20 packets of them and I should be thin as a rake!".
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
I am not doing that currently but thats an idea which is flowing in my project and i am totally against it. So wanted few points to prove that using IPC in every job is not a good approach. Can you Point me to any manual so that i will have the concrete proof.ray.wurlod wrote:but other posts from you and from your site suggest that you are doing that already.
".
Aim high
There's no manual that will say "don't add one of these to every job". The Inter-Process Stages are documented in the Server Job Developer's Guide in Chapter 12, but all that's going to tell you how much of a wonderful job it will do speeding all your jobs up.
The only way you'll get "concrete proof" is to make the change and observe the results.
The only way you'll get "concrete proof" is to make the change and observe the results.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers