Control de max # of Job related processes running on windows
Moderators: chulett, rschirm, roy
Control de max # of Job related processes running on windows
Hello,
By the end of the next month we are going to move a new completed Phase to production environment.
This phase, a collection of jobs and job sequences, will be running over an IBM xSeries with 2 quad-core CPU and 4GB RAM.
We want to control the system workload specifically related to job processes. Each job, when running, starts one or more system processes, depending of the stages used inside of that job.
We want to set a Maximum Number of job-processes running at any given time, and to put in a waiting queue all the subsequent jobs, until some of the running processes end.
Let's say, for example, we set this maximum to 40 processes, this number gives enough room for the first 5 jobs.
We determine that the sixth job is going to start 10 new processes (by the stage type it uses), so, we put the sixth job to wait until the number of processes running gets under 30.
Is it possible to implement this? Do you have any ideas on how to?
Thank you
By the end of the next month we are going to move a new completed Phase to production environment.
This phase, a collection of jobs and job sequences, will be running over an IBM xSeries with 2 quad-core CPU and 4GB RAM.
We want to control the system workload specifically related to job processes. Each job, when running, starts one or more system processes, depending of the stages used inside of that job.
We want to set a Maximum Number of job-processes running at any given time, and to put in a waiting queue all the subsequent jobs, until some of the running processes end.
Let's say, for example, we set this maximum to 40 processes, this number gives enough room for the first 5 jobs.
We determine that the sixth job is going to start 10 new processes (by the stage type it uses), so, we put the sixth job to wait until the number of processes running gets under 30.
Is it possible to implement this? Do you have any ideas on how to?
Thank you
All things are possible, but there's nothing "out of the box" to do this. You could, however, write your own job control for this - something responsible for launching and monitoring jobs / processes and only running new jobs when things fall below your desired threshold.
Now, our illustrious Ken Bland offers a set of Job Control Utilities free from his website that could get you an enormous leg up on the process. It controls things strictly by the number of jobs running at any given time and not 'processes' but seems like you could easily mod it to include that information in the config files and use that to juggle what's running. What's nice is it has all of the fun stuff built in - dependancies, notifications, parameter assignments, metadata capture, the whole nine yards.
Or you could roll yer own.
Now, our illustrious Ken Bland offers a set of Job Control Utilities free from his website that could get you an enormous leg up on the process. It controls things strictly by the number of jobs running at any given time and not 'processes' but seems like you could easily mod it to include that information in the config files and use that to juggle what's running. What's nice is it has all of the fun stuff built in - dependancies, notifications, parameter assignments, metadata capture, the whole nine yards.
![Cool 8)](./images/smilies/icon_cool.gif)
Or you could roll yer own.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
I probably wouldn't even try. I'd consider adding an additional bit of data to the jobs list, something akin to 'weight' or how 'heavy' you think the job is, process-wise. Then use that weight, rather than the raw job count, to do your load balancing.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
We did something similar at a project in the UK once, I think we called the application the "throttler" and we also put a lot of thought into which metric we were going to use in order to decide whether or not a new Job could be started or have to continue to wait. It got complicated, there were 2 types of jobs, the lightweight type or short-running type which would not be under control and would execute immediately. In the end we used the number of processes as the threshold amount and would know (from the score) how many processes a job would use and if it exceeded the threshold it would wait. The trick was to choose the high- and low-water marks correctly so that all processes could run and that the throttling mechanism would never make the system go idle. In the end, the jobs were re-written to be more efficient and the hardware was increased.
Ok. We discussed here your answers (not those involving The Simpsons
) and decided to give a 'weight' to each job based only in our knowledge of the job design and the tests and stats we have collected so far.
Now, Craig mentioned...
Could you provide a direct link, please?
![Smile :)](./images/smilies/icon_smile.gif)
Now, Craig mentioned...
But, I could not find it.Ken Bland offers a set of Job Control Utilities free from his website that could get you an enormous leg up on the process
Could you provide a direct link, please?
There's no direct link. You need to go to his site, ask for a userid and - once granted - grab the utilities from his download area.
http://www.kennethbland.com/
http://www.kennethbland.com/
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers