Control de max # of Job related processes running on windows

yserrano · Post by **yserrano** » Tue Oct 14, 2008 9:04 am

Hello,
By the end of the next month we are going to move a new completed Phase to production environment.
This phase, a collection of jobs and job sequences, will be running over an IBM xSeries with 2 quad-core CPU and 4GB RAM.

We want to control the system workload specifically related to job processes. Each job, when running, starts one or more system processes, depending of the stages used inside of that job.
We want to set a Maximum Number of job-processes running at any given time, and to put in a waiting queue all the subsequent jobs, until some of the running processes end.

Let's say, for example, we set this maximum to 40 processes, this number gives enough room for the first 5 jobs.
We determine that the sixth job is going to start 10 new processes (by the stage type it uses), so, we put the sixth job to wait until the number of processes running gets under 30.

Is it possible to implement this? Do you have any ideas on how to?

Thank you

chulett · Post by **chulett** » Tue Oct 14, 2008 9:27 am

All things are possible, but there's nothing "out of the box" to do this. You could, however, write your own job control for this - something responsible for launching and monitoring jobs / processes and only running new jobs when things fall below your desired threshold.

Now, our illustrious Ken Bland offers a set of Job Control Utilities free from his website that could get you an enormous leg up on the process. It controls things strictly by the number of jobs running at any given time and not 'processes' but seems like you could easily mod it to include that information in the config files and use that to juggle what's running. What's nice is it has all of the fun stuff built in - dependancies, notifications, parameter assignments, metadata capture, the whole nine yards.

Or you could roll yer own.

kduke · Post by **kduke** » Tue Oct 14, 2008 11:09 am

Just to figure out what processes are tied to one job is very difficult. I tried to do this in UNIX with a shell script. It is very hard. Some processes have the job name in them. Not all do though. I would bet this is lots harder to do in Windows.

chulett · Post by **chulett** » Tue Oct 14, 2008 11:51 am

I probably wouldn't even try. I'd consider adding an additional bit of data to the jobs list, something akin to 'weight' or how 'heavy' you think the job is, process-wise. Then use that weight, rather than the raw job count, to do your load balancing.

kduke · Post by **kduke** » Tue Oct 14, 2008 6:24 pm

We were trying to calculate a weight based on amount RAM used by each job. We were paging so we wanted to run just enough jobs not to page.

ArndW · Post by **ArndW** » Wed Oct 15, 2008 1:22 am

We did something similar at a project in the UK once, I think we called the application the "throttler" and we also put a lot of thought into which metric we were going to use in order to decide whether or not a new Job could be started or have to continue to wait. It got complicated, there were 2 types of jobs, the lightweight type or short-running type which would not be under control and would execute immediately. In the end we used the number of processes as the threshold amount and would know (from the score) how many processes a job would use and if it exceeded the threshold it would wait. The trick was to choose the high- and low-water marks correctly so that all processes could run and that the throttling mechanism would never make the system go idle. In the end, the jobs were re-written to be more efficient and the hardware was increased.

chulett · Post by **chulett** » Wed Oct 15, 2008 8:42 am

ArndW wrote:In the end, the jobs were re-written to be more efficient and the hardware was increased.

So, after that you stopped throttling your children?

ArndW · Post by **ArndW** » Wed Oct 15, 2008 8:59 am

We didn't want to be child-killers, we just throttled them a bit; sort of like Homer does to Bart

yserrano · Post by **yserrano** » Wed Oct 15, 2008 9:00 am

Hello,

that's very interesting and I have taken my notes regarding methrics. I would like to know more specificly what methods you used to "throttle" the jobs.

Regards

chulett · Post by **chulett** » Wed Oct 15, 2008 9:37 am

Don't start them.

chulett · Post by **chulett** » Wed Oct 15, 2008 9:38 am

ArndW wrote:We didn't want to be child-killers, we just throttled them a bit; sort of like Homer does to Bart

Why you little...

yserrano · Post by **yserrano** » Fri Oct 17, 2008 10:14 am

Ok. We discussed here your answers (not those involving The Simpsons

) and decided to give a 'weight' to each job based only in our knowledge of the job design and the tests and stats we have collected so far.

Now, Craig mentioned...

Ken Bland offers a set of Job Control Utilities free from his website that could get you an enormous leg up on the process

But, I could not find it.
Could you provide a direct link, please?

chulett · Post by **chulett** » Fri Oct 17, 2008 10:43 am

There's no direct link. You need to go to his site, ask for a userid and - once granted - grab the utilities from his download area.

http://www.kennethbland.com/