Semaphores / batch queues
Posted: Thu Sep 29, 2005 12:35 am
I have a job sequence that kicks off a large number of independent jobs. There are no links between the job activities, because there are no dependencies.
What I want is some mechanism to limit the load placed on the system. Ideally I'd like functionality such as the Unix at/batch/cron command that restricts the number of simultaneous jobs on a given queue. When one job finishes, another starts.
Things I have considered:
* Add links or bust the Job Sequence into multiple linked sequences. Not bad, but sub-optimal use of resources because I won't always have the maximum number of jobs running.
* Increment/decrement a number myself in a hash file - not nice as I don't like my chances of handling aborts.
* I could use a before/after-job-subroutine to acquire/release a semaphore. I've seen some references to semaphores in this forum, but cannot find the right manual to describe them properly. It sounds like I would have to nominate which lock number each job was to use, rather than allocating a pool of semaphores which every job could use.
* I've got some C programs that implement sem_init, sem_wait, and sem_post as executables you can call from a shell. I could call these from a before/after-job-subroutine - not too sure how they would react to aborts.
* I could get the Job Sequence to initiate the jobs via routines and the dsrun command using the Unix batch queue. Ugly, but it would work.
Any ideas for more elegant solutions?
What I want is some mechanism to limit the load placed on the system. Ideally I'd like functionality such as the Unix at/batch/cron command that restricts the number of simultaneous jobs on a given queue. When one job finishes, another starts.
Things I have considered:
* Add links or bust the Job Sequence into multiple linked sequences. Not bad, but sub-optimal use of resources because I won't always have the maximum number of jobs running.
* Increment/decrement a number myself in a hash file - not nice as I don't like my chances of handling aborts.
* I could use a before/after-job-subroutine to acquire/release a semaphore. I've seen some references to semaphores in this forum, but cannot find the right manual to describe them properly. It sounds like I would have to nominate which lock number each job was to use, rather than allocating a pool of semaphores which every job could use.
* I've got some C programs that implement sem_init, sem_wait, and sem_post as executables you can call from a shell. I could call these from a before/after-job-subroutine - not too sure how they would react to aborts.
* I could get the Job Sequence to initiate the jobs via routines and the dsrun command using the Unix batch queue. Ugly, but it would work.
Any ideas for more elegant solutions?