Load Balancing

[psluser]

Hi,

We are using a LoadBalancing routine to run "N" jobs in parallel from a given list of jobs. Given below is the code snippet for the same.

The code below achieves the following functionality :- It starts N jobs in parallel but only after all the N jobs finish execution it takes the next set of "N" jobs to run.

We want to modify the code such that as soon as one among the "N" job finished execution we want to start the next job in sequence.

Please let us know how can this be achieved by modifying the existing code given below.

Thanks,
* Array holding the jobs to run.
DIM SubFileRunListArray(30)
MAT SubFileRunListArray=0
DIM JobHandleArray(ParallelJobsToRun)
MAT JobHandleArray=0

* Setup to run "N" parallel jobs
Loop Until SubFileRunListArray(i) = 0
For j=1 To ParallelJobsToRun Step 1
If JobHandleArray(j)=0
Then
vJobName=SubFileRunListArray(i)
JobHandleArray(j) = DSAttachJob(vJobName, DSJ.ERRNONE)

JobHandleArray(j) = DSPrepareJob(JobHandleArray(j))

If NOT(JobHandleArray(j)) Then
Call DSLogWarn("Job Attach Failed: " : vJobName, "LoadBalancing")
End

ErrCode = DSRunJob(JobHandleArray(j), DSJ.RUNNORMAL)

i=i+1
end
NEXT j



* Check if any of the jobs started above is finished. In that case make that jobs handle as null. Then start the next jobs in seq.
For k=1 To ParallelJobsToRun Step 1
JobCurrStatus=DSGetJobInfo(JobHandleArray(k),DSJ.JOBSTATUS)
JobName=DSGetJobInfo(JobHandleArray(k),DSJ.JOBNAME)

If JobCurrStatus=DSJS.RUNNING
Then
Call DSLogInfo("Running ":JobName, "LoadBalancing")
ErrCode=DSWaitForJob(JobHandleArray(k))
JobHandleArray(k)=0
End
If JobCurrStatus=DSJS.RUNFAILED Or JobCurrStatus=DSJS.CRASHED
Then
Call DSLogWarn("Aborted ":JobName, "LoadBalancing")
JobHandleArray(k)=0
End

If JobCurrStatus=DSJS.RUNOK Or JobCurrStatus=DSJS.RUNWARN
Then
Call DSLogInfo ("Finished ":JobName, "LoadBalancing")
JobHandleArray(k)=0
End
Next k

LeftJobs = ctrJobsToRun - i
If LeftJobs < ParallelJobsToRun
Then
ParallelJobsToRun = LeftJobs +1
End

Repeat

[DS_SUPPORT]

Ken Bland has a nice utility in his website, for doing this.

if you want to change your code, the first thing you have to remove is DSWaitForJob(JobHandleArray(k)), it only make the control to wait until the job finishes.

And other thing is, i dont see DSDetach anywhere in your code. Have you posted the entire code or a part of your code? How the variable vJobName is assigned? I have commented the DSWaitForJob , and added JobHandleArray(j) = 0 when the job is not attached. I didnt even test this in the DS environment, i have just modified your code , take this as a starting point and proceed

Code: Select all

* Array holding the jobs to run. 
DIM SubFileRunListArray(30) 
MAT SubFileRunListArray=0 
DIM JobHandleArray(ParallelJobsToRun) 
MAT JobHandleArray=0 

* Setup to run "N" parallel jobs 

Loop Until SubFileRunListArray(i) = 0 
	For j=1 To ParallelJobsToRun Step 1 
		If JobHandleArray(j)=0 
		Then 
			vJobName=SubFileRunListArray(i) 
			JobHandleArray(j) = DSAttachJob(vJobName, DSJ.ERRNONE) 
			JobHandleArray(j) = DSPrepareJob(JobHandleArray(j)) 
			If NOT(JobHandleArray(j)) Then 
				Call DSLogWarn("Job Attach Failed: " : vJobName, "LoadBalancing") 
				JobHandleArray(j) = 0
			End 
			ErrCode = DSRunJob(JobHandleArray(j), DSJ.RUNNORMAL) 

			i=i+1 
		end 
	NEXT j 



	* Check if any of the jobs started above is finished. In that case make that jobs handle as null. Then start the next jobs in seq. 
	For k=1 To ParallelJobsToRun Step 1 
		JobCurrStatus=DSGetJobInfo(JobHandleArray(k),DSJ.JOBSTATUS) 
		JobName=DSGetJobInfo(JobHandleArray(k),DSJ.JOBNAME) 

		If JobCurrStatus=DSJS.RUNNING 
		Then 
			*Call DSLogInfo("Running ":JobName, "LoadBalancing") 
			*ErrCode=DSWaitForJob(JobHandleArray(k)) 
			*JobHandleArray(k)=0 
		End 
		If JobCurrStatus=DSJS.RUNFAILED Or JobCurrStatus=DSJS.CRASHED 
		Then 
			Call DSLogWarn("Aborted ":JobName, "LoadBalancing") 
			JobHandleArray(k)=0 
			LeftJobs = ctrJobsToRun - i 
		End 

		If JobCurrStatus=DSJS.RUNOK Or JobCurrStatus=DSJS.RUNWARN 
		Then 
			Call DSLogInfo ("Finished ":JobName, "LoadBalancing") 
			JobHandleArray(k)=0 
			LeftJobs = ctrJobsToRun - i 
		End 
	Next k 

	*LeftJobs = ctrJobsToRun - i 
	If LeftJobs < ParallelJobsToRun 
	Then 
		ParallelJobsToRun = LeftJobs +1 
	End 

Repeat 

[psluser]

Hi,

Thanks for the response.

If we dont use DSWait function, how can we be sure that the job which has been started has completed. Because there are chances that the controlling routine will finish before all the jobs have been completed.

Alternative is in the end i can check if all JobHandleArray has the value '0' to make sure that all jobs have been completed.

I have used "JobHandleArray(k)=0" in place of Detach job after the job completes(success or failure). Please confirm if this is the right approach or do i need to use DSDetach explicitly.

vJobName takes the values from SubFileRunListArray which will contain the jobs to run.

[DS_SUPPORT]

Yes, dont let your routine to complete until all the jobhandles are 0.

The current loop will execute until all the jobs are assigned, so start a loop after this, which will execute untill all jobhandles are 0. I would request to stop the process for a small amount of time by using SLEEP, instead of checking continiously.

[chulett]

You cannot 'WaitForJob' to get the result you want. You need to loop through your list of jobs, starting whatever you can (without a wait) until you hit the limit. Then keep looping though the list, checking each running job to see if it has completed and starting new jobs when the running count drops below your limit. Exit the loop when all jobs have been run.