Controlling the quantity of jobs running

gpbarsky · Post by **gpbarsky** » Tue May 27, 2003 3:56 pm

Hi.

How can I control the quantity of job that are triggered in the Director ?

I've built a process that has the ability of triggering several jobs at the same time, but I want to control the quantity of jobs that are running simultaneously.

Does anyone know how to do this ?

Thanks in advance.
[8)]

ray.wurlod · Post by **ray.wurlod** » Wed May 28, 2003 3:54 am

You do this in code.
Create a job sequence and compile it, and observe how it maintains arrays of information about jobs that are running.
You can count the number of items in such an array (whether you create your own or use one created by a job sequence) in order to determine whether to start any more jobs. (Then what do you do?)

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

kduke · Post by **kduke** » Wed May 28, 2003 9:56 pm

gpbarsky

If you are on UNIX then you can shell out and do a

ps -ef | grep jobname

if they are all the same job. This is a little tougher if you are on Windows. You could probably run a TCL command LIST.READU EVERY or PORT STATUS. If you need exact syntax then email me.

Thanks Kim.

Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com

gpbarsky · Post by **gpbarsky** » Tue Sep 23, 2003 9:39 am

Kim:

I am refreshing this topic because I need this solution. I thought there is a command (within DS BASIC) that stores all the job handlers that I am running.

In summary, I need a way to control the quantity of running jobs. I don't have the paralell extender. I just have DS 5.2, and myself.

Thanks in advance.

Guillermo P. Barsky
Buenos Aires - Argentina

kduke · Post by **kduke** » Tue Sep 23, 2003 10:44 am

Guillermo

Ray's way requires all of the jobs to be kicked off from one batch job. It is very easy that way. You can do the same thing in a sequence by creating sequences to slow things down. Have 5 jobs point to a sequence and wait for these to finish before starting the next set of jobs. If you want the ps version then it would look something like this.

MyLimit = 5
JobName = "MyJob"
Cmd = 'SH -c "ps -ef | grep ':JobName:' | grep -v grep | wc -l"'
execute Cmd capturing OutRec
NoOfProcesses = OutRec + 0
if NoOfProcesses

gpbarsky · Post by **gpbarsky** » Tue Sep 23, 2003 11:23 am

Kim:

Thank for your answer, but I don't understand what it does.

First of all, I am running the DS on a Windows 2000 platform. I don't have a Unix system.

As you said in your note, it is the same for me to control a "set of jobs". For example, to run 5 jobs and when the last is finished the I can run another set of 5 jobs.

I remember that there is a BASIC sentence that holds all the job handlers that I pass to it. Do you know this name ? Or do you have another method to control the quantity of running jobs ?

Thanks again.

[:)][:)][:)][:)][:)]

Guillermo P. Barsky
Buenos Aires - Argentina

kduke · Post by **kduke** » Tue Sep 23, 2003 12:03 pm

Guillermo

You can do it in a batch job. A batch job is a job with no stages. All it has is code. It only runs other jobs. If you go into the code section there is a drop down list of jobs. If you select a job and tell it to add the job then it will insert the code to run that job. If you take the wait for job routines and place them together at the bottom then all those jobs will run at the same time. Next start your next group or as one job finishes then start a new one.

It is easy you can figure it out.

Kim.

Kim Duke
DsWebMon - Monitor DataStage over the web
www.Duke-Consulting.com

ray.wurlod · Post by **ray.wurlod** » Tue Sep 23, 2003 4:40 pm

(Advertising, perhaps)

If you can't figure it out you can create a specification of exactly what you want it to do and hire a competent consultant to create the job control code for you.
There are several competent consultants who post on this forum and elsewhere.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

msigal · Post by **msigal** » Wed Sep 24, 2003 7:56 am

What if you have multiple users and many, many jobs that could be run? Is there a way to setup the server to que up jobs that users kick off so one doesn't have to always submit a job through a controller? We wouldn't want to set up another layer like that with as many jobs as we have and each with the potential to have various job parameters.

We're running 5.2.2 on AIX.

Thanks,
Myles

kduke · Post by **kduke** » Wed Sep 24, 2003 8:57 am

Myles

You would have to create a process to run all the jobs. You could do this in BASIC. Append the job and the parameter values to a sequential file and run these jobs 5 or 6 at a time or whatever. As one job finishes then start the next one in the list. Should be easy to write.

Kim.

Kim Duke
DsWebMon - Monitor DataStage over the web
www.Duke-Consulting.com

msigal · Post by **msigal** » Wed Sep 24, 2003 9:20 am

We do something similar to what you suggest, but there are restrictions on its use. We currently have a auto-conversion process in place that detects the receipt of a file and kicks off an associated job. It uses a web interface and is written in java using DataStage API. It has an internal control that keeps track of how many jobs it has running and in que, but doesn't check the server to see how many total jobs are running. This process requires some set up, but becomes fully automated after setup. I was hoping for some sort of ini file or the like on the server side that we could set a limit, but it doesn't sound like that's an option.

Thanks for info.

Myles

Myles Sigal
Technical Analyst
Thomson - Medstat
777 E. Eisenhower - 435B
Ann Arbor, MI 48108

myles.sigal@medstat.com
734-913-3466