Hi,
There are 100 parallel jobs which needs to be run. They can all run independently as their source + target tables are different in each job.
The records processed would be bit on the higher side and not sure on how many jobs can be run simultaneously in a sequence.
Is there a way in which I can kick off 5 jobs first then if 1 of them finshes then push 1 from the waiting list to start and so on? By this at any point I would have 5 runs in execution or please let me know if there are any other options.
Thanks for your time.
Thanks.
Running multiple jobs
Moderators: chulett, rschirm, roy
I have seen that done. It's a bit more complex but is possible. I have not had to setup a queue like that myself. I would want the limit (i.e. 5) to be a parameter, or at least easily changed. For example, you may find that 4 or 10 is an ideal limit.
A simple option is to create a sequence job with 5 separate sets of 20 jobs each. Each of the 20 jobs are linked to run one after the other. The 5 sets automatically run in parallel to each other because they're not linked to each other. This option is less flexible when it comes to changing your limit. It also requires some manual balancing of the longer running jobs across sets.
A simple option is to create a sequence job with 5 separate sets of 20 jobs each. Each of the 20 jobs are linked to run one after the other. The 5 sets automatically run in parallel to each other because they're not linked to each other. This option is less flexible when it comes to changing your limit. It also requires some manual balancing of the longer running jobs across sets.
Choose a job you love, and you will never have to work a day in your life. - Confucius
DataStage's built-in scheduler would not be able to perform this type of scheduling. This is a task that is a perfect fit for any of several workload scheduler products on the market.
You may be able to pull it off using a job sequence as Eric suggests or, more than likely, a BASIC routine which could loop through a list of stored dsjob commands and poll the status of currently active jobs.
Regards,
You may be able to pull it off using a job sequence as Eric suggests or, more than likely, a BASIC routine which could loop through a list of stored dsjob commands and poll the status of currently active jobs.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
Yes, we did something like this, using a Unix script.
Basically, as James suggested, the script loops through a set of dsjob commands.
e.g.
1. Build a file with a record for each of the parameters to be used for the dsjob command.
2. Loop for each of the records.
2a. Build & submit the dsjob command, without waiting for a status (in our case, we submitted the command as a background task, using the "&" suffix).
At this point, you could include another loop, e.g. if the maximum number of concurrent jobs is reached, it waits until one of the jobs has finished.
3. Loop until all of the jobs have finished.
3a. Check the status of any of the completed jobs
(in our case, our jobs wrote their status to a file when they finished, so the script kept monitoring the file, then set the script to fail if any jobs failed).
Basically, as James suggested, the script loops through a set of dsjob commands.
e.g.
1. Build a file with a record for each of the parameters to be used for the dsjob command.
2. Loop for each of the records.
2a. Build & submit the dsjob command, without waiting for a status (in our case, we submitted the command as a background task, using the "&" suffix).
At this point, you could include another loop, e.g. if the maximum number of concurrent jobs is reached, it waits until one of the jobs has finished.
3. Loop until all of the jobs have finished.
3a. Check the status of any of the completed jobs
(in our case, our jobs wrote their status to a file when they finished, so the script kept monitoring the file, then set the script to fail if any jobs failed).
