Limit to number of multiple instance

[dickfong]

Are there any limit to the number of multiple instances fired for a job?

I have a script wrapper job which wrap a UNIX script. This job is set to allow multiple instances. Then I use a job sequencer to call this job 60 times and passing different parameters to it. Just like you call a UNIX command with 70 different parameters.

However, I found that the job sequencer called only 30 of the job instance and then finish. No error, no warning. So I suspected that there is a limit to it. Anyone any idea? Thanks in advance.

[ray.wurlod]

There is no DataStage limit to the number of instances that may be started. However, there is almost certainly a resource limitation. You shouldn't really try to start more than two instances per CPU. DataStage jobs are - intentionally - resource hogs.
Do you REALLY need that many instances simultaneously, or were you just performing a load test?
If you really do, please tell us why. If your volumes of data are huge, you should be contemplating the use of Parallel Extender, which is capable of extremely large columns - again, provided you have the resources to throw at it.

[datastage]

There is no DataStage limit to the number of instances that may be started. However, there is almost certainly a resource limitation. You shouldn't really try to start more than two instances per CPU.

Naturally it depends on what the jobs do, and the processing power of the server... I've tended to use the guideline of never run more than 4 jobs or instances per CPU, and this has worked fairly well, but I can attest to having really bad throughput and even problems with jobs or other applications hanging when running 5 or more jobs per CPU.

[Teej]

Are there any limit to the number of multiple instances fired for a job?

Technically, I am not aware of any such limitation.

I have a script wrapper job which wrap a UNIX script. This job is set to allow multiple instances. Then I use a job sequencer to call this job 60 times and passing different parameters to it. Just like you call a UNIX command with 70 different parameters.

However, I found that the job sequencer called only 30 of the job instance and then finish. No error, no warning. So I suspected that there is a limit to it. Anyone any idea? Thanks in advance.

Sounds like a limitation to me. However, do this: For EVERY job, link it to a routine that calls UtilityAbortToLog(), and trigger the link with the following code:

Code: Select all

[stage name].$JobStatus <> 1 AND [stage name].$JobStatus <> 2

What would this help you with? It would actually inform you whether a job actually failed or not. Now if the job was never started... that's something I will have to investigate when I gets better (maybe tomorrow. I'm down and out with the flu and in between nap-times).

However, 60 TIMES?! Dayum! You probably would be better off building a small shell script calling dsjob directly to that job pulling the parameters from a text file, or in fact use Ken Bland's sequencer solution for this instead of trying to keep track of 60 equally-looking stages within the GUI environment. (120 using my suggestion even!)

-T.J.

[chulett]

On top of what everyone else already wrote, some of these answers will be tempered by the design of your Job Sequence. Typically, a Sequence will be setup to run jobs serially or I've set some up to run a handful of jobs simultaneously, squeeze them thru a Sequence stage set to 'All' and then run another handful. It would help if you explained how yours was built. Are you really trying to run 60 jobs at once? This is implied in your mention of Multiple Instances, but needs to be clarified, I believe.

Also, how are your triggers setup between stages? This can effect how jobs flow down your sequence and can cause a stream to stop - that is why they are there, after all. Sequences can stop because a job has not ended happily but you won't see naught but green entries in the Sequences logs, even the ones that report a controlled job aborted. That's one reason TJ suggested using 'AbortToLog'. Are you saying that when your runs there are no errors in any of your logs - the Sequence or any of the controlled jobs?

[ray.wurlod]

There is no DataStage limit to the number of instances that may be started.

Technically there is a limit, because the job ifentifier is a character string literal, and the maximum length of one of these is 255 characters. Therefore the DataStage limit on the maximum number of instances of a multi-instance job is the maximum possible number of character strings that are 255 characters long or shorter, excluding "".
You can do the arithmetic - it's a big number!