Page 1 of 1
Multi Instance Jobs - single node configuration - same CPU
Posted: Thu Mar 05, 2009 8:38 am
by sjordery
Hi All,
This is a bit of a theory question and I'm afraid probably reveals my ignorance about how DataStage works 'under the covers'.
Imagine I have a project that has a default config file with single node. In that project is a job sequence, that has start and end loop activities.
Inside the loop is a multi-instance job.
If the loop counter is 5, and I run the sequence, does it either:
1. Generate 5 instances of the job, each running on the same node.
2. Generate 5 instances of the job on any one available node.
3. Something else...
It is a machine with 4 dual core CPUs.
I'd like to know more about this - is there literature I can read that will help?
Many thanks in advance.
Regards,
S
Posted: Thu Mar 05, 2009 8:47 am
by eostic
...I'm not sure how your loop is functioning, as it might be serial (waiting until the job finishes before starting the next one), but if you look at the atomic pieces (EE job, with multi-instancing), you have two things going on......
a) You have a job with the potential of being run many times concurrently. Multi-Job instancing, at its simplest, is just the ability for a new job to be spawned, given its own dynamic "run time" name (<JobName.DSInvocationId> where DSInvocationId is assignable when you start it). This can be done programmatically, or just by sitting in the Director and starting the job "n" times.
b) Each of those Jobs has it's degree of parallelism, based on the config file that you have passed to it. Nodes are just logical things, as you'll see described in various threads here. A certain number of processes will be spawned for each of the Jobs you start up above. The jobs are independent of each other, although will behave similarly as far as process spawning is concerned, unless you pass differnt config files to each instance. How many processes are spawned and then managed by the OS is a larger subject, based on the number of Stages and amount of "combination" that is occurring.
Ernie
Posted: Thu Mar 05, 2009 8:57 am
by sjordery
Thanks Ernie.
The background is that the job that runs multi-instance has 3 stages:
SeqFile ---> Tfm ---> Dataset
The tfm uses RCP and a schema to basically load up a dataset.
There are 7 instances, and based on what you've said above they run sequentially I believe. The loop is a simple counter controlled one.
I was monitoring the server CPU at the time this sequence was running and saw:
Code: Select all
07:15:42 CPU %user %nice %system %iowait %irq %soft %idle intr/s
07:16:02 all 81.03 0.00 4.99 7.21 0.04 0.09 6.64 1375.35
07:16:02 0 88.10 0.00 5.90 0.55 0.05 0.00 5.40 5.35
07:16:02 1 96.80 0.00 3.10 0.00 0.00 0.00 0.10 3.75
07:16:02 2 52.50 0.00 6.40 2.85 0.00 0.00 38.25 2.25
07:16:02 3 93.35 0.00 4.65 1.85 0.00 0.00 0.15 151.45
07:16:02 4 95.10 0.00 4.10 0.05 0.00 0.00 0.80 500.20
07:16:02 5 88.00 0.00 4.80 0.80 0.00 0.05 6.40 350.05
07:16:02 6 92.70 0.00 4.80 1.25 0.00 0.00 1.20 0.00
07:16:02 7 41.65 0.00 6.25 50.40 0.25 0.60 0.85 362.20
So, looks like all 8 CPUs are being hit. I was trying to work out whether there was a chance that it is my sequence, spawning multiple jobs that were causing this, or whether other things are running at the same time.
I've put the question out locally to see who is running what, but was interested in the theory of the multi-instance jobs and their execution.
I'm trying to banish an intermittent case of the dreaded '-14 time outs' so am working through the posts related to that and server CPU load seems the most likely suspect.
Thanks again.