Triggering Multiple instances job multiple times parallely

[mail2hfz]

Hello All,

Can someone guide me, "how to approach" on the following scenario.

I have a sequence which in turn invokes routine activity, 3 datastage jobs and 3 command activties (unix korn shells). The input for the sequence is flat file and expected output after all the steps is flat file. I might have a single file or multiple files as input when the batch is started. Now, the question is, if there are mutliple files then how to trigger the sequence multiple times parallely. I am able to achieve this sequentially by running the jobs in loop but not in parallel.

Thanks..

[swarnkar]

Hi,

You have to make your top sequencer and all the jobs, as multi-instance.

Then through shell script or Datastage routine you have to run this top level sequncer multiple times, with proper invocation ids.

You can search this forum for dsjob -run commnand, if you are invoking your sequencer through shell script.

Thanks,
Nitin Swarnkar

[mail2hfz]

How do you run your top level sequencer multiple times parallely with different invocations. In a shell script, I can loop thru all the files and trigger my sequence but that's only one file at a time.

[swarnkar]

If your top level sequencer is a Multi-instance job, then you can mention invocation-id with job name.

[priyadarshikunal]

This will call the job for each file in the directory with invocation ID same as file name.

Code: Select all

ls -1 | while read output; 
do 
#echo $output 
dsjob -run  <.... JobName.$output .....>
done

[swarnkar]

Word of Caution :

Your logic should have some mechanism to control the number of job instance you invoke. People have faced issued when there are too many instances invoked.

Here "Too Many" could be 25 or more.

All the best !!!!!

Thanks,
Nitin Swarnakr

[mail2hfz]

Well, I am not sure if my post is not clear. I am looking for a triggering mechanism that is capable of running my dsjob/multi instance sequence paralley depending on the number of files.

Eg: 3 input files are available, run the sequence in 3 instances at the same time. I will have timestamp/jobname as my invocation id.


for `more srcdir/srcfiles*.dat`
do
dsjob -run params----- (invoc id/job name ....etc) ==>for file1
check retrun codes..

dsjob -run params----- (invoc id/job name ....etc) ==>>file2
check retrun codes..

so on...##how can i run all the instances at the same time. I have given this as an example but is there any other way I can achive this##
done


Thanks

[priyadarshikunal]

You can always put constaints and manipulate accouring to your needs. For example below code runs 15 instances at a time

Code: Select all

ls -1 | while read output; 
do 
#echo $output 
count= 2
while [ $count -ge 1 ]
do 
vNumInstance=ps -ef |grep JobName|grep -viE '^grep|monitor'|wc -l
if [ $vNumInstance -ge 15 ]
   sleep 300 # 5 mins
else 
count=0
fi
done
dsjob -run  <.... JobName.$output .....>
done 

Not on the unix box. So, you might need to debug it a bit.

[mail2hfz]

If you don't check for return codes after each run then it runs concurrently else it runs sequentially. I might be missing something here!! Can it run parallely by checking the return codes?

Code: Select all

cat ui.txt | while read op;
do
bin/dsjob -run -jobstatus -mode NORMAL <<proj>> test.$op 1>/dev/null 2>/dev/null
STATUS=$?
if [ $STATUS -le 2 ] && [ $STATUS -ge 0 ]
        then
                echo "JOB Successfully Completed with Status Code $STATUS"
        else
                echo "DataStage Job Failed.... Status Code is $STATUS"
                exit -1
fi
done

[swarnkar]

If you don't check for return codes after each run then it runs concurrently else it runs sequentially.

If you want to check the status of the job then you need to break entire logic into two scripts.

First one would acts as controller, which just invoke the second routine,
and in second routine you can call the datastage jobs.

So there will be multiple instances of second script also.

Thanks,
Nitin Swarnkar