Triggering Multiple instances job multiple times parallely
Moderators: chulett, rschirm, roy
Triggering Multiple instances job multiple times parallely
Hello All,
Can someone guide me, "how to approach" on the following scenario.
I have a sequence which in turn invokes routine activity, 3 datastage jobs and 3 command activties (unix korn shells). The input for the sequence is flat file and expected output after all the steps is flat file. I might have a single file or multiple files as input when the batch is started. Now, the question is, if there are mutliple files then how to trigger the sequence multiple times parallely. I am able to achieve this sequentially by running the jobs in loop but not in parallel.
Thanks..
Can someone guide me, "how to approach" on the following scenario.
I have a sequence which in turn invokes routine activity, 3 datastage jobs and 3 command activties (unix korn shells). The input for the sequence is flat file and expected output after all the steps is flat file. I might have a single file or multiple files as input when the batch is started. Now, the question is, if there are mutliple files then how to trigger the sequence multiple times parallely. I am able to achieve this sequentially by running the jobs in loop but not in parallel.
Thanks..
Hi,
You have to make your top sequencer and all the jobs, as multi-instance.
Then through shell script or Datastage routine you have to run this top level sequncer multiple times, with proper invocation ids.
You can search this forum for dsjob -run commnand, if you are invoking your sequencer through shell script.
Thanks,
Nitin Swarnkar
You have to make your top sequencer and all the jobs, as multi-instance.
Then through shell script or Datastage routine you have to run this top level sequncer multiple times, with proper invocation ids.
You can search this forum for dsjob -run commnand, if you are invoking your sequencer through shell script.
Thanks,
Nitin Swarnkar
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
This will call the job for each file in the directory with invocation ID same as file name.
Code: Select all
ls -1 | while read output;
do
#echo $output
dsjob -run <.... JobName.$output .....>
done
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.
Genius may have its limitations, but stupidity is not thus handicapped.
Well, I am not sure if my post is not clear. I am looking for a triggering mechanism that is capable of running my dsjob/multi instance sequence paralley depending on the number of files.
Eg: 3 input files are available, run the sequence in 3 instances at the same time. I will have timestamp/jobname as my invocation id.
for `more srcdir/srcfiles*.dat`
do
dsjob -run params----- (invoc id/job name ....etc) ==>for file1
check retrun codes..
dsjob -run params----- (invoc id/job name ....etc) ==>>file2
check retrun codes..
so on...##how can i run all the instances at the same time. I have given this as an example but is there any other way I can achive this##
done
Thanks
Eg: 3 input files are available, run the sequence in 3 instances at the same time. I will have timestamp/jobname as my invocation id.
for `more srcdir/srcfiles*.dat`
do
dsjob -run params----- (invoc id/job name ....etc) ==>for file1
check retrun codes..
dsjob -run params----- (invoc id/job name ....etc) ==>>file2
check retrun codes..
so on...##how can i run all the instances at the same time. I have given this as an example but is there any other way I can achive this##
done
Thanks
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
You can always put constaints and manipulate accouring to your needs. For example below code runs 15 instances at a time
Not on the unix box. So, you might need to debug it a bit.
Code: Select all
ls -1 | while read output;
do
#echo $output
count= 2
while [ $count -ge 1 ]
do
vNumInstance=ps -ef |grep JobName|grep -viE '^grep|monitor'|wc -l
if [ $vNumInstance -ge 15 ]
sleep 300 # 5 mins
else
count=0
fi
done
dsjob -run <.... JobName.$output .....>
done
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.
Genius may have its limitations, but stupidity is not thus handicapped.
If you don't check for return codes after each run then it runs concurrently else it runs sequentially. I might be missing something here!! Can it run parallely by checking the return codes?
Code: Select all
cat ui.txt | while read op;
do
bin/dsjob -run -jobstatus -mode NORMAL <<proj>> test.$op 1>/dev/null 2>/dev/null
STATUS=$?
if [ $STATUS -le 2 ] && [ $STATUS -ge 0 ]
then
echo "JOB Successfully Completed with Status Code $STATUS"
else
echo "DataStage Job Failed.... Status Code is $STATUS"
exit -1
fi
done
If you want to check the status of the job then you need to break entire logic into two scripts.mail2hfz wrote:If you don't check for return codes after each run then it runs concurrently else it runs sequentially.
First one would acts as controller, which just invoke the second routine,
and in second routine you can call the datastage jobs.
So there will be multiple instances of second script also.
Thanks,
Nitin Swarnkar