Need shell script to start job based on remote trigger file

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
sryarraguntla
Participant
Posts: 13
Joined: Fri Mar 14, 2008 4:30 pm

Need shell script to start job based on remote trigger file

Post by sryarraguntla »

Hi Everyone,

I have a requirement to look for a trigger file on other server and upon find will kick off a DS job. For this i need to write a shell script that looks for a trigger file, if finds then kick the DS job and if not the script has to run again untill it finds the file.. something like loop or sleep mechanism..

Any one who have implemeted this similar situation can help me in writing the script, any ideas on how to acheive this requirement will also be appreciated.

Thanks in Advance

**Note: Subject line made more descriptive - Content Editor **
JRodriguez
Premium Member
Premium Member
Posts: 425
Joined: Sat Nov 19, 2005 9:26 am
Location: New York City
Contact:

Re: Shell Script to kick DS job

Post by JRodriguez »

sryarraguntla,

Take a look at the Wait For File activity in the sequence palette. This activity address what you are looking for and more .... unless you wanted it done specifically by a shell script
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

If the file isn't easily accessible via the WaitForFile mechanism, your other solution is to write a shell script that wakes up periodically, and uses FTP to check for the trigger and move it to a local directory (when present).

Then you can still use the WaitForFile stage in your job to check for the local copy when it arrives.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
AmeyJoshi14
Participant
Posts: 334
Joined: Fri Dec 01, 2006 5:17 am
Location: Texas

Post by AmeyJoshi14 »

Hi,
You can use the following shell script

Code: Select all

#/bin/sh
PATHDIR=/opt/home
val=2
while [ $val -eq 2 ]
do
  find $PATHDIR -name "triggerfilename.txt" > $PATHDIR/gotit.txt  #finding the trigger file
  val1=`wc -l $PATHDIR/gotit.txt|cut -f1 -d'/'`
  if [ $val1 -ne 0 ]
   then
    #run the dsjob command script
    val=1
  else
    echo 'Still Running'
   fi
sleep 35  #Increase the sleep,if required
done
If you want to run the jobs from different servers you can use ssh command after 'if then'...ssh command will be :

Code: Select all

ssh -t -l username IP PATHDIR/dsjobrun.sh
Or as suggested by asorrell use ftp(scp) and transfer the file and then run the dsjob script. :wink:
Hope this will help you out ! :P
http://findingjobsindatastage.blogspot.com/
Theory is when you know all and nothing works. Practice is when all works and nobody knows why. In this case we have put together theory and practice: nothing works. and nobody knows why! (Albert Einstein)
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

I wrote something similar a while back. Its here.
You can edit it to fire a job once the file is found. You can also set a time limit.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
sryarraguntla
Participant
Posts: 13
Joined: Fri Mar 14, 2008 4:30 pm

Post by sryarraguntla »

Really thanks for all your replys. I appreciate your help and quick response.

Ameyjoshi14-- Can i connect to the server AS400 through ftp and execute the script which you gave to check for the trigger file if found run the dsjob if not sleep..

If means the code should looks like

#!/bin/sh
HOST='xxxxx'
USER='xxxxx'
PASSWD='xxxx'
ftp $HOST <<END_SCRIPT
user $USER
$PASSWD
------ the next command would be your script ----

Please let me know if iam correct

Once again thanks
AmeyJoshi14
Participant
Posts: 334
Joined: Fri Dec 01, 2006 5:17 am
Location: Texas

Post by AmeyJoshi14 »

Hi,
I am not sure if the above commands will be fine... :roll:
What we did is develop the scrpit, which can telnet automatically and run commands on the other machine and end the telnet session but for this you need to install EXPECT and TCL packages on your unix system. :shock:
Or as suggested by asorrell you can transfer the file from one server to another server (where you require it) and then run the dsjob command.
Then the script will look as :

Code: Select all

#/bin/sh
PATHDIR=/opt/home
val=2
while [ $val -eq 2 ]
do
  find $PATHDIR -name "dum12.txt" > $PATHDIR/gotit.txt  #finding the trigger file
  val1=`wc -l $PATHDIR/gotit.txt|cut -f1 -d'/'`
  if [ $val1 -ne 0 ]
   then
    val=1
    #you can create another script and use this as wrapper.
    ftp -v -n "YOUR.IP.ADD.RESS" << cmd
    user "user" "passwd"
    cd /distant/directory
    lcd /local/directory
    put your files 
#All you have to do after is to create the cron if you are "putting" a file,replace the put with ls and see if your file is in there.
    quit
    cmd
  else
    echo 'Still Running'
  fi
sleep 35  #Increase the sleep,if required
done
I was not able to test this code since right now i do not have access to unix box. :oops:
Hope this will solve your problem :wink:
http://findingjobsindatastage.blogspot.com/
Theory is when you know all and nothing works. Practice is when all works and nobody knows why. In this case we have put together theory and practice: nothing works. and nobody knows why! (Albert Einstein)
AmeyJoshi14
Participant
Posts: 334
Joined: Fri Dec 01, 2006 5:17 am
Location: Texas

Post by AmeyJoshi14 »

DSguru2B wrote:I wrote something similar a while back. Its here.
You can edit it to fire a job once the file is found. You can also set a time ...
Hey! That's really nice scipt... :D if i would have found it first....by little modification(as per the requirement) lot of work in our project would have been saved :wink:
http://findingjobsindatastage.blogspot.com/
Theory is when you know all and nothing works. Practice is when all works and nobody knows why. In this case we have put together theory and practice: nothing works. and nobody knows why! (Albert Einstein)
Post Reply