Initiating a DataStage Job from the UNIX command line

moose2079 · Post by **moose2079** » Tue Mar 07, 2006 9:39 pm

Hi there,
I am novice with UNIX command line and DataStage
- BUT I need to do a 'proof of concept' regarding firing(starting - executing) a DataStage Job from the UNIX command line
I am guessing that dsexecute is the way
- but I need simple step by step instructions on how to do this
I have a DataStage job called DoNothing
- yep it does nothing, and it has no parameters.
I what to initiate DoNothing using the UNIX command line and then I will verify the result by using director.
There are many DataStage development projects where I work so I need to 'point' UNIX to the correct DataStage project so the DataStage DoNothing job in my DataStage development project is the one that is run (I assume there must be a pathway in the UNIX line command to do this)

Please advise
M&M

kcbland · Post by **kcbland** » Tue Mar 07, 2006 10:02 pm

Check here for an example script:

viewtopic.php?t=85578

moose2079 · Post by **moose2079** » Tue Mar 07, 2006 10:57 pm

kcbland wrote:Check here for an example script:

viewtopic.php?t=85578

Thanks Kenneth
Script looks great - what is dsexecute used for?

M&M

kcbland · Post by **kcbland** » Wed Mar 08, 2006 7:43 am

DSExecute is an API. Read your DS BASIC manual for more details. The command line interface program dsjob is well documented in its own manual under your Start menu.

kduke · Post by **kduke** » Wed Mar 08, 2006 10:45 am

Ken script is great but I think it is hard to understand. I think this script viewtopic.php?t=93365&highlight=runjob is a little easier to understand.

moose2079 · Post by **moose2079** » Thu Mar 09, 2006 5:26 pm

Hi Ken,
I am running XP
RE: 'The command line interface program dsjob is well documented in its own manual under your Start menu'
I assume you mean the Windows START menu
- can U advise where on the START menu
- tried Help/ search
- no luck

chulett · Post by **chulett** » Thu Mar 09, 2006 5:31 pm

In your 'Ascential DataStage' entry on the Start menu, you'll find a link to the pdf manuals. One of them is the Server Job Developer's Guide and there is a full chapter in there (14?) on the Command Line Interface.

moose2079 · Post by **moose2079** » Sun Mar 12, 2006 6:48 pm

kduke wrote:Ken script is great but I think it is hard to understand. I think this script viewtopic.php?t=93365&highlight=runjob is a little easier to understand.

Hi Kim,
I am editing your script - and yes it is easier to understand than Kens script - thank you

On the bottom of the first page you have ..
cd $DSHOME
. ./dsenv
cd ../Projects/$PROJECT/Staging

What does
. ./dsenv do?

I assume double dots('..') means stay in the same directory path and go to a 'lower directory' within the path?

why does the second line have no cd (change directory)?

Why does the third line use 'cd' to prefix the '..'

M&M

ray.wurlod · Post by **ray.wurlod** » Sun Mar 12, 2006 7:45 pm

If you're on Windows XP, then you don't need dsenv. It's only for UNIX.

If in a cmd shell you set your PATH to include the DataStage Engine's bin folder, you can execute dsjob directly. Otherwise you will need to specify its entire pathname, for example C:\Ascential\DataStage\Engine\bin\dsjob

Entered by itself with no command line options it will output a list of the top-level syntax options.

Entered with a main option but still incomplete (for example dsjob -run) it will output a list of the sub-options for that main option.

That, in combination with the manual should get you there. For your POC use

Code: Select all

dsjob -run projectname DoNothing

if you're on the same machine as DataStage.

kduke · Post by **kduke** » Sun Mar 12, 2006 7:48 pm

If you run another script by using dot and a space then the path to the other shell script then it runs it as if it is a part of this script. This is called sourcing a script. If you just run the script then it runs as a child process. A child process can never change the value of a shell variable in the parent. The dsenv file sets up the libraries and other shell variables needed by DataStage.

The cd command after is not neccessary.

ray.wurlod · Post by **ray.wurlod** » Sun Mar 12, 2006 7:54 pm

All DataStage manuals are installed with your DataStage client software in a sub-folder called Docs. By default C:\Ascential\DataStage\Docs.

You can also get there (which is what Craig was suggesting) from the Start menu on your client machine.
Start > All Programs > Ascential DataStage > Online Manuals > DataStage Documentation

moose2079 · Post by **moose2079** » Sun Mar 12, 2006 9:05 pm

[quote="ray.wurlod"]If you're on Windows XP, then you don't need Thanks Kim
- I am on Windows XP - BUT I am running this on UNIX
- as a UNIX korn shell script(.ksh)
- I have your script close to working (POC must be thru a UNIX script)

- re my question about
cd $DSHOME
. ./dsenv
cd ../Projects/$PROJECT/Staging
I assume form your comments I leave
cd $DSHOME
. ./dsenv
as is.

Re:
cd ../Projects/$PROJECT/Staging
the script is failing to find the datastage job
I assume the pathway I gave it is the problem
Thye script section follows:
DSHOME="/progs/dstage/Ascential/DataStage/DSEngine/"
# PROJECT="PolSubD3"
PROJECT=$1
# DSJOB="DoNothing"
DSJOB=$2
shift
shift
# -param ParamName=Value
SET_PARAMS="$*"
cd $DSHOME
. ./dsenv
cd ../Projects/$PROJECT/Staging
#End Script section

I get the following lines on UNIX
../Projects/$PROJECT/Staging Not Found
DoNothing.ksh dsjob not found
Before run Job_Status =
dsjob -run - mode RESET -wait PolSubD3 DoNothing
DoNothing.ksh dsjob not found

I will chase this with my client re pathway
- but if U see something that can help please advise

M&M

ray.wurlod · Post by **ray.wurlod** » Mon Mar 13, 2006 5:54 am

I take it you have multiple -param ParamName=Value options on your script command line also. That's the only way the correct syntax would be picked up through $*.

But, yes, you must solve the cd problem first.

chulett · Post by **chulett** » Mon Mar 13, 2006 7:08 am

moose2079 wrote:I get the following lines on UNIX
../Projects/$PROJECT/Staging Not Found
DoNothing.ksh dsjob not found
Before run Job_Status =
dsjob -run - mode RESET -wait PolSubD3 DoNothing
DoNothing.ksh dsjob not found

You need to remove the line that cd's to staging - it is specific to Kim's scipt and not needed in yours as that directory obviously doesn't exist on your server.

You also either need to make sure that the directory that the 'dsjob' command lives in is in your current user's path or you need to fully path the call to it. Simplest way is the latter - change the line of the 'dsjob' from:

Code: Select all

dsjob  xxx yyy

to:

Code: Select all

$DSHOME/bin/dsjob xxx yyy

Those two things should get you alot closer.

moose2079 · Post by **moose2079** » Tue Mar 14, 2006 5:32 pm

chulett wrote:
moose2079 wrote:I get the following lines on UNIX
../Projects/$PROJECT/Staging Not Found
DoNothing.ksh dsjob not found
Before run Job_Status =
dsjob -run - mode RESET -wait PolSubD3 DoNothing
DoNothing.ksh dsjob not found
You need to remove the line that cd's to staging - it is specific to Kim's scipt and not needed in yours as that directory obviously doesn't exist on your server.

You also either need to make sure that the directory that the 'dsjob' command lives in is in your current user's path or you need to fully path the call to it. Simplest way is the latter - change the line of the 'dsjob' from:
Code: Select all
dsjob  xxx yyy
to:
Code: Select all
$DSHOME/bin/dsjob xxx yyy
Those two things should get you alot closer.

Job ran successfully and here is the new and improve script
#!/bin/ksh
################################################################################
# File Name : RunJob.ksh
# Description : runs DataStage job
#
# Author : Michael MIlls
# Created : 10/03/2006
# Modified :
# Invocation : This process is called by autosys
# Prerequisites: None
#
# Steps : 1) Run DoSomething
# Change History
# Who When What
# ############### ########## #################################################
# Michael Mills 10/03/2006 Original Version
#
################################################################################

THIS_SCRIPT="DoSomething.ksh"
EMAIL_USERS="Michael.Mills@Workcover.nsw.gov.au"

CURRENT_DATE=`date +%Y%m%d`
EXTRACT_DATE=`date +'%Y%m%dT%H%M%S'`
DSHOME="/progs/dstage/Ascential/DataStage/DSEngine"
DSJOBPATH="/progs/dstage/Ascential/DataStage/DSEngine/bin"
# PROJECT="PolSubD3"
PROJECT=$1
# DSJOB="DoSomethingO2CJobCode"
DSJOB=$2
shift
shift
PARAMS=$1
# -param ParamName=Value
SET_PARAMS="$*"
cd $DSHOME
. ./dsenv
# cd ../Projects/$PROJECT/Staging
cd $DSJOBPATH

################################################################################
# Check job status here
################################################################################
JOB_STATUS=`dsjob -jobinfo $PROJECT $DSJOB | head -1 | cut -d"(" -f2 | cut -d")" -f1`
echo "Before run JOB_STATUS=$JOB_STATUS"
case $JOB_STATUS in
################################################################################
# 0 "Running"
################################################################################
0)
echo "Job $DSJOB already running." | mailx -s "Error: Job $DSJOB already running ${THIS_SCRIPT} failed." $EMAIL_USERS
exit 1
;;
################################################################################
# Runnable Job Status (do something)
# 1 "Finished"
# 2 "Finished (see log)"
# 9 "Has been reset"
# 11 "Validated OK"
# 12 "Validated (see log)"
# 21 "Has been reset"
# 99 "Compiled"
################################################################################
1|2|7|9|11|12|21|99)
:
;;
################################################################################
# NOT Runnable Job Status (reset job)
# 0 "Running"
# 3 "Aborted"
# 8 "Failed validation"
# 13 "Failed validation"
# 96 "Aborted"
# 97 "Stopped"
# 98 "Not Compiled"
################################################################################
*)
echo "dsjob -run -mode RESET -wait $PROJECT $DSJOB"
dsjob -run -mode RESET -wait $PROJECT $DSJOB
RETURN_VALUE=$?
if [ $RETURN_VALUE -ne 0 ]
then
echo "Unable to reset job $DSJOB alreaady running." | mailx -s "Error: Unable to reset job $DSJOB alreaady running ${THIS_SCRIPT} failed." $EMAIL_USERS
exit $RETURN_VALUE
fi
esac

################################################################################
# Run job here
################################################################################
echo "dsjob -run -jobstatus -wait ${SET_PARAMS} $PROJECT $DSJOB"
dsjob -run -jobstatus -wait ${SET_PARAMS} $PROJECT $DSJOB
RETURN_VALUE=$?
################################################################################
# NOT Runnable Job Status (reset job)
# 0 "Running"
# 3 "Aborted"
# 8 "Failed validation"
# 13 "Failed validation"
# 96 "Aborted"
# 97 "Stopped"
# 98 "Not Compiled"
################################################################################
case $RETURN_VALUE in
1|2)
echo "job completed successfully "
finished="job completed successfully "
;;
0)
echo "Error: $DSJOB job failed. error code was -ALREADY RUNNING- Code $RETURN_VALUE"
finished="Error: $DSJOB job failed. error code was -ALREADY RUNNING- Code $RETURN_VALUE"
;;
3|96)
echo "Error: $DSJOB job failed. error code was -ABORT- Code $RETURN_VALUE"
finished="Error: $DSJOB job failed. error code was -ABORT- Code $RETURN_VALUE"
;;
8|13)
echo "Error: $DSJOB job failed. error code was -Failed Validation- Code $RETURN_VALUE"
finished="Error: $DSJOB job failed. error code was -Failed Validation- Code $RETURN_VALUE"
;;
97)
echo "Error: $DSJOB job failed. error code was -Stopped- Code $RETURN_VALUE"
finished="Error: $DSJOB job failed. error code was -Stopped- Code $RETURN_VALUE"
;;
98)
echo "Error: $DSJOB job failed. error code was -Not Compiled- Code $RETURN_VALUE"
finished="Error: $DSJOB job failed. error code was -Not Compiled- Code $RETURN_VALUE"
;;
*)
echo "Error: $DSJOB job failed. error code was $RETURN_VALUE"
finished="Error: $DSJOB job failed. error code was $RETURN_VALUE"
;;
esac

EMAIL_MSG=`dsjob -report $PROJECT $DSJOB`
mailx -s "${THIS_SCRIPT} $finished." $EMAIL_USERS << MEOF
Shell script : ${THIS_SCRIPT}
$EMAIL_MSG
MEOF