Page 1 of 1
How do I get dsjob -logdetail to list the log from the last
Posted: Tue May 05, 2009 5:50 am
by sbass1
Hi,
I did search on "dsjob -logdetail", plus a lot of hacking with dsjob, before posting...
How do I get dsjob -logdetail to list the log from the last job run? IOW, how can I derive the <first event id>? The only way I could see is issue dsjob -logsum <project> <jobname>, capture the first line, and cut the event id. And I think this would only work if I'd first cleared the log before the job run.
Even better would be if I could limit the output to Warnings. But I think grep would take care of that.
Thanks,
Scott
Posted: Tue May 05, 2009 7:19 am
by nagarjuna
Are you trying to capture the lateset log ? May be you can try using dsjob -lognewest.
Posted: Tue May 05, 2009 8:04 am
by chulett
That would be where to start.
Posted: Tue May 05, 2009 11:56 am
by kduke
We did it the way you said. Use both logsum and logdetail.
Posted: Tue May 05, 2009 4:11 pm
by sbass1
nagarjuna wrote:Are you trying to capture the lateset log ? May be you can try using dsjob -lognewest.
I need the log
details of the last job run. I'm getting a lot of data truncation warnings writing to the database, and need to pass this information to the provider of the source data so they can clean it.
Posted: Tue May 05, 2009 4:19 pm
by ray.wurlod
So?
The -lognewest allows you to find the most recent "started" event, and you can use that information to filter what you get back from -logdetail.
It's probably easier to do this with a DataStage BASIC routine than with dsjob commands.
Posted: Tue May 05, 2009 5:31 pm
by sbass1
ray.wurlod wrote:So?
The -lognewest allows you to find the most recent "started" event, and you can use that information to filter what you get back from -logdetail.
It's probably easier to do this with a DataStage BASIC routine than with dsjob commands.
Hi Ray,
-lognewest yields:
Code: Select all
dsjob -lognewest MY_PROJECT MyJob
Newest id = 44
Status code = 0
OK, so now I have the "Newest id" (after writing script code to cut it into a shell variable)
So I go into Director, open the
last entry for that job, and see that it's #44.
The
first entry for the last run of that job (the first "black" line in the log) is #17.
I need to list -logdetail for entry #17 - #44, filtering for warnings only (the actual job has thousands of lines in the log).
I can leave the last entry off the -logdetail command, but how do I get the first entry (thanks Kim).
Now, wouldn't
Code: Select all
dsjob -logdetail -lastrun MY_PROJECT MyJob
be a handy option?
If anyone has written a Unix wrapper script that does this, please post it. Otherwise, if i write one, I'll post it.
I just need to do this as a one off email to the source file providers, so I'm not getting how a BASIC routine would be easier.
Thanks,
Scott
Posted: Tue May 05, 2009 5:37 pm
by ray.wurlod
dsjob -lognewest can be qualified by an event type.
Type in just dsjob -lognewest to get syntax summary.
Posted: Wed May 06, 2009 3:17 pm
by mfavero
Code: Select all
#!/bin/ksh
#######
################################################################################
#######
####### FILE: dsjob_logwarns.ksh
#######
####### DESCRIPTION: Retrieve Warning and Fatal error messages from datastage job
####### log for latest run
#######
####### Steps:
####### 1) Use dsjob -logsum to find the START EVENT ID of the most
####### recent run of the job
####### 2) Use dsjob -logsum to find all the WARNING EVENT IDs on the log
####### 3) Use dsjob -logsum to find all the FATAL EVENT IDs on the log
####### 4) Loop through the warnings and fatal events and get the details
####### for each log entry which is in the most recent run using
####### dsjob -logdetail
####### 5) Write all these out to a file ${PROG}_${JOBID}.out
#######
#######
####### PARAMETERS: SERVERID="${1}"
####### USERID="${2}"
####### PASSWORD="${3}"
####### PROJECTID="${4}"
####### JOBID="${5}"
#######
####### Date Developer Description
####### ---------- -------------- ----------------------------------------------
####### 2009-05-06 Mike Favero Initial Release
#######
#######
PROG="`basename ${0}`"
Now=`date`
ReturnValue=0
#
# Check for correct number of parameters
#
if [ ${#} -ne 5 ]
then
ReturnValue=1
echo "${Now} ${PROG} : Invalid parameter list."
echo "${Now} ${PROG} : The script needs 5 parameters: 1. SERVERID=${1}"
echo "${Now} ${PROG} : 2. USERID=${2}"
echo "${Now} ${PROG} : 3. PASSWORD=${3}"
echo "${Now} ${PROG} : 4. PROJECTID=${4}"
echo "${Now} ${PROG} : 5. JOBID=${5}"
echo "${Now} ${PROG} Exiting with ReturnValue of: ${ReturnValue}"
exit ${ReturnValue}
fi
SERVERID="${1}"
USERID="${2}"
PASSWORD="${3}"
PROJECTID="${4}"
JOBID="${5}"
START_ID=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type STARTED ${PROJECTID} ${JOBID} | nawk 'ORS=(FNR%2)?FS:RS' | grep Starting | tail -1 | awk '{print $1 }'`
WARN_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type WARNING ${PROJECTID} ${JOBID} | grep WARNING | awk '{print $1 }'`
FATAL_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type FATAL ${PROJECTID} ${JOBID} | grep FATAL | awk '{print $1 }'`
echo "Job ${PROJECTID} ${JOBID} warning and fatal messages:" > ${PROG}_${JOBID}.out
ALL_IDS=`echo "${WARN_IDS} ${FATAL_IDS}"`
for TEST_ID in ${ALL_IDS}
do
if [[ "${TEST_ID}" -gt "${START_ID}" ]]
then
WARN_DTL=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logdetail ${PROJECTID} ${JOBID} ${TEST_ID}`
echo ${WARN_DTL} >> ${PROG}_${JOBID}.out
fi
done
exit ${ReturnValue}
Posted: Fri Dec 16, 2011 10:30 am
by PhilHibbs
Excellent! Wonderful! Thanks for this. I had to changed nawk to gawk because our box doesn't have nawk installed.
Posted: Fri Dec 16, 2011 11:12 am
by FranklinE
Michael, many thanks also.
Question: I've encountered important details in INFO messages, especially from FTP stages. I was wondering do you, Phil or anyone else have an idea of how to identify them in a generic way? My self-interest is in being able to display for non-DS users the log messages, and leaving out the messages coming in from the external process -- which end up in INFO types -- just makes them wake me up to logon and find them.
Posted: Fri Dec 16, 2011 11:18 am
by PhilHibbs
FranklinE wrote:Michael, many thanks also.
Question: I've encountered important details in INFO messages, especially from FTP stages. I was wondering do you, Phil or anyone else have an idea of how to identify them in a generic way? My self-interest is in being able to display for non-DS users the log messages, and leaving out the messages coming in from the external process -- which end up in INFO types -- just makes them wake me up to logon and find them.
You could easily add INFO messages to this script, but grep them for specific pieces of text. This is particulary true of sequences, e.g. "JobSeq.285.JobControl (DSWaitForJob): Job Job_Name.285 has finished, status = 3 (Aborted)" is an Info.
That would look something like this:
Code: Select all
INFO_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type INFO ${PROJECTID} ${JOBID} | grep "JobControl.+status = 3.+Aborted" | awk '{print $1 }'`
and then...
Code: Select all
ALL_IDS=`echo "${WARN_IDS} ${FATAL_IDS} ${INFO_IDS}"`
I think... if a Unix script guru can confirm... multiple types of INFO messages would look something like this:
Code: Select all
INFO_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type INFO ${PROJECTID} ${JOBID} | grep "(JobControl.+status = 3.+Aborted)|(rsMyRoutine.+Major error in routine)|(some_other_pattern)" | awk '{print $1 }'`
My regex experience is mostly in Perl so I'm not sure if these are correct for grep.
Posted: Fri Dec 16, 2011 12:02 pm
by pandeesh
if you search here, you can find the script what you require.
I have posted the script few months back,