How do I get dsjob -logdetail to list the log from the last

[sbass1]

Hi,

I did search on "dsjob -logdetail", plus a lot of hacking with dsjob, before posting...

How do I get dsjob -logdetail to list the log from the last job run? IOW, how can I derive the <first event id>? The only way I could see is issue dsjob -logsum <project> <jobname>, capture the first line, and cut the event id. And I think this would only work if I'd first cleared the log before the job run.

Even better would be if I could limit the output to Warnings. But I think grep would take care of that.

Thanks,
Scott

[nagarjuna]

Are you trying to capture the lateset log ? May be you can try using dsjob -lognewest.

[chulett]

That would be where to start.

[kduke]

We did it the way you said. Use both logsum and logdetail.

[sbass1]

Are you trying to capture the lateset log ? May be you can try using dsjob -lognewest.

I need the log details of the last job run. I'm getting a lot of data truncation warnings writing to the database, and need to pass this information to the provider of the source data so they can clean it.

[ray.wurlod]

So?

The -lognewest allows you to find the most recent "started" event, and you can use that information to filter what you get back from -logdetail.

It's probably easier to do this with a DataStage BASIC routine than with dsjob commands.

[sbass1]

So?

The -lognewest allows you to find the most recent "started" event, and you can use that information to filter what you get back from -logdetail.

It's probably easier to do this with a DataStage BASIC routine than with dsjob commands.

Hi Ray,

-lognewest yields:

Code: Select all

dsjob -lognewest MY_PROJECT MyJob
Newest id = 44

Status code = 0

OK, so now I have the "Newest id" (after writing script code to cut it into a shell variable)

So I go into Director, open the last entry for that job, and see that it's #44.

The first entry for the last run of that job (the first "black" line in the log) is #17.

I need to list -logdetail for entry #17 - #44, filtering for warnings only (the actual job has thousands of lines in the log).

I can leave the last entry off the -logdetail command, but how do I get the first entry (thanks Kim).

Now, wouldn't

Code: Select all

dsjob -logdetail -lastrun MY_PROJECT MyJob

be a handy option?

If anyone has written a Unix wrapper script that does this, please post it. Otherwise, if i write one, I'll post it.

I just need to do this as a one off email to the source file providers, so I'm not getting how a BASIC routine would be easier.

Thanks,
Scott

[ray.wurlod]

dsjob -lognewest can be qualified by an event type.

Type in just dsjob -lognewest to get syntax summary.

[mfavero]

Code: Select all

#!/bin/ksh
#######
################################################################################
#######
####### FILE: dsjob_logwarns.ksh
#######
####### DESCRIPTION: Retrieve Warning and Fatal error messages from datastage job
#######              log for latest run
#######
#######              Steps:
#######              1) Use dsjob -logsum to find the START EVENT ID of the most
#######                 recent run of the job
#######              2) Use dsjob -logsum to find all the WARNING EVENT IDs on the log
#######              3) Use dsjob -logsum to find all the FATAL EVENT IDs on the log
#######              4) Loop through the warnings and fatal events and get the details 
#######                 for each log entry which is in the most recent run using 
#######                 dsjob -logdetail
#######              5) Write all these out to a file  ${PROG}_${JOBID}.out
#######
#######
####### PARAMETERS:   SERVERID="${1}" 
#######               USERID="${2}"   
#######               PASSWORD="${3}" 
#######               PROJECTID="${4}"
#######               JOBID="${5}"    
#######
####### Date       Developer      Description
####### ---------- -------------- ----------------------------------------------
####### 2009-05-06 Mike Favero    Initial Release
#######
#######

PROG="`basename ${0}`"
Now=`date`
ReturnValue=0

#
# Check for correct number of parameters
#
if [ ${#} -ne 5 ]
 then
   ReturnValue=1
   echo "${Now} ${PROG} : Invalid parameter list."
   echo "${Now} ${PROG} : The script needs 5 parameters:  1. SERVERID=${1}"  
   echo "${Now} ${PROG} :                                 2. USERID=${2}"    
   echo "${Now} ${PROG} :                                 3. PASSWORD=${3}"  
   echo "${Now} ${PROG} :                                 4. PROJECTID=${4}" 
   echo "${Now} ${PROG} :                                 5. JOBID=${5}"     
   echo "${Now} ${PROG} Exiting with ReturnValue of: ${ReturnValue}"
   exit ${ReturnValue}                                    
fi                                                        

SERVERID="${1}"
USERID="${2}"
PASSWORD="${3}"
PROJECTID="${4}"
JOBID="${5}"

START_ID=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type STARTED ${PROJECTID} ${JOBID} | nawk 'ORS=(FNR%2)?FS:RS' | grep Starting | tail -1 | awk '{print $1 }'`
WARN_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type WARNING ${PROJECTID} ${JOBID} | grep WARNING | awk '{print $1 }'`
FATAL_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type FATAL ${PROJECTID} ${JOBID} | grep FATAL | awk '{print $1 }'`
echo "Job ${PROJECTID} ${JOBID} warning and fatal messages:" > ${PROG}_${JOBID}.out

ALL_IDS=`echo "${WARN_IDS} ${FATAL_IDS}"`

for TEST_ID in ${ALL_IDS}
 do
	if [[ "${TEST_ID}" -gt "${START_ID}" ]]
	then
		WARN_DTL=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logdetail ${PROJECTID} ${JOBID} ${TEST_ID}`
		echo ${WARN_DTL} >> ${PROG}_${JOBID}.out
	fi
 done

exit ${ReturnValue}

[PhilHibbs]

Excellent! Wonderful! Thanks for this. I had to changed nawk to gawk because our box doesn't have nawk installed.

[FranklinE]

Michael, many thanks also.

Question: I've encountered important details in INFO messages, especially from FTP stages. I was wondering do you, Phil or anyone else have an idea of how to identify them in a generic way? My self-interest is in being able to display for non-DS users the log messages, and leaving out the messages coming in from the external process -- which end up in INFO types -- just makes them wake me up to logon and find them.

[PhilHibbs]

Michael, many thanks also.

Question: I've encountered important details in INFO messages, especially from FTP stages. I was wondering do you, Phil or anyone else have an idea of how to identify them in a generic way? My self-interest is in being able to display for non-DS users the log messages, and leaving out the messages coming in from the external process -- which end up in INFO types -- just makes them wake me up to logon and find them.

You could easily add INFO messages to this script, but grep them for specific pieces of text. This is particulary true of sequences, e.g. "JobSeq.285.JobControl (DSWaitForJob): Job Job_Name.285 has finished, status = 3 (Aborted)" is an Info.

That would look something like this:

Code: Select all

INFO_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type INFO ${PROJECTID} ${JOBID} | grep "JobControl.+status = 3.+Aborted" | awk '{print $1 }'` 

and then...

Code: Select all

ALL_IDS=`echo "${WARN_IDS} ${FATAL_IDS} ${INFO_IDS}"` 

I think... if a Unix script guru can confirm... multiple types of INFO messages would look something like this:

Code: Select all

INFO_IDS=`dsjob -server ${SERVERID} -user ${USERID} -password ${PASSWORD} -logsum -type INFO ${PROJECTID} ${JOBID} | grep "(JobControl.+status = 3.+Aborted)|(rsMyRoutine.+Major error in routine)|(some_other_pattern)" | awk '{print $1 }'` 

My regex experience is mostly in Perl so I'm not sure if these are correct for grep.

[pandeesh]

if you search here, you can find the script what you require.

I have posted the script few months back,