Total Records processed

[dspxlearn]

We have a requirement in which we have to capture the total number of records pulled from the source, total number of records loaded and number of records rejected. We can do this using the "dsjob -lstages" from the scripts by using the linknames and stage names and putting it in a loop to get the count at all the links.

But,our job designs are not unique and the number of stages used will not be same across the jobs. So. we have to come up with a generic method to use it for all the jobs and get the no. of records pulled from the source,loaded to the target and rejected.
Can anyone please give me a hint how this can be done?

[kumar_s]

So modify you method.
Have consistent Stage / Link names atleast.
So that you can loop through the avaialble links and search for "Input", "Output", "Reject" (for example) and fetch the required data.

[dspxlearn]

Thanks Kumar-
Right. As you said, i have to use names something like 'SRC_RECS' for input links, 'TGT_RECS' for output links and 'REJ_RECS' for reject link.

In this way, after using these naming conventing across jobs, i need to just use the link names to capture the record count at these 3 different links.
As of now, i will implement this. But, let me know if there are any other methods.

[kumar_s]

If you are not even sure about number of stages available in each job, this is the only possible solution that I could think of as of now.

[AmeyJoshi14]

For the same issue we have opted for using the shell scripts.
I know as you said you do not want to use shell script but still i would like to give the script which might help you out.

Code: Select all

#!/bin/ksh
# This script will have to parameters porject name and jobname
project=$1
jobname=$2
CURRDIR=pathname
. $DSHOME/dsenv
#All the stage names used in the job is stored into stage.txt file
$DSHOME/bin/dsjob -lstages $project $jobname 2>/dev/null > $CURRDIR/stage.txt
cat $CURRDIR/stage.txt |while read line
do
      stagename=`echo $line`
      rowcount=`dsjob -stageinfo $project $jobname $stagename 2>/dev/null | grep "In Row Number" |cut -d':' -f2|awk '{printf $1}'`
      #The above command will give you the row count for all the stages in the particular job
      echo "The rowcount for stage $stagename=$rowcount  
done       

Since this script takes two parameters you can run this script for any jobs in particular project.
Hope this will help you!

[dspxlearn]

Thanks AmeyJoshi14-

I never said i didn't like to use shell . Infact, i am writing the similar script as you posted. I fact, shell scripts are more convenient for these kind of requirements rather than using DSroutines.

Thanks for your time.