Page 1 of 1

How to capture the total no. of rows passed through a link

Posted: Mon Feb 07, 2005 7:05 am
by sun786
I need to capture the total number of rows that pass through any link in a job . For eg. 1. The number of rows from Input to transformer 2. The number of rows from Transformer to ouput1. 3. The number of rows from Transformer to ouput2. and so on.
I have tried with DSGetLinkInfo.
Thanks
sun

Posted: Mon Feb 07, 2005 7:19 am
by kduke
Download EtlStats.zip from my tips page.

Posted: Mon Feb 07, 2005 7:49 am
by sun786
Hi, I just need to count the the total number of rows passed through any link....this should be done in the transformer stage and the output to another seq file.
let me know if more input is needed.

Posted: Mon Feb 07, 2005 8:09 am
by chulett
I'm curious why this "should be done in the transformer"? Do you really want to collect these stats every time a row comes through your job?

This is typically done after job. If you download Kim's code, you'll see how it can be accomplished. Also, you've said you tried using DSGetLinkInfo... how exactly did you try it? What errors did you get?

Posted: Mon Feb 07, 2005 8:14 am
by Sainath.Srinivasan
In that case, maybe you need a simple @OUTROWNUM sent to an agg stage with a max() clause and then written to your output file !!

Re: How to capture the total no. of rows passed through a li

Posted: Mon Feb 07, 2005 8:26 am
by supernova2005
Can you do it on the Unix level? Just use wc -l filename to count the total number of lines on the input file.


sun786 wrote:I need to capture the total number of rows that pass through any link in a job . For eg. 1. The number of rows from Input to transformer 2. The number of rows from Transformer to ouput1. 3. The number of rows from Transformer to ouput2. and so on.
I have tried with DSGetLinkInfo.
Thanks
sun

Posted: Mon Feb 07, 2005 9:07 am
by kduke
This is how you call DSGetStageInfo. This code is a part of DsWebMon.

Code: Select all

      subroutine DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Program Name              : DwnGetMonitor
* Program Dir               : DWNBP
* Created by                : Kim Greggory Duke
* Copyright (C)             : Duke Consulting
* Programmer                : kgd
* Date Created              : 12-01-2002
* -----------------------------------------------------------------------
* Last Modified             : 07-16-2003
* -----------------------------------------------------------------------
* Language                  : UniVerse BASIC
* -----------------------------------------------------------------------
* Usage                     : DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Notes                     :
* -----------------------------------------------------------------------
* initialize standard variables
* ----------------------------------------------------------------------
      ProgId = 'DwnGetMonitor'
$INCLUDE DSINCLUDE JOBCONTROL.H
$include DwnVersion
* ----------------------------------------------------------------------
      MonitorGrid = ''
* -----------------------------------------------------------------------
      open 'DS_JOBS' to DsJobs else goto TheEnd
* -----------------------------------------------------------------------
      JobStatus = ''
* -----------------------------------------------------------------------
      OldJobName = JobName
      if index(JobName, '.', 1) > 0 then
        MultipleInstace = @true
        InvocationId = field(JobName, '.', 2, 9999)
        JobName = field(JobName, '.', 1)
      end else
        MultipleInstace = @false
        InvocationId = ''
      end
      read JobRec from DsJobs, JobName then
         JobNo = JobRec<5>
         * print 'DSAttachJob(':JobName:',3)'
         RunHandle = DSAttachJob(OldJobName, DSJ.ERRNONE)
         JStat = DSGetJobInfo(RunHandle, DSJ.JOBSTATUS)
         ContinueFlag = @true
         CanRunFlag = 1
         CanStopFlag = 0
         CanResetFlag = 1
* -----------------------------------------------------------------------
         begin case
            case JStat = 0
               JobStatus = 'Running'
               CanRunFlag = 0
               CanResetFlag = 0
               CanStopFlag = 1
            case JStat = 1
               JobStatus = 'Finished'
            case JStat = 2               ; * Finished with warnings
               JobStatus = 'Finished'
            case JStat = 3
               ContinueFlag = @false
               JobStatus = 'Aborted'
               CanRunFlag = 0
            case JStat = 11
               JobStatus = 'Finished'
            case JStat = 12              ; * Validated with warnings
               JobStatus = 'Finished'
            case JStat = 13
               * ContinueFlag = @false
               JobStatus = 'Finished'
            case JStat = 21
               JobStatus = 'Finished'
            case JStat = 96
               ContinueFlag = @false
               JobStatus = 'Aborted'
               CanRunFlag = 0
            case JStat = 97
               * ContinueFlag = @false
               JobStatus = 'Stopped'
               CanRunFlag = 0
            case JStat = 98
               JobStatus = 'Finished'
            case JStat = 99
               JobStatus = 'Finished'
            case @true
               ContinueFlag = @false
               JobStatus = 'Not Compiled'
               CanRunFlag = 0
               CanResetFlag = 0
         end case
* -----------------------------------------------------------------------
         if ContinueFlag then
            StageName = ''
            StageStatus = ''
            LinkName = ''
            LinkType =''
            RowCount = ''
            StartedTime = ''
            ElapsedTime = ''
            RowsPerSec = ''
            PercentCP = ''
* -----------------------------------------------------------------------
            StartedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBSTARTTIMESTAMP)
            iStartedDate = iconv(StartedDateTime[1,10], 'D4-YMD')
            iStartedTime = iconv(StartedDateTime[12,8], 'MT')

            if JobStatus = 'Running' then
               FinishedDateTime = oconv(date(), 'D4-YMD'):' ':oconv(time(), 'MTS')
            end else
               FinishedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBLASTTIMESTAMP)
            end
            iFinishedDate = iconv(FinishedDateTime[1,10], 'D4-YMD')
            iFinishedTime = iconv(FinishedDateTime[12,8], 'MT')
            
            * print iFinishedTime:' - ':iStartedTime
            if FinishedDateTime<>'' and StartedDateTime<>'' and iFinishedTime<>'' and iStartedTime<>'' then
               ElapsedTime = ((iFinishedDate-iStartedDate)*60*60)+iFinishedTime-iStartedTime
            end
            if ElapsedTime<0 then
               ElapsedTime = ''
            end
* -----------------------------------------------------------------------
* Get stage names
* -----------------------------------------------------------------------
            Cnt = 0
            StageNames = DSGetJobInfo(RunHandle, DSJ.STAGELIST)
            NoStages = dcount(StageNames, ',')
            for i=1 to NoStages
               StageName = field(StageNames, ',', i)
               * print i:'. ':StageName
* -----------------------------------------------------------------------
* Get link names per stage
* -----------------------------------------------------------------------
               LinkType = DSGetStageInfo(RunHandle, StageName, DSJ.STAGETYPE)
               LinkNames = DSGetStageInfo(RunHandle, StageName, DSJ.LINKLIST)
               NoLinks = dcount(LinkNames, ',')
               for j=1 to NoLinks
                  LinkName = field(LinkNames, ',', j)
                  * print j:'. ':LinkName
* -----------------------------------------------------------------------
                  RowCount = DSGetLinkInfo(RunHandle, StageName, LinkName, DSJ.LINKROWCOUNT)
                  if ElapsedTime>0 and ElapsedTime<>'' then
                     RowsPerSec = (RowCount / ElapsedTime)
                     RowsPerSec = oconv(RowsPerSec, 'MD0')
                  end
* -----------------------------------------------------------------------
                  Cnt += 1
                  if j=1 then
                     MonitorGrid < 1, Cnt> = StageName
                     MonitorGrid < 2, Cnt> = JobStatus
                  end else
                     MonitorGrid < 1, Cnt> = ' '
                     MonitorGrid < 2, Cnt> = ' '
                  end
                  MonitorGrid < 3, Cnt> = LinkName
                  MonitorGrid < 4, Cnt> = LinkType
                  MonitorGrid < 5, Cnt> = RowCount
                  MonitorGrid < 6, Cnt> = StartedDateTime
                  MonitorGrid < 7, Cnt> = ElapsedTime
                  MonitorGrid < 8, Cnt> = RowsPerSec
                  MonitorGrid < 9, Cnt> = PercentCP
                  MonitorGrid <10, Cnt> = CanRunFlag
                  MonitorGrid <11, Cnt> = CanStopFlag
                  MonitorGrid <12, Cnt> = CanResetFlag
                  MonitorGrid <13, Cnt> = FinishedDateTime
               next j
            next i
         end
         JStat = DSDetachJob(RunHandle)
      end
* -----------------------------------------------------------------------
TheEnd:
         return

Posted: Mon Feb 07, 2005 9:08 am
by kduke
You will have to do some work to convert it to a routine but it will get you close.

Posted: Fri Dec 30, 2005 10:35 am
by Titto
kduke wrote:This is how you call DSGetStageInfo. This code is a part of DsWebMon.
Hi Kim,

could you please guide me to write the MonitorGrid to a Flat file by Stage name, LinkName and other details

Thanks in advance

Code: Select all

      subroutine DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Program Name              : DwnGetMonitor
* Program Dir               : DWNBP
* Created by                : Kim Greggory Duke
* Copyright (C)             : Duke Consulting
* Programmer                : kgd
* Date Created              : 12-01-2002
* -----------------------------------------------------------------------
* Last Modified             : 07-16-2003
* -----------------------------------------------------------------------
* Language                  : UniVerse BASIC
* -----------------------------------------------------------------------
* Usage                     : DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Notes                     :
* -----------------------------------------------------------------------
* initialize standard variables
* ----------------------------------------------------------------------
      ProgId = 'DwnGetMonitor'
$INCLUDE DSINCLUDE JOBCONTROL.H
$include DwnVersion
* ----------------------------------------------------------------------
      MonitorGrid = ''
* -----------------------------------------------------------------------
      open 'DS_JOBS' to DsJobs else goto TheEnd
* -----------------------------------------------------------------------
      JobStatus = ''
* -----------------------------------------------------------------------
      OldJobName = JobName
      if index(JobName, '.', 1) > 0 then
        MultipleInstace = @true
        InvocationId = field(JobName, '.', 2, 9999)
        JobName = field(JobName, '.', 1)
      end else
        MultipleInstace = @false
        InvocationId = ''
      end
      read JobRec from DsJobs, JobName then
         JobNo = JobRec<5>
         * print 'DSAttachJob(':JobName:',3)'
         RunHandle = DSAttachJob(OldJobName, DSJ.ERRNONE)
         JStat = DSGetJobInfo(RunHandle, DSJ.JOBSTATUS)
         ContinueFlag = @true
         CanRunFlag = 1
         CanStopFlag = 0
         CanResetFlag = 1
* -----------------------------------------------------------------------
         begin case
            case JStat = 0
               JobStatus = 'Running'
               CanRunFlag = 0
               CanResetFlag = 0
               CanStopFlag = 1
            case JStat = 1
               JobStatus = 'Finished'
            case JStat = 2               ; * Finished with warnings
               JobStatus = 'Finished'
            case JStat = 3
               ContinueFlag = @false
               JobStatus = 'Aborted'
               CanRunFlag = 0
            case JStat = 11
               JobStatus = 'Finished'
            case JStat = 12              ; * Validated with warnings
               JobStatus = 'Finished'
            case JStat = 13
               * ContinueFlag = @false
               JobStatus = 'Finished'
            case JStat = 21
               JobStatus = 'Finished'
            case JStat = 96
               ContinueFlag = @false
               JobStatus = 'Aborted'
               CanRunFlag = 0
            case JStat = 97
               * ContinueFlag = @false
               JobStatus = 'Stopped'
               CanRunFlag = 0
            case JStat = 98
               JobStatus = 'Finished'
            case JStat = 99
               JobStatus = 'Finished'
            case @true
               ContinueFlag = @false
               JobStatus = 'Not Compiled'
               CanRunFlag = 0
               CanResetFlag = 0
         end case
* -----------------------------------------------------------------------
         if ContinueFlag then
            StageName = ''
            StageStatus = ''
            LinkName = ''
            LinkType =''
            RowCount = ''
            StartedTime = ''
            ElapsedTime = ''
            RowsPerSec = ''
            PercentCP = ''
* -----------------------------------------------------------------------
            StartedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBSTARTTIMESTAMP)
            iStartedDate = iconv(StartedDateTime[1,10], 'D4-YMD')
            iStartedTime = iconv(StartedDateTime[12,8], 'MT')

            if JobStatus = 'Running' then
               FinishedDateTime = oconv(date(), 'D4-YMD'):' ':oconv(time(), 'MTS')
            end else
               FinishedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBLASTTIMESTAMP)
            end
            iFinishedDate = iconv(FinishedDateTime[1,10], 'D4-YMD')
            iFinishedTime = iconv(FinishedDateTime[12,8], 'MT')
            
            * print iFinishedTime:' - ':iStartedTime
            if FinishedDateTime<>'' and StartedDateTime<>'' and iFinishedTime<>'' and iStartedTime<>'' then
               ElapsedTime = ((iFinishedDate-iStartedDate)*60*60)+iFinishedTime-iStartedTime
            end
            if ElapsedTime<0 then
               ElapsedTime = ''
            end
* -----------------------------------------------------------------------
* Get stage names
* -----------------------------------------------------------------------
            Cnt = 0
            StageNames = DSGetJobInfo(RunHandle, DSJ.STAGELIST)
            NoStages = dcount(StageNames, ',')
            for i=1 to NoStages
               StageName = field(StageNames, ',', i)
               * print i:'. ':StageName
* -----------------------------------------------------------------------
* Get link names per stage
* -----------------------------------------------------------------------
               LinkType = DSGetStageInfo(RunHandle, StageName, DSJ.STAGETYPE)
               LinkNames = DSGetStageInfo(RunHandle, StageName, DSJ.LINKLIST)
               NoLinks = dcount(LinkNames, ',')
               for j=1 to NoLinks
                  LinkName = field(LinkNames, ',', j)
                  * print j:'. ':LinkName
* -----------------------------------------------------------------------
                  RowCount = DSGetLinkInfo(RunHandle, StageName, LinkName, DSJ.LINKROWCOUNT)
                  if ElapsedTime>0 and ElapsedTime<>'' then
                     RowsPerSec = (RowCount / ElapsedTime)
                     RowsPerSec = oconv(RowsPerSec, 'MD0')
                  end
* -----------------------------------------------------------------------
                  Cnt += 1
                  if j=1 then
                     MonitorGrid < 1, Cnt> = StageName
                     MonitorGrid < 2, Cnt> = JobStatus
                  end else
                     MonitorGrid < 1, Cnt> = ' '
                     MonitorGrid < 2, Cnt> = ' '
                  end
                  MonitorGrid < 3, Cnt> = LinkName
                  MonitorGrid < 4, Cnt> = LinkType
                  MonitorGrid < 5, Cnt> = RowCount
                  MonitorGrid < 6, Cnt> = StartedDateTime
                  MonitorGrid < 7, Cnt> = ElapsedTime
                  MonitorGrid < 8, Cnt> = RowsPerSec
                  MonitorGrid < 9, Cnt> = PercentCP
                  MonitorGrid <10, Cnt> = CanRunFlag
                  MonitorGrid <11, Cnt> = CanStopFlag
                  MonitorGrid <12, Cnt> = CanResetFlag
                  MonitorGrid <13, Cnt> = FinishedDateTime
               next j
            next i
         end
         JStat = DSDetachJob(RunHandle)
      end
* -----------------------------------------------------------------------
TheEnd:
         return

Posted: Fri Dec 30, 2005 3:06 pm
by kduke
You should edit your post and ask a question.

Posted: Fri Dec 30, 2005 8:09 pm
by chulett
Actually, they did. Never understood why people felt the need to quote entire messages for no good reason and then compound the problem by burying their response inside the quote.
Titto wrote:Hi Kim,

could you please guide me to write the MonitorGrid to a Flat file by Stage name, LinkName and other details

Thanks in advance

Posted: Fri Dec 30, 2005 9:54 pm
by kduke
Thanks Craig. I never saw the question.

You need to probably change this into a job. All this code would go in job control code. You need to openseq, writeseq for each line then closeseq. Ray has posted a lot of code to show you how.