Page 1 of 1
How to capture the total no. of rows passed through a link
Posted: Mon Feb 07, 2005 7:05 am
by sun786
I need to capture the total number of rows that pass through any link in a job . For eg. 1. The number of rows from Input to transformer 2. The number of rows from Transformer to ouput1. 3. The number of rows from Transformer to ouput2. and so on.
I have tried with DSGetLinkInfo.
Thanks
sun
Posted: Mon Feb 07, 2005 7:19 am
by kduke
Download EtlStats.zip from my tips page.
Posted: Mon Feb 07, 2005 7:49 am
by sun786
Hi, I just need to count the the total number of rows passed through any link....this should be done in the transformer stage and the output to another seq file.
let me know if more input is needed.
Posted: Mon Feb 07, 2005 8:09 am
by chulett
I'm curious why this "should be done in the transformer"? Do you really want to collect these stats every time a row comes through your job?
This is typically done after job. If you download Kim's code, you'll see how it can be accomplished. Also, you've said you tried using DSGetLinkInfo... how exactly did you try it? What errors did you get?
Posted: Mon Feb 07, 2005 8:14 am
by Sainath.Srinivasan
In that case, maybe you need a simple @OUTROWNUM sent to an agg stage with a max() clause and then written to your output file !!
Re: How to capture the total no. of rows passed through a li
Posted: Mon Feb 07, 2005 8:26 am
by supernova2005
Can you do it on the Unix level? Just use wc -l filename to count the total number of lines on the input file.
sun786 wrote:I need to capture the total number of rows that pass through any link in a job . For eg. 1. The number of rows from Input to transformer 2. The number of rows from Transformer to ouput1. 3. The number of rows from Transformer to ouput2. and so on.
I have tried with DSGetLinkInfo.
Thanks
sun
Posted: Mon Feb 07, 2005 9:07 am
by kduke
This is how you call DSGetStageInfo. This code is a part of DsWebMon.
Code: Select all
subroutine DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Program Name : DwnGetMonitor
* Program Dir : DWNBP
* Created by : Kim Greggory Duke
* Copyright (C) : Duke Consulting
* Programmer : kgd
* Date Created : 12-01-2002
* -----------------------------------------------------------------------
* Last Modified : 07-16-2003
* -----------------------------------------------------------------------
* Language : UniVerse BASIC
* -----------------------------------------------------------------------
* Usage : DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Notes :
* -----------------------------------------------------------------------
* initialize standard variables
* ----------------------------------------------------------------------
ProgId = 'DwnGetMonitor'
$INCLUDE DSINCLUDE JOBCONTROL.H
$include DwnVersion
* ----------------------------------------------------------------------
MonitorGrid = ''
* -----------------------------------------------------------------------
open 'DS_JOBS' to DsJobs else goto TheEnd
* -----------------------------------------------------------------------
JobStatus = ''
* -----------------------------------------------------------------------
OldJobName = JobName
if index(JobName, '.', 1) > 0 then
MultipleInstace = @true
InvocationId = field(JobName, '.', 2, 9999)
JobName = field(JobName, '.', 1)
end else
MultipleInstace = @false
InvocationId = ''
end
read JobRec from DsJobs, JobName then
JobNo = JobRec<5>
* print 'DSAttachJob(':JobName:',3)'
RunHandle = DSAttachJob(OldJobName, DSJ.ERRNONE)
JStat = DSGetJobInfo(RunHandle, DSJ.JOBSTATUS)
ContinueFlag = @true
CanRunFlag = 1
CanStopFlag = 0
CanResetFlag = 1
* -----------------------------------------------------------------------
begin case
case JStat = 0
JobStatus = 'Running'
CanRunFlag = 0
CanResetFlag = 0
CanStopFlag = 1
case JStat = 1
JobStatus = 'Finished'
case JStat = 2 ; * Finished with warnings
JobStatus = 'Finished'
case JStat = 3
ContinueFlag = @false
JobStatus = 'Aborted'
CanRunFlag = 0
case JStat = 11
JobStatus = 'Finished'
case JStat = 12 ; * Validated with warnings
JobStatus = 'Finished'
case JStat = 13
* ContinueFlag = @false
JobStatus = 'Finished'
case JStat = 21
JobStatus = 'Finished'
case JStat = 96
ContinueFlag = @false
JobStatus = 'Aborted'
CanRunFlag = 0
case JStat = 97
* ContinueFlag = @false
JobStatus = 'Stopped'
CanRunFlag = 0
case JStat = 98
JobStatus = 'Finished'
case JStat = 99
JobStatus = 'Finished'
case @true
ContinueFlag = @false
JobStatus = 'Not Compiled'
CanRunFlag = 0
CanResetFlag = 0
end case
* -----------------------------------------------------------------------
if ContinueFlag then
StageName = ''
StageStatus = ''
LinkName = ''
LinkType =''
RowCount = ''
StartedTime = ''
ElapsedTime = ''
RowsPerSec = ''
PercentCP = ''
* -----------------------------------------------------------------------
StartedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBSTARTTIMESTAMP)
iStartedDate = iconv(StartedDateTime[1,10], 'D4-YMD')
iStartedTime = iconv(StartedDateTime[12,8], 'MT')
if JobStatus = 'Running' then
FinishedDateTime = oconv(date(), 'D4-YMD'):' ':oconv(time(), 'MTS')
end else
FinishedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBLASTTIMESTAMP)
end
iFinishedDate = iconv(FinishedDateTime[1,10], 'D4-YMD')
iFinishedTime = iconv(FinishedDateTime[12,8], 'MT')
* print iFinishedTime:' - ':iStartedTime
if FinishedDateTime<>'' and StartedDateTime<>'' and iFinishedTime<>'' and iStartedTime<>'' then
ElapsedTime = ((iFinishedDate-iStartedDate)*60*60)+iFinishedTime-iStartedTime
end
if ElapsedTime<0 then
ElapsedTime = ''
end
* -----------------------------------------------------------------------
* Get stage names
* -----------------------------------------------------------------------
Cnt = 0
StageNames = DSGetJobInfo(RunHandle, DSJ.STAGELIST)
NoStages = dcount(StageNames, ',')
for i=1 to NoStages
StageName = field(StageNames, ',', i)
* print i:'. ':StageName
* -----------------------------------------------------------------------
* Get link names per stage
* -----------------------------------------------------------------------
LinkType = DSGetStageInfo(RunHandle, StageName, DSJ.STAGETYPE)
LinkNames = DSGetStageInfo(RunHandle, StageName, DSJ.LINKLIST)
NoLinks = dcount(LinkNames, ',')
for j=1 to NoLinks
LinkName = field(LinkNames, ',', j)
* print j:'. ':LinkName
* -----------------------------------------------------------------------
RowCount = DSGetLinkInfo(RunHandle, StageName, LinkName, DSJ.LINKROWCOUNT)
if ElapsedTime>0 and ElapsedTime<>'' then
RowsPerSec = (RowCount / ElapsedTime)
RowsPerSec = oconv(RowsPerSec, 'MD0')
end
* -----------------------------------------------------------------------
Cnt += 1
if j=1 then
MonitorGrid < 1, Cnt> = StageName
MonitorGrid < 2, Cnt> = JobStatus
end else
MonitorGrid < 1, Cnt> = ' '
MonitorGrid < 2, Cnt> = ' '
end
MonitorGrid < 3, Cnt> = LinkName
MonitorGrid < 4, Cnt> = LinkType
MonitorGrid < 5, Cnt> = RowCount
MonitorGrid < 6, Cnt> = StartedDateTime
MonitorGrid < 7, Cnt> = ElapsedTime
MonitorGrid < 8, Cnt> = RowsPerSec
MonitorGrid < 9, Cnt> = PercentCP
MonitorGrid <10, Cnt> = CanRunFlag
MonitorGrid <11, Cnt> = CanStopFlag
MonitorGrid <12, Cnt> = CanResetFlag
MonitorGrid <13, Cnt> = FinishedDateTime
next j
next i
end
JStat = DSDetachJob(RunHandle)
end
* -----------------------------------------------------------------------
TheEnd:
return
Posted: Mon Feb 07, 2005 9:08 am
by kduke
You will have to do some work to convert it to a routine but it will get you close.
Posted: Fri Dec 30, 2005 10:35 am
by Titto
kduke wrote:This is how you call DSGetStageInfo. This code is a part of DsWebMon.
Hi Kim,
could you please guide me to write the MonitorGrid to a Flat file by Stage name, LinkName and other details
Thanks in advance
Code: Select all
subroutine DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Program Name : DwnGetMonitor
* Program Dir : DWNBP
* Created by : Kim Greggory Duke
* Copyright (C) : Duke Consulting
* Programmer : kgd
* Date Created : 12-01-2002
* -----------------------------------------------------------------------
* Last Modified : 07-16-2003
* -----------------------------------------------------------------------
* Language : UniVerse BASIC
* -----------------------------------------------------------------------
* Usage : DwnGetMonitor(JobName, MonitorGrid)
* -----------------------------------------------------------------------
* Notes :
* -----------------------------------------------------------------------
* initialize standard variables
* ----------------------------------------------------------------------
ProgId = 'DwnGetMonitor'
$INCLUDE DSINCLUDE JOBCONTROL.H
$include DwnVersion
* ----------------------------------------------------------------------
MonitorGrid = ''
* -----------------------------------------------------------------------
open 'DS_JOBS' to DsJobs else goto TheEnd
* -----------------------------------------------------------------------
JobStatus = ''
* -----------------------------------------------------------------------
OldJobName = JobName
if index(JobName, '.', 1) > 0 then
MultipleInstace = @true
InvocationId = field(JobName, '.', 2, 9999)
JobName = field(JobName, '.', 1)
end else
MultipleInstace = @false
InvocationId = ''
end
read JobRec from DsJobs, JobName then
JobNo = JobRec<5>
* print 'DSAttachJob(':JobName:',3)'
RunHandle = DSAttachJob(OldJobName, DSJ.ERRNONE)
JStat = DSGetJobInfo(RunHandle, DSJ.JOBSTATUS)
ContinueFlag = @true
CanRunFlag = 1
CanStopFlag = 0
CanResetFlag = 1
* -----------------------------------------------------------------------
begin case
case JStat = 0
JobStatus = 'Running'
CanRunFlag = 0
CanResetFlag = 0
CanStopFlag = 1
case JStat = 1
JobStatus = 'Finished'
case JStat = 2 ; * Finished with warnings
JobStatus = 'Finished'
case JStat = 3
ContinueFlag = @false
JobStatus = 'Aborted'
CanRunFlag = 0
case JStat = 11
JobStatus = 'Finished'
case JStat = 12 ; * Validated with warnings
JobStatus = 'Finished'
case JStat = 13
* ContinueFlag = @false
JobStatus = 'Finished'
case JStat = 21
JobStatus = 'Finished'
case JStat = 96
ContinueFlag = @false
JobStatus = 'Aborted'
CanRunFlag = 0
case JStat = 97
* ContinueFlag = @false
JobStatus = 'Stopped'
CanRunFlag = 0
case JStat = 98
JobStatus = 'Finished'
case JStat = 99
JobStatus = 'Finished'
case @true
ContinueFlag = @false
JobStatus = 'Not Compiled'
CanRunFlag = 0
CanResetFlag = 0
end case
* -----------------------------------------------------------------------
if ContinueFlag then
StageName = ''
StageStatus = ''
LinkName = ''
LinkType =''
RowCount = ''
StartedTime = ''
ElapsedTime = ''
RowsPerSec = ''
PercentCP = ''
* -----------------------------------------------------------------------
StartedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBSTARTTIMESTAMP)
iStartedDate = iconv(StartedDateTime[1,10], 'D4-YMD')
iStartedTime = iconv(StartedDateTime[12,8], 'MT')
if JobStatus = 'Running' then
FinishedDateTime = oconv(date(), 'D4-YMD'):' ':oconv(time(), 'MTS')
end else
FinishedDateTime = DSGetJobInfo(RunHandle, DSJ.JOBLASTTIMESTAMP)
end
iFinishedDate = iconv(FinishedDateTime[1,10], 'D4-YMD')
iFinishedTime = iconv(FinishedDateTime[12,8], 'MT')
* print iFinishedTime:' - ':iStartedTime
if FinishedDateTime<>'' and StartedDateTime<>'' and iFinishedTime<>'' and iStartedTime<>'' then
ElapsedTime = ((iFinishedDate-iStartedDate)*60*60)+iFinishedTime-iStartedTime
end
if ElapsedTime<0 then
ElapsedTime = ''
end
* -----------------------------------------------------------------------
* Get stage names
* -----------------------------------------------------------------------
Cnt = 0
StageNames = DSGetJobInfo(RunHandle, DSJ.STAGELIST)
NoStages = dcount(StageNames, ',')
for i=1 to NoStages
StageName = field(StageNames, ',', i)
* print i:'. ':StageName
* -----------------------------------------------------------------------
* Get link names per stage
* -----------------------------------------------------------------------
LinkType = DSGetStageInfo(RunHandle, StageName, DSJ.STAGETYPE)
LinkNames = DSGetStageInfo(RunHandle, StageName, DSJ.LINKLIST)
NoLinks = dcount(LinkNames, ',')
for j=1 to NoLinks
LinkName = field(LinkNames, ',', j)
* print j:'. ':LinkName
* -----------------------------------------------------------------------
RowCount = DSGetLinkInfo(RunHandle, StageName, LinkName, DSJ.LINKROWCOUNT)
if ElapsedTime>0 and ElapsedTime<>'' then
RowsPerSec = (RowCount / ElapsedTime)
RowsPerSec = oconv(RowsPerSec, 'MD0')
end
* -----------------------------------------------------------------------
Cnt += 1
if j=1 then
MonitorGrid < 1, Cnt> = StageName
MonitorGrid < 2, Cnt> = JobStatus
end else
MonitorGrid < 1, Cnt> = ' '
MonitorGrid < 2, Cnt> = ' '
end
MonitorGrid < 3, Cnt> = LinkName
MonitorGrid < 4, Cnt> = LinkType
MonitorGrid < 5, Cnt> = RowCount
MonitorGrid < 6, Cnt> = StartedDateTime
MonitorGrid < 7, Cnt> = ElapsedTime
MonitorGrid < 8, Cnt> = RowsPerSec
MonitorGrid < 9, Cnt> = PercentCP
MonitorGrid <10, Cnt> = CanRunFlag
MonitorGrid <11, Cnt> = CanStopFlag
MonitorGrid <12, Cnt> = CanResetFlag
MonitorGrid <13, Cnt> = FinishedDateTime
next j
next i
end
JStat = DSDetachJob(RunHandle)
end
* -----------------------------------------------------------------------
TheEnd:
return
Posted: Fri Dec 30, 2005 3:06 pm
by kduke
You should edit your post and ask a question.
Posted: Fri Dec 30, 2005 8:09 pm
by chulett
Actually, they did. Never understood why people felt the need to quote entire messages for no good reason and then compound the problem by burying their response
inside the quote.
Titto wrote:Hi Kim,
could you please guide me to write the MonitorGrid to a Flat file by Stage name, LinkName and other details
Thanks in advance
Posted: Fri Dec 30, 2005 9:54 pm
by kduke
Thanks Craig. I never saw the question.
You need to probably change this into a job. All this code would go in job control code. You need to openseq, writeseq for each line then closeseq. Ray has posted a lot of code to show you how.