Issue retrieving number of rows processed

ds_team · Post by **ds_team** » Mon Jan 15, 2007 1:35 am

I have two stages in my job, one ODBC stage named say "ODBC1" and another sequential file stage named say "FILE1".

ODBC1---------------->FILE1

The ODBC stage executes a stored procedure and the output rows are stored in the sequential file.
In the after job subroutine of this job i call a routine with some input parameters. This routines writes to a log file number of rows retrieved by ODBC stage to sequential file stage. For getting number of rows processed i was passing the sequential file stage name ie; "FILE1" as my first input parameter to my subroutine. In the routine i have code as below for getting total number of rows processed.
IStage = Field(InputArg,';',1)
JobHandle = DSJ.ME
ReportText ="***** Total rows processed: " : DSGetStageInfo(JobHandle, IStage , DSJ.STAGEINROWNUM) : Char(13) : Char(10)

But this was giving total number of rows processed as -7 always. So i passed the stage name as "FILE1.IDENT1" as input parameter and it did give proper output.

1) MY FIRST QUESTION IS WHY DO I HAVE TO PASS THE stage name as FILE1.IDENT1 instead of FILE1

2) Second issue i have is my same job with out any after job routine is getting aborted by throwing the following warning.

"Abnormal termination of stage FILE1.IDENT1 detected."

when i run this job individually there are rare chances of errors. but when i call this job using a job activity in a sequence it is randomly giving me erros. could you please suggest me a reason to avoid this ramdom error.

kumar_s · Post by **kumar_s** » Mon Jan 15, 2007 1:55 am

Hi,

Try to include a Transformer in between the two passive stages and find the stage info for that. I rather use DSGetLinkInfo(JobHandle, StageName, LinkName, InfoType).

ds_team · Post by **ds_team** » Mon Jan 15, 2007 2:03 am

The link name you mentioned is it the name of the link between transformer and sequential file stage?
What should i give for Infotype?

please answer my second question too. My same job is getting aborted randomly in the sequence eventhough i dont have a after job subroutine. Is it some other problem, may be network connection.

kumar_s wrote:Hi,

Try to include a Transformer in between the two passive stages and find the stage info for that. I rather use DSGetLinkInfo(JobHandle, StageName, LinkName, InfoType). ...

kumar_s · Post by **kumar_s** » Mon Jan 15, 2007 2:11 am

DSJ.LINKROWCOUNT is the info type.
The IDENT1 is an invisible Transformer stage that the job compiler has inserted because your design includes a link between two passive stages. Hope both the issue should get solved by including a transformer between the stages.

ds_team · Post by **ds_team** » Mon Jan 15, 2007 3:06 am

I inserted a transformer stage and still it is giving that random error. My second scenario is still occuring. its getting aborted randomly saying

"Abnormal termination of stage sp_crepp_mi_ohq_extr_jan15..Transformer_8"

sp_crepp_mi_ohq_extr_jan15=jobname
transformer name=Transformer_8
Is there any possibility that this is hapenning due to network issue?

kumar_s wrote:DSJ.LINKROWCOUNT is the info type.
The IDENT1 is an invisible Transformer stage that the job compiler has inserted because your design includes a link between two passive stages. Hope both the issue should get solved by including a transformer between the stages.

kcbland · Post by **kcbland** » Mon Jan 15, 2007 7:34 am

Abnormal terminations are usually data related, such as using a NULL in a math calculation or using an uncompiled Function in a derivation/constraint.

rameshrr3 · Post by **rameshrr3** » Mon Jan 15, 2007 7:48 am

The Invisible transformer stage is probably termed as IDENTITY stage. Thats why you have the 'IDENT'.

chulett · Post by **chulett** » Mon Jan 15, 2007 7:57 am

As noted in your other thread on this issue, you need to reset the aborted job and let us know if anything labelled 'From previous run...' shows up.

kumar_s · Post by **kumar_s** » Tue Jan 16, 2007 2:27 am

In your other post Ray has given all these info with the necessary code to include in your routine.
$IFNDEF JOBCONTROL.H
$INCLUDE DSINCLUDE JOBCONTROL.H
$ENDIF

chulett · Post by **chulett** » Tue Jan 16, 2007 8:17 am

I don't see how that would help. The OP stated that it failed occassionally, not constantly. Lack of that JOBCONTROL header file would cause something that needed it to not even run.

kumar_s · Post by **kumar_s** » Tue Jan 16, 2007 4:51 pm

chulett wrote: I don't see how that would help. The OP stated that it failed occassionally, not constantly. Lack of that JOBCONTROL header file would cause something that needed it to not even run.

Its a good point to consider. For testing purpose, change the routing to ExecSH and use dsjob --linkinfo with the respective parameters.

ds_team · Post by **ds_team** » Fri Jan 19, 2007 6:15 am

I am still getting abnormal termination while running the job and that too randomly. I am using the same scenario and data is also same in all the cases and it is aborting randomly giving a warning abnormal termination. I have only two stages as below

ODBC ----------> Sequential file stage

And in all test cases I have same scenario. But it gets aborted randomly.
What can be the posible reason for this. Previous we had a patch update on our database. But since we were getting these issues they reverted back the patch update. Still we are having the abnormal termination issue.

kumar_s wrote:
chulett wrote: I don't see how that would help. The OP stated that it failed occassionally, not constantly. Lack of that JOBCONTROL header file would cause something that needed it to not even run.
Its a good point to consider. For testing purpose, change the routing to ExecSH and use dsjob --linkinfo with the respective parameters.

chulett · Post by **chulett** » Fri Jan 19, 2007 7:50 am

Add a transformer between the two stages, then use that name when you go to retrieve the rows processed. As a general rule, I never link two passive stages directly together. DataStage will add an 'invisible' one for you, why not make it visible and have full control? Besides, I've seen way too many flakey, 'it only happens occassionally' problems from jobs designed like that that went away when a 'real' transformer was introduced.

Give that a shot.

kumar_s · Post by **kumar_s** » Sun Jan 21, 2007 9:11 pm

ds_team wrote:I inserted a transformer stage and still it is giving that random error.

Have you tried the other approaches suggested?

chulett · Post by **chulett** » Sun Jan 21, 2007 10:17 pm

ds_team wrote:I have only two stages as below

ODBC ----------> Sequential file stage

Four days after the post you quoted.

Suggest it goes back in the job design and stays there. Regardless.