Dumping The Log
Dumping The Log
This is a topic for an orphaned message.
You need this function. Here it is. There is an internal-to-DataStage function that you could call instead - I leave this as an exercise of your ingenuity. (Note, too, that the ArchiveJobLog function is not optimal - it can be improved in a number of ways.)
Function: GetJobNumber
Short Description: Returns the job number associated with a job name
Long Description:
Mapping of job names to job numbers is kept in a repository hashed file called DS_JOBS. The job number is required for access to the job log, and other status files at run time. This routine contains proprietary information of Informix Software, Inc. No guarantee is given or implied that DataStage itself uses the mechanisms or algorithms described in this code.
Code:
FUNCTION GetJobNumber(Arg1)
* Arg1 is the job name.
* We search for it on the DS_JOBS file, returning the corresponding
* job number if it is found.
JobName$ = Arg1
If IsNull(JobName$) Or JobName$ = ""
Then
Ans = @NULL
End
Else
Command = "SELECT DS_JOBS WITH NAME = ":Quote(JobName$):" SAVING JOBNO"
Call DSExecute("UV", Command, Output, System.Return.Code)
If @Selected = 1
Then
ReadList JobNumber Else JobNumber = @NULL
Ans = JobNumber
End
Else
Ans = @NULL
End
End
RETURN(Ans)
> ----------
> From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> Reply To: informix-datastage@oliver.com
> Sent: Thursday, 10 August 2000 19:53
> To: informix-datastage@oliver.com
> Subject: RE: Dumping The Log thru - Unix dsjob
>
> I am trying out this code it is giving me error at a function
> GetJobNumber do i need to define this function or have to include
> library for that regards Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
> ---------------------------------------- Message History
> ----------------------------------------
>
>
> From: ray.wurlod@informix.com on 10/08/2000 04:43 GMT
>
> Please respond to informix-datastage@oliver.com
>
> To: informix-datastage@oliver.com
> cc:
> Subject: RE: Dumping The Log thru - Unix dsjob
>
>
>
> A far more efficient way would be to construct a BASIC subroutine to
> dump the log, and run this either from within a Job Control routine or
> from within a before-job subroutine. A secondary benefit is that the
> output can be exactly as you desire.
> Such a technique is taught on the "Programming with DataStage BASIC"
> class.
> Here is a sample, set up as a before-job subroutine. JobNumber is another
> routine in the Routines branch of the repository. It can obviously be
> changed (improved), for example by decoding the severity.
>
> Short Description: Archives a DataStage log file
>
> Long Description:
> Archives a DataStage log file. The job name is provided as an input
> parameter. If no job name, or an invalid job name, is provided, then a
> warning message
> is logged.
> This routine contains proprietary information of Informix Software, Inc.
> No
> guarantee is given or implied that DataStage itself uses the mechanisms or
> algorithms described in this code.
>
> Code:
>
> SUBROUTINE ArchiveLog(InputArg, ErrorCode)
>
> DEFFUN GetJobNumber(Arg1) Calling "DSU.JobNumber"
>
> $INCLUDE UNIVERSE.INCLUDE FILEINFO.H
> EQUATE ArchiveDir To "/u03/dsdata/logarchive"
> EQUATE LF TO "+"
>
> ErrorCode = 0 ; * set this to non-zero to stop
> the stage/job
> RoutineName = "ArchiveLog"
>
> * InputArg is the name of the job whose log file is to be archived.
> JobName$ = InputArg
>
> * Initialize file variables
> ArchiveLog.Fvar = 0
> JobLog.Fvar = 0
>
> * Ensure that job name is given
> If IsNull(JobName$) Or Trim(JobName$) = ""
> Then
> Message = No job name given.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Determine job number (and, as a side effect, that the job name
> * is valid for the project)
> JobNo = GetJobNumber(JobName$)
> If IsNull(JobNo)
> Then
> Message = Job name (":JobName$:") is not valid for this
> project.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Create archive log file name and open it. Position to end of file
> * if archive log file already exists.
> LogFileName = ArchiveDir : "/" : JobName$
> OpenSeq LogFileName To ArchiveLog.Fvar
> On Error
> Message = Fatal error opening ":LogFileName:" file.
> Message = Error code = :Status()
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Locked
> Message = Unable to open log file ":LogFileName:".
> Message = It is already in use by DataStage process
> #:Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Then
> * Position to end of file
> Seek ArchiveLog.Fvar,0,2
> Else
> Message = Unable to position to end-of-file
> in":LogFileName:".
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> End
> Else
> If Status() = 0
> Then
> * File opened OK but does not yet exist. Create it.
> Create ArchiveLog.Fvar
> Else
> Message = Unable to create file ":LogFileName:".
> Call DSLogWarn(Message, RoutineName)
> GoTo MainExit
> End
> End
> Else
> Message = Unable to open log file ":LogFileName:".
> Message = Incorrect file type or non-existent directory.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> End
>
> * Open job log file (this is a hashed file)
> JobLogName = "RT_LOG" : JobNo
> Open JobLogName To JobLog.Fvar
> On Error
> Message = Fatal error opening ":JobLogName:" file.
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Then
> End
> Else
> Message = Unable to open log file ":JobLogName:".
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Ensure no other process is using the log file. This is
> * usually indicative that the job is running.
> FileLock JobLog.Fvar
> On Error
> Message = Fatal error attempting to lock ":JobLogName:" file.
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Locked
> Message = Unable to lock log file for job :JobName$:.
> Message = Check that job is not running.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Whew! Thats all the idiot-proofing done. Now we can archive the
> log.
> * In the log file, field 1 is the message severity (an integer),
> * field 10 is the message text, which can contain printf style
> parameters,
> * and field 5 is a list of parameter values. Fields 10 and 5 are
> multi-
> * valued. Field 3 is a timestamp, already in timestamp format.
>
> * In the output (the archive) we record the timestamp, the severity
> and
> * the full text of the message. Timestamp is in columns 1-19,
> * severity is
> * in column 21, and the message itself begins in column 23. Lines of
> * multiple line messages are separated by (A).
>
> * Form a Select List of log records in logged order. Since the log
> file
> has
> * a generated sequential numeric key, we can use this.
> SSELECT JobLog.Fvar
>
> * Process this list, bypassing any records that do not have a numeric
> key
> * (these are control records, like //PURGE.PARAMS and //SEQUENCE.NO).
> Loop
> While ReadNext LogID
> If Num(LogID)
> Then
> Read LogRecord From JobLog.Fvar,LogID
> On Error
> Message = Fatal error reading record ":LogID:" from
> job log.
> Message = Job name = ":JobName$:", log file
> is":JobLogName:".
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Then
> ArchiveLine = LogRecord ; * Timestamp
> ArchiveLine := " " : LogRecord : " " ; * severity
> code
> * [1.0.2]
> * FullText = Subr("DSR_MESSAGE", LogID, LogRecord,
> Raise(LogRecord))
> FullText = Subr("*DataStage*DSR_MESSAGE", LogID,
> LogRecord, Raise(LogRecord))
> * [1.0.1]
> * FullText = Ereplace(FullText, @VM, "(A)", -1, -1)
> FullText = Ereplace(FullText, @VM, LF, -1, -1)
> ArchiveLine := FullText
> WriteSeq ArchiveLine To ArchiveLog.Fvar
> Else
> Message = Unable to append to log
> file":LogFileName:".
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> End
> Else
> Message = Selected record ":LogID:" not found
> in":JobLogName:" file.
> Call DSLogWarn(Message, RoutineName)
> End
> End
> Repeat
> FileUnlock JobLog.Fvar
>
> MainExit:
> * Close any open files
> If FileInfo(ArchiveLog.Fvar, FINFO$IS.FILEVAR)
> Then
> CloseSeq ArchiveLog.Fvar
> End
> If FileInfo(JobLog.Fvar, FINFO$IS.FILEVAR)
> Then
> Close JobLog.Fvar
> End
>
> RETURN
>
> > ----------
> > From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> > Reply To: informix-datastage@oliver.com
> > Sent: Thursday, 10 August 2000 11:33
> > To: informix-mv@oliver.com
> > Cc: informix-datastage@oliver.com
> > Subject: Dumping The Log thru - Unix dsjob
> >
> > Hi
> > We are dumping the datastage log by following command from unix.
> >
> > dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
> > dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
> >
> > This unix command is working fine but the problem is that it is
> > creating
> a
> > lot of uv processes while
> > executing this command. for a single command it is creating around
> > 25 -
> 35
> > uv process. and some times
> > the uv process reach more than 100 ( only by this command).
> >
> > is there any way that we can limit the number of uv process for this
> > command? is there any efficient way to dump the datstage log.
> >
> >
> > thanks and regards
> > Sudhir
> > APHOIT/CA$H Datawarehouse Team
> > (65) 423-7449
> >
> > regards
> > Sudhir
> > APHOIT/CA$H Datawarehouse Team
> > (65) 423-7449
> >
> >
>
>
>
>
Function: GetJobNumber
Short Description: Returns the job number associated with a job name
Long Description:
Mapping of job names to job numbers is kept in a repository hashed file called DS_JOBS. The job number is required for access to the job log, and other status files at run time. This routine contains proprietary information of Informix Software, Inc. No guarantee is given or implied that DataStage itself uses the mechanisms or algorithms described in this code.
Code:
FUNCTION GetJobNumber(Arg1)
* Arg1 is the job name.
* We search for it on the DS_JOBS file, returning the corresponding
* job number if it is found.
JobName$ = Arg1
If IsNull(JobName$) Or JobName$ = ""
Then
Ans = @NULL
End
Else
Command = "SELECT DS_JOBS WITH NAME = ":Quote(JobName$):" SAVING JOBNO"
Call DSExecute("UV", Command, Output, System.Return.Code)
If @Selected = 1
Then
ReadList JobNumber Else JobNumber = @NULL
Ans = JobNumber
End
Else
Ans = @NULL
End
End
RETURN(Ans)
> ----------
> From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> Reply To: informix-datastage@oliver.com
> Sent: Thursday, 10 August 2000 19:53
> To: informix-datastage@oliver.com
> Subject: RE: Dumping The Log thru - Unix dsjob
>
> I am trying out this code it is giving me error at a function
> GetJobNumber do i need to define this function or have to include
> library for that regards Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
> ---------------------------------------- Message History
> ----------------------------------------
>
>
> From: ray.wurlod@informix.com on 10/08/2000 04:43 GMT
>
> Please respond to informix-datastage@oliver.com
>
> To: informix-datastage@oliver.com
> cc:
> Subject: RE: Dumping The Log thru - Unix dsjob
>
>
>
> A far more efficient way would be to construct a BASIC subroutine to
> dump the log, and run this either from within a Job Control routine or
> from within a before-job subroutine. A secondary benefit is that the
> output can be exactly as you desire.
> Such a technique is taught on the "Programming with DataStage BASIC"
> class.
> Here is a sample, set up as a before-job subroutine. JobNumber is another
> routine in the Routines branch of the repository. It can obviously be
> changed (improved), for example by decoding the severity.
>
> Short Description: Archives a DataStage log file
>
> Long Description:
> Archives a DataStage log file. The job name is provided as an input
> parameter. If no job name, or an invalid job name, is provided, then a
> warning message
> is logged.
> This routine contains proprietary information of Informix Software, Inc.
> No
> guarantee is given or implied that DataStage itself uses the mechanisms or
> algorithms described in this code.
>
> Code:
>
> SUBROUTINE ArchiveLog(InputArg, ErrorCode)
>
> DEFFUN GetJobNumber(Arg1) Calling "DSU.JobNumber"
>
> $INCLUDE UNIVERSE.INCLUDE FILEINFO.H
> EQUATE ArchiveDir To "/u03/dsdata/logarchive"
> EQUATE LF TO "+"
>
> ErrorCode = 0 ; * set this to non-zero to stop
> the stage/job
> RoutineName = "ArchiveLog"
>
> * InputArg is the name of the job whose log file is to be archived.
> JobName$ = InputArg
>
> * Initialize file variables
> ArchiveLog.Fvar = 0
> JobLog.Fvar = 0
>
> * Ensure that job name is given
> If IsNull(JobName$) Or Trim(JobName$) = ""
> Then
> Message = No job name given.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Determine job number (and, as a side effect, that the job name
> * is valid for the project)
> JobNo = GetJobNumber(JobName$)
> If IsNull(JobNo)
> Then
> Message = Job name (":JobName$:") is not valid for this
> project.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Create archive log file name and open it. Position to end of file
> * if archive log file already exists.
> LogFileName = ArchiveDir : "/" : JobName$
> OpenSeq LogFileName To ArchiveLog.Fvar
> On Error
> Message = Fatal error opening ":LogFileName:" file.
> Message = Error code = :Status()
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Locked
> Message = Unable to open log file ":LogFileName:".
> Message = It is already in use by DataStage process
> #:Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Then
> * Position to end of file
> Seek ArchiveLog.Fvar,0,2
> Else
> Message = Unable to position to end-of-file
> in":LogFileName:".
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> End
> Else
> If Status() = 0
> Then
> * File opened OK but does not yet exist. Create it.
> Create ArchiveLog.Fvar
> Else
> Message = Unable to create file ":LogFileName:".
> Call DSLogWarn(Message, RoutineName)
> GoTo MainExit
> End
> End
> Else
> Message = Unable to open log file ":LogFileName:".
> Message = Incorrect file type or non-existent directory.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> End
>
> * Open job log file (this is a hashed file)
> JobLogName = "RT_LOG" : JobNo
> Open JobLogName To JobLog.Fvar
> On Error
> Message = Fatal error opening ":JobLogName:" file.
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Then
> End
> Else
> Message = Unable to open log file ":JobLogName:".
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Ensure no other process is using the log file. This is
> * usually indicative that the job is running.
> FileLock JobLog.Fvar
> On Error
> Message = Fatal error attempting to lock ":JobLogName:" file.
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Locked
> Message = Unable to lock log file for job :JobName$:.
> Message = Check that job is not running.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
>
> * Whew! Thats all the idiot-proofing done. Now we can archive the
> log.
> * In the log file, field 1 is the message severity (an integer),
> * field 10 is the message text, which can contain printf style
> parameters,
> * and field 5 is a list of parameter values. Fields 10 and 5 are
> multi-
> * valued. Field 3 is a timestamp, already in timestamp format.
>
> * In the output (the archive) we record the timestamp, the severity
> and
> * the full text of the message. Timestamp is in columns 1-19,
> * severity is
> * in column 21, and the message itself begins in column 23. Lines of
> * multiple line messages are separated by (A).
>
> * Form a Select List of log records in logged order. Since the log
> file
> has
> * a generated sequential numeric key, we can use this.
> SSELECT JobLog.Fvar
>
> * Process this list, bypassing any records that do not have a numeric
> key
> * (these are control records, like //PURGE.PARAMS and //SEQUENCE.NO).
> Loop
> While ReadNext LogID
> If Num(LogID)
> Then
> Read LogRecord From JobLog.Fvar,LogID
> On Error
> Message = Fatal error reading record ":LogID:" from
> job log.
> Message = Job name = ":JobName$:", log file
> is":JobLogName:".
> Message = Error code = :Status():.
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> Then
> ArchiveLine = LogRecord ; * Timestamp
> ArchiveLine := " " : LogRecord : " " ; * severity
> code
> * [1.0.2]
> * FullText = Subr("DSR_MESSAGE", LogID, LogRecord,
> Raise(LogRecord))
> FullText = Subr("*DataStage*DSR_MESSAGE", LogID,
> LogRecord, Raise(LogRecord))
> * [1.0.1]
> * FullText = Ereplace(FullText, @VM, "(A)", -1, -1)
> FullText = Ereplace(FullText, @VM, LF, -1, -1)
> ArchiveLine := FullText
> WriteSeq ArchiveLine To ArchiveLog.Fvar
> Else
> Message = Unable to append to log
> file":LogFileName:".
> Call DSLogWarn(Message, RoutineName)
> ErrorCode = -1
> GoTo MainExit
> End
> End
> Else
> Message = Selected record ":LogID:" not found
> in":JobLogName:" file.
> Call DSLogWarn(Message, RoutineName)
> End
> End
> Repeat
> FileUnlock JobLog.Fvar
>
> MainExit:
> * Close any open files
> If FileInfo(ArchiveLog.Fvar, FINFO$IS.FILEVAR)
> Then
> CloseSeq ArchiveLog.Fvar
> End
> If FileInfo(JobLog.Fvar, FINFO$IS.FILEVAR)
> Then
> Close JobLog.Fvar
> End
>
> RETURN
>
> > ----------
> > From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> > Reply To: informix-datastage@oliver.com
> > Sent: Thursday, 10 August 2000 11:33
> > To: informix-mv@oliver.com
> > Cc: informix-datastage@oliver.com
> > Subject: Dumping The Log thru - Unix dsjob
> >
> > Hi
> > We are dumping the datastage log by following command from unix.
> >
> > dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
> > dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
> >
> > This unix command is working fine but the problem is that it is
> > creating
> a
> > lot of uv processes while
> > executing this command. for a single command it is creating around
> > 25 -
> 35
> > uv process. and some times
> > the uv process reach more than 100 ( only by this command).
> >
> > is there any way that we can limit the number of uv process for this
> > command? is there any efficient way to dump the datstage log.
> >
> >
> > thanks and regards
> > Sudhir
> > APHOIT/CA$H Datawarehouse Team
> > (65) 423-7449
> >
> > regards
> > Sudhir
> > APHOIT/CA$H Datawarehouse Team
> > (65) 423-7449
> >
> >
>
>
>
>