Hi
We are dumping the datastage log by following command from unix.
dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
This unix command is working fine but the problem is that it is creating a lot of uv processes while executing this command. for a single command it is creating around 25 - 35 uv process. and some times the uv process reach more than 100 ( only by this command).
is there any way that we can limit the number of uv process for this command? is there any efficient way to dump the datstage log.
thanks and regards
Sudhir
APHOIT/CA$H Datawarehouse Team
(65) 423-7449
regards
Sudhir
APHOIT/CA$H Datawarehouse Team
(65) 423-7449
Dumping The Log thru - Unix dsjob
Thanks, Ray. Ive been meaning to write one of these for a long time, just hadnt gotten that far down my list yet. Now the job is half done for me already. 
I was also contemplating decoding the stage finished messages as well so I can track over time how long the jobs take, when they start and finish, how many rows they process etc. This would be very useful in identifying developing performance problems as well as quantifying performance improvements.
Dont suppose you have one of these lying around too?? (Hey, it was worth
asking.)
-----Original Message-----
From: Ray Wurlod [SMTP:ray.wurlod@informix.com]
Sent: Thursday, 10 August 2000 14:43
To: informix-datastage@oliver.com
Subject: RE: Dumping The Log thru - Unix dsjob
A far more efficient way would be to construct a BASIC subroutine to dump
the log, and run this either from within a Job Control routine or from
within a before-job subroutine. A secondary benefit is that the output can
be exactly as you desire.
Such a technique is taught on the "Programming with DataStage BASIC" class.
Here is a sample, set up as a before-job subroutine. JobNumber is another
routine in the Routines branch of the repository. It can obviously be
changed (improved), for example by decoding the severity.
*************************************************************************
This e-mail and any files transmitted with it may be confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in
error, please notify the sender by return e-mail, and delete this e-mail from your in-box. Do not copy it to anybody else
*************************************************************************
I was also contemplating decoding the stage finished messages as well so I can track over time how long the jobs take, when they start and finish, how many rows they process etc. This would be very useful in identifying developing performance problems as well as quantifying performance improvements.
Dont suppose you have one of these lying around too?? (Hey, it was worth
asking.)
-----Original Message-----
From: Ray Wurlod [SMTP:ray.wurlod@informix.com]
Sent: Thursday, 10 August 2000 14:43
To: informix-datastage@oliver.com
Subject: RE: Dumping The Log thru - Unix dsjob
A far more efficient way would be to construct a BASIC subroutine to dump
the log, and run this either from within a Job Control routine or from
within a before-job subroutine. A secondary benefit is that the output can
be exactly as you desire.
Such a technique is taught on the "Programming with DataStage BASIC" class.
Here is a sample, set up as a before-job subroutine. JobNumber is another
routine in the Routines branch of the repository. It can obviously be
changed (improved), for example by decoding the severity.
*************************************************************************
This e-mail and any files transmitted with it may be confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in
error, please notify the sender by return e-mail, and delete this e-mail from your in-box. Do not copy it to anybody else
*************************************************************************
A far more efficient way would be to construct a BASIC subroutine to dump the log, and run this either from within a Job Control routine or from within a before-job subroutine. A secondary benefit is that the output can be exactly as you desire. Such a technique is taught on the "Programming with DataStage BASIC" class. Here is a sample, set up as a before-job subroutine. JobNumber is another routine in the Routines branch of the repository. It can obviously be changed (improved), for example by decoding the severity.
Short Description: Archives a DataStage log file
Long Description:
Archives a DataStage log file. The job name is provided as an input parameter. If no job name, or an invalid job name, is provided, then a warning message is logged. This routine contains proprietary information of Informix Software, Inc. No guarantee is given or implied that DataStage itself uses the mechanisms or algorithms described in this code.
Code:
SUBROUTINE ArchiveLog(InputArg, ErrorCode)
DEFFUN GetJobNumber(Arg1) Calling "DSU.JobNumber"
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
EQUATE ArchiveDir To "/u03/dsdata/logarchive"
EQUATE LF TO "+"
ErrorCode = 0 ; * set this to non-zero to stop
the stage/job
RoutineName = "ArchiveLog"
* InputArg is the name of the job whose log file is to be archived.
JobName$ = InputArg
* Initialize file variables
ArchiveLog.Fvar = 0
JobLog.Fvar = 0
* Ensure that job name is given
If IsNull(JobName$) Or Trim(JobName$) = ""
Then
Message = No job name given.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Determine job number (and, as a side effect, that the job name
* is valid for the project)
JobNo = GetJobNumber(JobName$)
If IsNull(JobNo)
Then
Message = Job name (":JobName$:") is not valid for this project.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Create archive log file name and open it. Position to end of file
* if archive log file already exists.
LogFileName = ArchiveDir : "/" : JobName$
OpenSeq LogFileName To ArchiveLog.Fvar
On Error
Message = Fatal error opening ":LogFileName:" file.
Message = Error code = :Status()
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to open log file ":LogFileName:".
Message = It is already in use by DataStage process #:Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
* Position to end of file
Seek ArchiveLog.Fvar,0,2
Else
Message = Unable to position to end-of-file in":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
If Status() = 0
Then
* File opened OK but does not yet exist. Create it.
Create ArchiveLog.Fvar
Else
Message = Unable to create file ":LogFileName:".
Call DSLogWarn(Message, RoutineName)
GoTo MainExit
End
End
Else
Message = Unable to open log file ":LogFileName:".
Message = Incorrect file type or non-existent directory.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
* Open job log file (this is a hashed file)
JobLogName = "RT_LOG" : JobNo
Open JobLogName To JobLog.Fvar
On Error
Message = Fatal error opening ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
End
Else
Message = Unable to open log file ":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Ensure no other process is using the log file. This is
* usually indicative that the job is running.
FileLock JobLog.Fvar
On Error
Message = Fatal error attempting to lock ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to lock log file for job :JobName$:.
Message = Check that job is not running.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Whew! Thats all the idiot-proofing done. Now we can archive the
log.
* In the log file, field 1 is the message severity (an integer),
* field 10 is the message text, which can contain printf style
parameters,
* and field 5 is a list of parameter values. Fields 10 and 5 are multi-
* valued. Field 3 is a timestamp, already in timestamp format.
* In the output (the archive) we record the timestamp, the severity and
* the full text of the message. Timestamp is in columns 1-19,
* severity is
* in column 21, and the message itself begins in column 23. Lines of
* multiple line messages are separated by (A).
* Form a Select List of log records in logged order. Since the log file
has
* a generated sequential numeric key, we can use this.
SSELECT JobLog.Fvar
* Process this list, bypassing any records that do not have a numeric
key
* (these are control records, like //PURGE.PARAMS and //SEQUENCE.NO).
Loop
While ReadNext LogID
If Num(LogID)
Then
Read LogRecord From JobLog.Fvar,LogID
On Error
Message = Fatal error reading record ":LogID:" from job log.
Message = Job name = ":JobName$:", log file is":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
ArchiveLine = LogRecord ; * Timestamp
ArchiveLine := " " : LogRecord : " " ; * severity
code
* [1.0.2]
* FullText = Subr("DSR_MESSAGE", LogID, LogRecord,
Raise(LogRecord))
FullText = Subr("*DataStage*DSR_MESSAGE", LogID, LogRecord, Raise(LogRecord))
* [1.0.1]
* FullText = Ereplace(FullText, @VM, "(A)", -1, -1)
FullText = Ereplace(FullText, @VM, LF, -1, -1)
ArchiveLine := FullText
WriteSeq ArchiveLine To ArchiveLog.Fvar
Else
Message = Unable to append to log file":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
Message = Selected record ":LogID:" not found in":JobLogName:" file.
Call DSLogWarn(Message, RoutineName)
End
End
Repeat
FileUnlock JobLog.Fvar
MainExit:
* Close any open files
If FileInfo(ArchiveLog.Fvar, FINFO$IS.FILEVAR)
Then
CloseSeq ArchiveLog.Fvar
End
If FileInfo(JobLog.Fvar, FINFO$IS.FILEVAR)
Then
Close JobLog.Fvar
End
RETURN
> ----------
> From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> Reply To: informix-datastage@oliver.com
> Sent: Thursday, 10 August 2000 11:33
> To: informix-mv@oliver.com
> Cc: informix-datastage@oliver.com
> Subject: Dumping The Log thru - Unix dsjob
>
> Hi
> We are dumping the datastage log by following command from unix.
>
> dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
> dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
>
> This unix command is working fine but the problem is that it is
> creating a lot of uv processes while executing this command. for a
> single command it is creating around 25 - 35 uv process. and some
> times the uv process reach more than 100 ( only by this command).
>
> is there any way that we can limit the number of uv process for this
> command? is there any efficient way to dump the datstage log.
>
>
> thanks and regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
> regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
Short Description: Archives a DataStage log file
Long Description:
Archives a DataStage log file. The job name is provided as an input parameter. If no job name, or an invalid job name, is provided, then a warning message is logged. This routine contains proprietary information of Informix Software, Inc. No guarantee is given or implied that DataStage itself uses the mechanisms or algorithms described in this code.
Code:
SUBROUTINE ArchiveLog(InputArg, ErrorCode)
DEFFUN GetJobNumber(Arg1) Calling "DSU.JobNumber"
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
EQUATE ArchiveDir To "/u03/dsdata/logarchive"
EQUATE LF TO "+"
ErrorCode = 0 ; * set this to non-zero to stop
the stage/job
RoutineName = "ArchiveLog"
* InputArg is the name of the job whose log file is to be archived.
JobName$ = InputArg
* Initialize file variables
ArchiveLog.Fvar = 0
JobLog.Fvar = 0
* Ensure that job name is given
If IsNull(JobName$) Or Trim(JobName$) = ""
Then
Message = No job name given.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Determine job number (and, as a side effect, that the job name
* is valid for the project)
JobNo = GetJobNumber(JobName$)
If IsNull(JobNo)
Then
Message = Job name (":JobName$:") is not valid for this project.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Create archive log file name and open it. Position to end of file
* if archive log file already exists.
LogFileName = ArchiveDir : "/" : JobName$
OpenSeq LogFileName To ArchiveLog.Fvar
On Error
Message = Fatal error opening ":LogFileName:" file.
Message = Error code = :Status()
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to open log file ":LogFileName:".
Message = It is already in use by DataStage process #:Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
* Position to end of file
Seek ArchiveLog.Fvar,0,2
Else
Message = Unable to position to end-of-file in":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
If Status() = 0
Then
* File opened OK but does not yet exist. Create it.
Create ArchiveLog.Fvar
Else
Message = Unable to create file ":LogFileName:".
Call DSLogWarn(Message, RoutineName)
GoTo MainExit
End
End
Else
Message = Unable to open log file ":LogFileName:".
Message = Incorrect file type or non-existent directory.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
* Open job log file (this is a hashed file)
JobLogName = "RT_LOG" : JobNo
Open JobLogName To JobLog.Fvar
On Error
Message = Fatal error opening ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
End
Else
Message = Unable to open log file ":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Ensure no other process is using the log file. This is
* usually indicative that the job is running.
FileLock JobLog.Fvar
On Error
Message = Fatal error attempting to lock ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to lock log file for job :JobName$:.
Message = Check that job is not running.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Whew! Thats all the idiot-proofing done. Now we can archive the
log.
* In the log file, field 1 is the message severity (an integer),
* field 10 is the message text, which can contain printf style
parameters,
* and field 5 is a list of parameter values. Fields 10 and 5 are multi-
* valued. Field 3 is a timestamp, already in timestamp format.
* In the output (the archive) we record the timestamp, the severity and
* the full text of the message. Timestamp is in columns 1-19,
* severity is
* in column 21, and the message itself begins in column 23. Lines of
* multiple line messages are separated by (A).
* Form a Select List of log records in logged order. Since the log file
has
* a generated sequential numeric key, we can use this.
SSELECT JobLog.Fvar
* Process this list, bypassing any records that do not have a numeric
key
* (these are control records, like //PURGE.PARAMS and //SEQUENCE.NO).
Loop
While ReadNext LogID
If Num(LogID)
Then
Read LogRecord From JobLog.Fvar,LogID
On Error
Message = Fatal error reading record ":LogID:" from job log.
Message = Job name = ":JobName$:", log file is":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
ArchiveLine = LogRecord ; * Timestamp
ArchiveLine := " " : LogRecord : " " ; * severity
code
* [1.0.2]
* FullText = Subr("DSR_MESSAGE", LogID, LogRecord,
Raise(LogRecord))
FullText = Subr("*DataStage*DSR_MESSAGE", LogID, LogRecord, Raise(LogRecord))
* [1.0.1]
* FullText = Ereplace(FullText, @VM, "(A)", -1, -1)
FullText = Ereplace(FullText, @VM, LF, -1, -1)
ArchiveLine := FullText
WriteSeq ArchiveLine To ArchiveLog.Fvar
Else
Message = Unable to append to log file":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
Message = Selected record ":LogID:" not found in":JobLogName:" file.
Call DSLogWarn(Message, RoutineName)
End
End
Repeat
FileUnlock JobLog.Fvar
MainExit:
* Close any open files
If FileInfo(ArchiveLog.Fvar, FINFO$IS.FILEVAR)
Then
CloseSeq ArchiveLog.Fvar
End
If FileInfo(JobLog.Fvar, FINFO$IS.FILEVAR)
Then
Close JobLog.Fvar
End
RETURN
> ----------
> From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> Reply To: informix-datastage@oliver.com
> Sent: Thursday, 10 August 2000 11:33
> To: informix-mv@oliver.com
> Cc: informix-datastage@oliver.com
> Subject: Dumping The Log thru - Unix dsjob
>
> Hi
> We are dumping the datastage log by following command from unix.
>
> dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
> dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
>
> This unix command is working fine but the problem is that it is
> creating a lot of uv processes while executing this command. for a
> single command it is creating around 25 - 35 uv process. and some
> times the uv process reach more than 100 ( only by this command).
>
> is there any way that we can limit the number of uv process for this
> command? is there any efficient way to dump the datstage log.
>
>
> thanks and regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
> regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
I am trying out this code it is giving me error at a function GetJobNumber do i need to define this function or have to include library for that regards Sudhir APHOIT/CA$H Datawarehouse Team
(65) 423-7449
---------------------------------------- Message History ----------------------------------------
From: ray.wurlod@informix.com on 10/08/2000 04:43 GMT
Please respond to informix-datastage@oliver.com
To: informix-datastage@oliver.com
cc:
Subject: RE: Dumping The Log thru - Unix dsjob
A far more efficient way would be to construct a BASIC subroutine to dump the log, and run this either from within a Job Control routine or from within a before-job subroutine. A secondary benefit is that the output can be exactly as you desire. Such a technique is taught on the "Programming with DataStage BASIC" class. Here is a sample, set up as a before-job subroutine. JobNumber is another routine in the Routines branch of the repository. It can obviously be changed (improved), for example by decoding the severity.
Short Description: Archives a DataStage log file
Long Description:
Archives a DataStage log file. The job name is provided as an input parameter. If no job name, or an invalid job name, is provided, then a warning message is logged. This routine contains proprietary information of Informix Software, Inc. No guarantee is given or implied that DataStage itself uses the mechanisms or algorithms described in this code.
Code:
SUBROUTINE ArchiveLog(InputArg, ErrorCode)
DEFFUN GetJobNumber(Arg1) Calling "DSU.JobNumber"
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
EQUATE ArchiveDir To "/u03/dsdata/logarchive"
EQUATE LF TO "+"
ErrorCode = 0 ; * set this to non-zero to stop
the stage/job
RoutineName = "ArchiveLog"
* InputArg is the name of the job whose log file is to be archived.
JobName$ = InputArg
* Initialize file variables
ArchiveLog.Fvar = 0
JobLog.Fvar = 0
* Ensure that job name is given
If IsNull(JobName$) Or Trim(JobName$) = ""
Then
Message = No job name given.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Determine job number (and, as a side effect, that the job name
* is valid for the project)
JobNo = GetJobNumber(JobName$)
If IsNull(JobNo)
Then
Message = Job name (":JobName$:") is not valid for this project.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Create archive log file name and open it. Position to end of file
* if archive log file already exists.
LogFileName = ArchiveDir : "/" : JobName$
OpenSeq LogFileName To ArchiveLog.Fvar
On Error
Message = Fatal error opening ":LogFileName:" file.
Message = Error code = :Status()
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to open log file ":LogFileName:".
Message = It is already in use by DataStage process #:Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
* Position to end of file
Seek ArchiveLog.Fvar,0,2
Else
Message = Unable to position to end-of-file in":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
If Status() = 0
Then
* File opened OK but does not yet exist. Create it.
Create ArchiveLog.Fvar
Else
Message = Unable to create file ":LogFileName:".
Call DSLogWarn(Message, RoutineName)
GoTo MainExit
End
End
Else
Message = Unable to open log file ":LogFileName:".
Message = Incorrect file type or non-existent directory.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
* Open job log file (this is a hashed file)
JobLogName = "RT_LOG" : JobNo
Open JobLogName To JobLog.Fvar
On Error
Message = Fatal error opening ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
End
Else
Message = Unable to open log file ":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Ensure no other process is using the log file. This is
* usually indicative that the job is running.
FileLock JobLog.Fvar
On Error
Message = Fatal error attempting to lock ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to lock log file for job :JobName$:.
Message = Check that job is not running.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Whew! Thats all the idiot-proofing done. Now we can archive the
log.
* In the log file, field 1 is the message severity (an integer),
* field 10 is the message text, which can contain printf style
parameters,
* and field 5 is a list of parameter values. Fields 10 and 5 are multi-
* valued. Field 3 is a timestamp, already in timestamp format.
* In the output (the archive) we record the timestamp, the severity and
* the full text of the message. Timestamp is in columns 1-19,
* severity is
* in column 21, and the message itself begins in column 23. Lines of
* multiple line messages are separated by (A).
* Form a Select List of log records in logged order. Since the log file
has
* a generated sequential numeric key, we can use this.
SSELECT JobLog.Fvar
* Process this list, bypassing any records that do not have a numeric
key
* (these are control records, like //PURGE.PARAMS and //SEQUENCE.NO).
Loop
While ReadNext LogID
If Num(LogID)
Then
Read LogRecord From JobLog.Fvar,LogID
On Error
Message = Fatal error reading record ":LogID:" from job log.
Message = Job name = ":JobName$:", log file is":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
ArchiveLine = LogRecord ; * Timestamp
ArchiveLine := " " : LogRecord : " " ; * severity
code
* [1.0.2]
* FullText = Subr("DSR_MESSAGE", LogID, LogRecord,
Raise(LogRecord))
FullText = Subr("*DataStage*DSR_MESSAGE", LogID, LogRecord, Raise(LogRecord))
* [1.0.1]
* FullText = Ereplace(FullText, @VM, "(A)", -1, -1)
FullText = Ereplace(FullText, @VM, LF, -1, -1)
ArchiveLine := FullText
WriteSeq ArchiveLine To ArchiveLog.Fvar
Else
Message = Unable to append to log file":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
Message = Selected record ":LogID:" not found in":JobLogName:" file.
Call DSLogWarn(Message, RoutineName)
End
End
Repeat
FileUnlock JobLog.Fvar
MainExit:
* Close any open files
If FileInfo(ArchiveLog.Fvar, FINFO$IS.FILEVAR)
Then
CloseSeq ArchiveLog.Fvar
End
If FileInfo(JobLog.Fvar, FINFO$IS.FILEVAR)
Then
Close JobLog.Fvar
End
RETURN
> ----------
> From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> Reply To: informix-datastage@oliver.com
> Sent: Thursday, 10 August 2000 11:33
> To: informix-mv@oliver.com
> Cc: informix-datastage@oliver.com
> Subject: Dumping The Log thru - Unix dsjob
>
> Hi
> We are dumping the datastage log by following command from unix.
>
> dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
> dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
>
> This unix command is working fine but the problem is that it is
> creating a lot of uv processes while executing this command. for a
> single command it is creating around 25 - 35 uv process. and some
> times the uv process reach more than 100 ( only by this command).
>
> is there any way that we can limit the number of uv process for this
> command? is there any efficient way to dump the datstage log.
>
>
> thanks and regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
> regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
(65) 423-7449
---------------------------------------- Message History ----------------------------------------
From: ray.wurlod@informix.com on 10/08/2000 04:43 GMT
Please respond to informix-datastage@oliver.com
To: informix-datastage@oliver.com
cc:
Subject: RE: Dumping The Log thru - Unix dsjob
A far more efficient way would be to construct a BASIC subroutine to dump the log, and run this either from within a Job Control routine or from within a before-job subroutine. A secondary benefit is that the output can be exactly as you desire. Such a technique is taught on the "Programming with DataStage BASIC" class. Here is a sample, set up as a before-job subroutine. JobNumber is another routine in the Routines branch of the repository. It can obviously be changed (improved), for example by decoding the severity.
Short Description: Archives a DataStage log file
Long Description:
Archives a DataStage log file. The job name is provided as an input parameter. If no job name, or an invalid job name, is provided, then a warning message is logged. This routine contains proprietary information of Informix Software, Inc. No guarantee is given or implied that DataStage itself uses the mechanisms or algorithms described in this code.
Code:
SUBROUTINE ArchiveLog(InputArg, ErrorCode)
DEFFUN GetJobNumber(Arg1) Calling "DSU.JobNumber"
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
EQUATE ArchiveDir To "/u03/dsdata/logarchive"
EQUATE LF TO "+"
ErrorCode = 0 ; * set this to non-zero to stop
the stage/job
RoutineName = "ArchiveLog"
* InputArg is the name of the job whose log file is to be archived.
JobName$ = InputArg
* Initialize file variables
ArchiveLog.Fvar = 0
JobLog.Fvar = 0
* Ensure that job name is given
If IsNull(JobName$) Or Trim(JobName$) = ""
Then
Message = No job name given.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Determine job number (and, as a side effect, that the job name
* is valid for the project)
JobNo = GetJobNumber(JobName$)
If IsNull(JobNo)
Then
Message = Job name (":JobName$:") is not valid for this project.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Create archive log file name and open it. Position to end of file
* if archive log file already exists.
LogFileName = ArchiveDir : "/" : JobName$
OpenSeq LogFileName To ArchiveLog.Fvar
On Error
Message = Fatal error opening ":LogFileName:" file.
Message = Error code = :Status()
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to open log file ":LogFileName:".
Message = It is already in use by DataStage process #:Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
* Position to end of file
Seek ArchiveLog.Fvar,0,2
Else
Message = Unable to position to end-of-file in":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
If Status() = 0
Then
* File opened OK but does not yet exist. Create it.
Create ArchiveLog.Fvar
Else
Message = Unable to create file ":LogFileName:".
Call DSLogWarn(Message, RoutineName)
GoTo MainExit
End
End
Else
Message = Unable to open log file ":LogFileName:".
Message = Incorrect file type or non-existent directory.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
* Open job log file (this is a hashed file)
JobLogName = "RT_LOG" : JobNo
Open JobLogName To JobLog.Fvar
On Error
Message = Fatal error opening ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
End
Else
Message = Unable to open log file ":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Ensure no other process is using the log file. This is
* usually indicative that the job is running.
FileLock JobLog.Fvar
On Error
Message = Fatal error attempting to lock ":JobLogName:" file.
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Locked
Message = Unable to lock log file for job :JobName$:.
Message = Check that job is not running.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
* Whew! Thats all the idiot-proofing done. Now we can archive the
log.
* In the log file, field 1 is the message severity (an integer),
* field 10 is the message text, which can contain printf style
parameters,
* and field 5 is a list of parameter values. Fields 10 and 5 are multi-
* valued. Field 3 is a timestamp, already in timestamp format.
* In the output (the archive) we record the timestamp, the severity and
* the full text of the message. Timestamp is in columns 1-19,
* severity is
* in column 21, and the message itself begins in column 23. Lines of
* multiple line messages are separated by (A).
* Form a Select List of log records in logged order. Since the log file
has
* a generated sequential numeric key, we can use this.
SSELECT JobLog.Fvar
* Process this list, bypassing any records that do not have a numeric
key
* (these are control records, like //PURGE.PARAMS and //SEQUENCE.NO).
Loop
While ReadNext LogID
If Num(LogID)
Then
Read LogRecord From JobLog.Fvar,LogID
On Error
Message = Fatal error reading record ":LogID:" from job log.
Message = Job name = ":JobName$:", log file is":JobLogName:".
Message = Error code = :Status():.
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
Then
ArchiveLine = LogRecord ; * Timestamp
ArchiveLine := " " : LogRecord : " " ; * severity
code
* [1.0.2]
* FullText = Subr("DSR_MESSAGE", LogID, LogRecord,
Raise(LogRecord))
FullText = Subr("*DataStage*DSR_MESSAGE", LogID, LogRecord, Raise(LogRecord))
* [1.0.1]
* FullText = Ereplace(FullText, @VM, "(A)", -1, -1)
FullText = Ereplace(FullText, @VM, LF, -1, -1)
ArchiveLine := FullText
WriteSeq ArchiveLine To ArchiveLog.Fvar
Else
Message = Unable to append to log file":LogFileName:".
Call DSLogWarn(Message, RoutineName)
ErrorCode = -1
GoTo MainExit
End
End
Else
Message = Selected record ":LogID:" not found in":JobLogName:" file.
Call DSLogWarn(Message, RoutineName)
End
End
Repeat
FileUnlock JobLog.Fvar
MainExit:
* Close any open files
If FileInfo(ArchiveLog.Fvar, FINFO$IS.FILEVAR)
Then
CloseSeq ArchiveLog.Fvar
End
If FileInfo(JobLog.Fvar, FINFO$IS.FILEVAR)
Then
Close JobLog.Fvar
End
RETURN
> ----------
> From: sudhir.mahendru@db.com[SMTP:sudhir.mahendru@db.com]
> Reply To: informix-datastage@oliver.com
> Sent: Thursday, 10 August 2000 11:33
> To: informix-mv@oliver.com
> Cc: informix-datastage@oliver.com
> Subject: Dumping The Log thru - Unix dsjob
>
> Hi
> We are dumping the datastage log by following command from unix.
>
> dsjob -logsum ntpspro MMCustAcctMA01 >> Summary.log
> dsjob -logdetail ntpspro MMCustAcctMA01 event >> Detail.log
>
> This unix command is working fine but the problem is that it is
> creating a lot of uv processes while executing this command. for a
> single command it is creating around 25 - 35 uv process. and some
> times the uv process reach more than 100 ( only by this command).
>
> is there any way that we can limit the number of uv process for this
> command? is there any efficient way to dump the datstage log.
>
>
> thanks and regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
> regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>