Purging of DataStage Log from Unix Script
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 31
- Joined: Wed Feb 04, 2004 2:04 pm
- Location: Canada
Purging of DataStage Log from Unix Script
We run all our DataStage jobs using dsjob command from Unix scripts. Few days ago, we made the project wide change using DS Administrator to auto purge the log upto previous 2 runs. I understand that this default 'auto purge' will only apply to the new jobs created after this change.
We have some old jobs and would like to clear the log automatically upto last 2 runs. Can we use any dsjob command in Unix script to purge the log before we run the job? or if we recompile the old programs then will they be purged automatically upto last 2 runs as we have a project wide 'Auto Purge' default?
Thanks
We have some old jobs and would like to clear the log automatically upto last 2 runs. Can we use any dsjob command in Unix script to purge the log before we run the job? or if we recompile the old programs then will they be purged automatically upto last 2 runs as we have a project wide 'Auto Purge' default?
Thanks
Harsh Kohli
Nope, sorry. The OP has it correct, this setting only applies to new jobs added to the Project.
You'd need to understand the underlying structures and write some code to update the Auto Purge information in old jobs. Luckily, I believe you'll find that someone out there has already written a little Batch job control type program to do just that. We'll just have to see if they are willing to share the fruits of their labors.
You'd need to understand the underlying structures and write some code to update the Auto Purge information in old jobs. Luckily, I believe you'll find that someone out there has already written a little Batch job control type program to do just that. We'll just have to see if they are willing to share the fruits of their labors.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
I have a utility available for mass-updating the auto-purge setting. This works for DS 5 and 6, don't know yet about 7.
USE AT YOUR OWN RISK. The author makes no guarantee to results, and make sure you have your project backed up whenever attempting internal hacking.
Paste this into a BATCH job, and setup four parameters:
Folder -- Optional - Category\Subcategory...
JobNameSubstringMatch -- Optional - FullJobName OR ...xxxx...
PurgeType -- DAY=purge by day interval, RUN=purge by number of runs
PurgeInterval -- interval setting, integer value
USE AT YOUR OWN RISK. The author makes no guarantee to results, and make sure you have your project backed up whenever attempting internal hacking.
Paste this into a BATCH job, and setup four parameters:
Folder -- Optional - Category\Subcategory...
JobNameSubstringMatch -- Optional - FullJobName OR ...xxxx...
PurgeType -- DAY=purge by day interval, RUN=purge by number of runs
PurgeInterval -- interval setting, integer value
Code: Select all
TCL = 'SSELECT DS_JOBS'
If JobNameSubstringMatch Then TCL:= ' LIKE "...':JobNameSubstringMatch:'..."'
If Folder Then TCL:= ' WITH F3 = "':Folder:'"'
Call DSExecute("TCL", TCL, ScreenOutput, SystemReturnCode)
Call DSLogInfo("TCL Output for: ":TCL:" >> ":ScreenOutput, "Msg")
OPEN "DS_JOBS" TO JobsXrefTable Else
Call DSLogFatal("Unable to open DS_JOBS", "Msg")
End
JobList = ""
AllJobsDone = @FALSE
Loop
Readnext JobName Else AllJobsDone = @TRUE
Until AllJobsDone Do
If LEFT(JobName,2) # "\\" Then
Read Row From JobsXrefTable, JobName Then
JobList<-1> = JobName:"|":Row<5>
End
End
Repeat
Call DSLogInfo("Processing the following jobs: ":JobList, "Msg")
JobCount = Dcount(JobList,@AM)
For Job = 1 TO JobCount
JobName = FIELD(JobList<Job>, "|",1)
TableName = "RT_LOG":FIELD(JobList<Job>, "|",2)
OPEN TableName TO F.FILE Then
Read PurgeSettings FROM F.FILE, "//PURGE.SETTINGS" Else PurgeSettings = ""
If PurgeType = "DAY" Then
PurgeSettings<1> = 1
PurgeSettings<2> = PurgeInterval
PurgeSettings<3> = 0
End
If PurgeType = "RUN" Then
PurgeSettings<1> = 2
PurgeSettings<2> = 0
PurgeSettings<3> = PurgeInterval
End
WRITE PurgeSettings ON F.FILE, "//PURGE.SETTINGS"
End
Next Job
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Project-wide purge parameters are stored in the DSParams file. There is a section that looks like this:
A routine to update all jobs with these defaults should read these settings, then update the control record //PURGE.SETTINGS in the RT_LOGnnn file for each job. To determine the value of nnn, map from the DS_LOGS table.
Ken's code could be improved in a couple of ways. Note that the following is not exactly the same solution; it is an "all or nothing" approach, except as noted in code comments. Most error checking has been omitted for clarity.
The same caveats apply to this code as to Ken's.
Code: Select all
[AUTO-PURGE]
PurgeEnabled=0
DaysOld=0
PrevRuns=0
Ken's code could be improved in a couple of ways. Note that the following is not exactly the same solution; it is an "all or nothing" approach, except as noted in code comments. Most error checking has been omitted for clarity.
Code: Select all
PurgeEnabled = 0
DaysOld = 0
PrevRuns = 0
OpenSeq "DSParams" To ParamFileVar
Then
Loop
While ReadSeq Line From ParamFileVar
If Trim(Line) = "[AUTO-PURGE]"
Then
ReadSeq Line From ParamFileVar Then PurgeEnabled = Field(Line, "=", 2)
ReadSeq Line From ParamFileVar Then DaysOld = Field(Line, "=", 2)
ReadSeq Line From ParamFileVar Then PrevRuns = Field(Line, "=", 2)
Exit
End
Repeat
CloseSeq ParamFileVar
End
ControlKey = "//PURGE.SETTINGS"
ControlRec = PurgeEnabled : @FM : DaysOld : @FM : PrevRuns
Open "DS_JOBS" To DSJobsVar
Then
Select DSJobsVar To 9
Loop
While ReadNext JobName From 9
If JobName Matches "1A0X"
Then
ReadV JobNumber From DSJobsVar, JobName, 5
Then
LogFileName = Convert(" ", "", "RT_LOG" : JobNumber)
Open LogFileName To LogFileVar
Then
ReadU PurgeSettings From LogFileVar, ControlKey
Locked
* Control record in use by another process - bypass
Msg = "Cannot access purge settings for " : Quote(JobName)
Call DSLogWarn(Msg, "ResetPurgeParams")
End
Then
* Disable this statement if you do not want to override
* jobs that already have purge settings
Write ControlRec To LogFileVar, ControlKey
End
Else
Write ControlRec To LogFileVar, ControlKey
End
Close LogFileVar
End
End
End
Repeat
End
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 31
- Joined: Wed Feb 04, 2004 2:04 pm
- Location: Canada
Thank you all for your replies and sharing the code.
To my original question, just to recap -
1. There is no dsjob command in DataStage for purging the log.
2. Project wide Auto-purge will only take affect for the newer jobs created (not compiled) after the project wide auto-purge is set.
Thanks.
Harsh Kohli
To my original question, just to recap -
1. There is no dsjob command in DataStage for purging the log.
2. Project wide Auto-purge will only take affect for the newer jobs created (not compiled) after the project wide auto-purge is set.
Thanks.
Harsh Kohli
Harsh Kohli
If you take Ray's routine and change the lines from:
to:
then it will clear the file.
Code: Select all
Open LogFileName To LogFileVar
then
...
end
Code: Select all
execute "CLEAR.FILE ":LogFileName capturing output
Mamu Kim
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
... including the control records!
A better command to execute (though a little slower) is:
A better command to execute (though a little slower) is:
Code: Select all
"DELETE FROM " : LogFileName : " WHERE CAST(@ID AS VARCHAR) NOT LIKE '//%';"
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 2
- Joined: Wed May 05, 2004 8:48 pm
Could this code be run in DataStage 4?
We have a situation where all the jobs in our DataStage 4.1 system have their AutoPurge option set correctly but, from time to time, this setting becomes un-set.
Would the code listed above work in a 4.1 system? If not, is there another method I could use?
Would the code listed above work in a 4.1 system? If not, is there another method I could use?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 385
- Joined: Tue Oct 07, 2003 4:55 am
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: