Multi instance job logs

chrisk · Post by **chrisk** » Thu Oct 21, 2004 5:13 pm

Hi,

I have a multi instance job that is executed from a control job. The multi instance job is executed 24 times and is passed the hour of the day as a parameter. The instance name is the jobname concatenated with the date and hour. A DS log is created for each instance executed. This results in 24 new log files being created each day. Eg.

...
JobName.20041022_10
JobName.20041022_11
JobName.20041022_12
...

I can't see any option in DS to automatically remove these logs. Note that I'm not trying to "purge" the logs but to have the last n days worth of log files deleted without manual intervention.

Is there an option in DS to achieve this without coding?

Thanks,
Chris.

chulett · Post by **chulett** » Thu Oct 21, 2004 5:21 pm

chrisk wrote:I can't see any option in DS to automatically remove these logs. Note that I'm not trying to "purge" the logs but to have the last n days worth of log files deleted without manual intervention.

Well, that is a purge... it's just not an Immediate / Clear All Entries type of purge.

You can set the job logs to AutoPurge with two choices below that: Keep up to X job runs or keep up to X days in the log.

In all honesty, I'm not sure how a job being a Multiple Instance job affects that... but I don't think it does.

Have you tried setting the Auto Purge option for the jobs in question?

chrisk · Post by **chrisk** » Thu Oct 21, 2004 5:40 pm

Thanks for your response.

Yes, I have tried setting the auto purge options for "JobName". The auto purge option for my job is "Up to previous" 5 runs.

However, since each instance name is unique, each instance will only ever have 1 runs worth of log info. Even if it did get to 5, the contents of the log would be purged but not the log itself.

In Director, I have entries in the Status view such as:

JobName
JobName.20041022_01
JobName.20041022_02
...
JobName.20041022_10
JobName.20041022_11
...

The log for JobName itself only ever contains 5 "runs" as this is the job for which the auto purge option is set. It is not possible to set the purge options for the instances for obvious reasons.

So my goal is to automatically delete the instances of JobName.* from Director.

Ta,
C.

chrisk · Post by **chrisk** » Thu Oct 21, 2004 6:14 pm

Some additional info from DS Director help:

Note that, if you are running multiple invocations of the same job, all invocations share the same job log. This means that if you, for example, choose to purge entries for all but the last four runs, and the job is invoked 6 times, then the entries for only 4 of the invocations will be kept. When using invocations to run jobs it may be better to auto-purge based on the number of days entries to be kept.

This info is correct. I.e. only the last 5 invocations of JobName have entries in the log. However, the problem still remains that I still have 24 new entries per day created in my Status view in Director. Big deal? The downside is that is takes 15-30 seconds for my Status view to refresh in Director.

Any clues?

Ta,
C.

kcbland · Post by **kcbland** » Thu Oct 21, 2004 7:38 pm

After each instance run extract the log rows for that instance run and tuck it away in your own repository table. Basically "roll your own". Set the purge to be daily and ignore DataStage's log completely.

chrisk · Post by **chrisk** » Thu Oct 21, 2004 7:50 pm

Ken,

Are you implying that DS does NOT have the smarts inbuilt to remove these multi instance entries from the Status view in Director?

I'd like to use an inbuilt mechanism if possible rather than going down the "roll my own" path.

Ta,
C.

kcbland · Post by **kcbland** » Thu Oct 21, 2004 8:24 pm

When you use the APIs to access the log of an instance, it returns all messages across all instances of the root. You will have to discriminate the rows that pertain to the instance you have interest in.

As far as Director is concerned, the only way to filter log messages on an instance is to change to the instance log in the Director. When viewing the root job log, you see everything across all instances commingled.

This is why you may consider writing a subroutine to extract and spool all log messages for a job to a file, and consider putting that as a standard after-job routine call, or use custom job control that "exports" job log information post-run of a job.

kduke · Post by **kduke** » Thu Oct 21, 2004 8:30 pm

I agree with Ken. This would not be a hard routine to code.

I think we may need a generic job to archive logs. If someone has done this then they should post their code or jobs. This question gets asked way too often.

chulett · Post by **chulett** » Thu Oct 21, 2004 9:35 pm

chrisk wrote:So my goal is to automatically delete the instances of JobName.* from Director.

chrisk also wrote:Are you implying that DS does NOT have the smarts inbuilt to remove these multi instance entries from the Status view in Director?

Wait... so you are not so much worried about the contents of the logs? You are trying to get the Status View in the Director for this job back down to a single entry from the 25 you have at the end of the day? In essence, remove the instantiated entries from that particular view? I guess this is what you were referring to when you said you wanted to 'purge the log itself'.

Recompile the job. This can be done via code and scheduled, if you like. How often were you wanting to do this? After each run? Once per day?

Guys - is there some other way to do this other than recompiling? DSJ.RUNRESET with DSRunJob?

kcbland · Post by **kcbland** » Thu Oct 21, 2004 9:58 pm

Sorry, I've confused myself.

You're talking about getting rid of job invocations, not log entries.

How to get rid of job invocations that have a date as the invocation id, meaning an infinite number of invocations. Craig's suggestion is to recompile the base job, which removes all invocations. You need to be able to delete older invocations (instances).

If you are interested in keeping the job log information around, do as I suggest and extract it after a run. I've enhanced my own job control library to have switches for recompiling the jobstream at startup (thus removing all invocations/instances) or recompiling a job prior to running it (achieving the same result). I also trap out the logs for each job to a log file in the OS.

There is no feature to remove instances other than recompiling or deleting them from Director (notice Manager doesn't show instances). There is no API for removing an instance that I know of up to release 7.1. Those with 7.5 need to speak up.

chrisk · Post by **chrisk** » Thu Oct 21, 2004 10:02 pm

Yes Craig, you have understood me correctly. I was starting to feel like I was speaking an alien language for a while there

However, the problem still remains that I still have 24 new entries per day created in my Status view in Director. Big deal? The downside is that is takes 15-30 seconds for my Status view to refresh in Director.

I'm looking for a method that is part of the tool if possible in the same way that the auto purge facility has been created in order to *empty* logs. I.e. I'm rather surprised that this requirement hasn't been catered for in the tool already?

Anyway, any tricks to get around this deficiency are appreciated.

Ta,
C.

chrisk · Post by **chrisk** » Thu Oct 21, 2004 10:05 pm

Thanks Kim, Ken and Craig.

I've logged a call with Ascential to see if there's an *official* method. I'll post the outcome.

Ta,
C.

kcbland · Post by **kcbland** » Thu Oct 21, 2004 10:05 pm

Why use date? Could you use a fixed range of numbers, thereby reusing the instance? Since you're using hour, could jobname.hournumber suffice? You'll have 24 instances fixed in the status view, and can utilize the purging capabilities.

chrisk · Post by **chrisk** » Thu Oct 21, 2004 10:15 pm

Nice suggestion Ken and one I've considered. My problem is that my control job will usually run the child job 24 times (1 days worth) but there are times when it has to *catch up* and run multiple days worth.

At face value this is no big deal, but we have our own custom job restart mechanism (I won't bore you with the details) which requires a unique instance name which it borrows from the job instance name.

Ta,
C.

chrisk · Post by **chrisk** » Fri Oct 22, 2004 1:26 am

The response from my support call to Ascential is that there are 3 manual methods to delete invocation entries from the Status View in Director:

1. Manual Delete
2. Clear Status File
3. Re Compile

I've logged an enhancement request for an "Auto-delete" dialog similar to the "Auto-purge" dialog.

Ta,
C.

DSXchange

Multi instance job logs

Multi instance job logs

Re: Multi instance job logs