DataStage Job Automated Monitoring

[fober]

I am new to DataStage and I am looking for some pointers into what design options I may have in instrumenting a DataStage job for automated monitoring. Automated monitoring means we are not monitoring the DS Director GUI, but rather the job itself is instrumented (coded) to alert via an SNMP message out to a product such as Tivoli or NetIQ/MOM.

Some ideas I have (please shoot them down or correct me at will.)
Is there a command set that the DS BASIC language provides that would allow me to code in an SNMP event message if a certain condition or WARNING/FAILURE were to be raised in a job?
If not my belief is that I would be basically stuck to use COMMAND Stage implants into my JOB and JOB SEQUENCE designs that would provide condition "triggers" to the Job objects that would go off and run an snmp event generator program via the command line that we have written in-house. In fact that paradigm is in-use today within my Company (for other products), but I would prefer to move to something more integral to DataStage or something more elegant.

Could I use the WS Message Adapter in some way? What framework for SNMP event message passing does Ascential provide with their DataStage tool.

I have searched the entire dstage7.pdx (*cross index of the Ascential docs) and have found no hits on SNMP so I am a little concerned that a homegrown and kludgier method on my part may be needed.

Thanks for any insights in advance.
Frank Ober

[ray.wurlod]

Quick response.

Search for SMTP rather than SNMP.

DataStage has its own inbuilt function DSSendMail, which is an interlude to sendmail() on UNIX or any other mail program. On Windows, a lot of people have had good experiences with blat. Again, search the forum for "email" and for "blat".

It's straightforward to configure and strange, though not difficult, to use, and there are example routines supplied (for example a test routine for DSSendMail).

In job sequences you can use a Notification Activity which, under the covers, is a call to DSSendMail.

HTH

[ogmios]

For what you want DataStage is pretty much "roll your own". No SNMP interface currently exists, but it may be a "shiny object" (read: for the leaflets) implemented in the next version if you friendly asked it to Ascential (it's late, I'm getting cynical )

Even if you built it in in your jobs using command stages you're never sure that a DataStage job will actually execute even when scheduled properly, sometimes DataStage has its little quirks.
This sometimes happens when your machine is very overloaded, you've reached the T30FILES limit of DataStage or your server has been running for a few months without cleaning up your projects.

And then you're off monitoring jobs with dsjob which also sometimes fails, ...

Ogmios

[mhester]

I would have to agree with ogmios in that you may have to roll your own since there are not plugins for this type of interface. Using SMTP will not do what you want since you really want to interface at the network level and not via email. Since you are talking Tivoli you are talking about an entire strategy built around things like SNMP.

If your product can start network or server processes than you can create a wrapper to start DS jobs and return status back and do something based on this status.

Regards,

Michael Hester

[lpadrta]

Hello,

I am using two types of automated monitoring, and they involve using a combination of the tools in DataStage. Not sure if they will help you but they might.

First, I put together groups of individual jobs using the Batch Job Control code which end with a status code based on results of the individual jobs (i.e., when a job fails, so the batch fails). I wrapped the batch inside a sequencer with notification of the results via email. This was pretty simple.

Second, at the end of each successful batch, I write a single line small file with a filename that indicates what it is for, then I have a separate "watchdog" job that checks to see if the file of the subject batch is present (i.e., the batch was finished OK). I have each "watchdog" job wrapped in a sequencer with notification of the results via email. The "watchdog" job in the sequencer is scheduled to run when the batch is expected to be completed.

There are probably a million other ways to do this, maybe even some simpler , but this seems to be manageable in my circumstance.

Hope this is the least bit helpful.
Lynda

[kduke]

Kenneth Bland posted a real nice script which uses the return value to see if the job finished okay. Do a search. It is probably a more elegant solution. This is the way to use it with control m and other products which can automate processes across platforms. Many of us have had to integrate DataStage with these products. If you need a more specific answer then ask a more specific question. I am sure someone will have had experience with your problem.