job aborting after 50 warning

hsahay · Post by **hsahay** » Mon Jun 29, 2015 1:48 pm

Unix script calls DSJOB to invoke a job.
It does not use -warn0

Job generates a lot of warnings but DOES NOT abort.
This is in version 8.1

The same job in version 11.3 aborts after 50 warnings.

Question is Why ?

I have changed the global warning LIMITs through director to unlimited. But the job still fails after 50 warnings

I understand that we should work to get rid of the warnings or use -warn 0 in the unix script, but we are in the middle of a migration from 8.1 to 11.3 and are trying to run everything that used to work in 8.1 without any modification in 11.3

Any ideas ?

hsahay · Post by **hsahay** » Tue Jun 30, 2015 10:52 am

Can somebody at least answer the following questions -

When running a job from director, one can set the limit on warnings after which the job will abort. There is an option to make it unlimited.
Question 1 - Does it apply to only that session or is it global ? Meaning If i set it to unlimited on director client on my machine, will it be unlimited to someone else also ?
Question 2 - When the limit is changed does it apply only to that specific job or to all jobs ?
Question 3 - Is there a global setting that can be used to set number of warning as unlimited across the board ?
Question 4 - Are the limits set via director client applicable even when a job is invoked via command line ? So in other words, if -warn 0 option is not used, what is datastage looking at in order to determine whether to abort the job after 50 warning or not ?

chulett · Post by **chulett** » Tue Jun 30, 2015 1:09 pm

1. Just that session.
2. From what I recall, you can set the Director's default to unlimited but it's still just for you.
3. Not that I am aware of... and I'm firmly in the That Would Be A Bad Idea camp.
4. The Director limits are not applicable at the command line. I don't recall if the default of "50" is hard-coded or set somewhere, been too long... perhaps someone else does.

hsahay · Post by **hsahay** » Tue Jun 30, 2015 2:31 pm

Thanks Chulett

So when a job is run from the command line, what causes it to abort after 50 warnings ?

Where is that setting kept.

The reason I am asking is because in our production server running datastage 8.1, the job does not abort after 50 warnings.

But on the new 11.3 datastage it does.

This is a monthly scheduled job. Every month, IBM's TWS invokes the unix script that has the dsjob command (without the -warn 0 option) to invoke this datastage job.

It runs fine on all our 8.1 environments. Dev/Qa and production despite generating the same amount of warnings that it produces in 11.3.

hsahay · Post by **hsahay** » Wed Jul 01, 2015 8:29 am

This is the response i got from IBM tech support -

I believe you can use DSSetJobLimit to set the default warning limit
across all sessions.
http://www-01.ibm.com/support/knowledge ... om.ibm.swg.
im.iis.ds.cliapi.ref.doc/topics/r_dsvjbref_DSSetJobLimit.html

When the job is run via command line and -warn is not used, the default
warning limit is used.

To which i posted these questions -

The DSSetJobLimit looks promising but i am not clear how to use it.

Do I need to put it somewhere in the failing job ? Which means I need
to change the code.
But if I do that, would the default limit apply to that job only ?

What do i need to do to set the limit to 0 (unlimited across the board)

Many of our jobs, in the absence of -warn 0, are failing, when run
under 11.3, after 50 warnings. Which means the default limit is set to
50 somewhere for all of the jobs.

How do I change it to unlimited without changing our unix script or the
job itself ?

Can somebody answer this please ?

chulett · Post by **chulett** » Wed Jul 01, 2015 8:53 am

So you asked those questions of IBM Support?

That function is the equivalent of the -warn option from the command line, for use by BASIC job control code - i.e. when using DSRunJob(). Which you are not.

hsahay · Post by **hsahay** » Wed Jul 01, 2015 8:56 am

chulett wrote:So you asked those questions of IBM Support?

I did but based upon the quality and timing of response so far, I thought let me ask it here also and perhaps get a better answer quicker ....

Notice, that he uses the phrase "I believe" - which does not engender a lot of faith in the response

chulett · Post by **chulett** » Wed Jul 01, 2015 8:57 am

I edited my post to include more information probably just as you were replying.

rkashyap · Post by **rkashyap** » Wed Jul 01, 2015 9:03 am

Have you looked into message handler?

chulett · Post by **chulett** » Wed Jul 01, 2015 9:09 am

Or reducing the number of warnings the jobs produce?

qt_ky · Post by **qt_ky** » Wed Jul 01, 2015 9:09 am

See also:

viewtopic.php?t=142670

hsahay · Post by **hsahay** » Wed Jul 01, 2015 9:17 am

Rakesh - Yes I have.

However, On my 8.1 servers, DEV/QA and PROD, this same job, after being invoked from a unix script without the benefit of -warn 0, runs to completion despite all the warnings. These jobs have been running fine on these servers for years and there is no message handler or anything there.

But when run under 11.3 they consistently fail after 50 warnings.

So all i want to know is why they are not failing in 8.1 and can i replicate the same behavior under 11.3 without changing my unix scripts so that i can complete the migration from 8.1 to 11.3

We are out of support and are paying 8 grand to IBM every month for tech support of an out of support product (8.1)

Chulette - As soon as this migration is done, I will ask all my developers to look into all these warnings and fix them. But the sheer number of such jobs we have makes it impossible to embark upon that activity while we are in the middle of a migration.

qt_ky · Post by **qt_ky** » Wed Jul 01, 2015 9:53 am

It sounds like you are paying for tech support on both old and new versions, so I think you are best off to demand an answer from Support. I suspect they will tell you that you will have to update your scripts on 11.x to use the -warn 0 option. The behavior described on 8.1 is surprising, unless perhaps it was set to default to no warning limit in that release vs. the usual default of 50.

chulett · Post by **chulett** » Wed Jul 01, 2015 10:11 am

I think this is just one more example in a long line of examples of bugs / issues in the old versions that have since been corrected in newer releases. And you are in a long line of people who were (unknowingly) reliant on those bugs.

hsahay · Post by **hsahay** » Wed Jul 01, 2015 10:25 am

Found this article on the net -

http://www-01.ibm.com/support/docview.w ... wg21611065

As per this, the escalation of 50 warnings to a fatal error may not happen before the job finishes due to the asynchronous nature of osh events.

In such a case the job will finish with warnings.

May be this is what is happening in 8.1 and perhaps in 11.3 they have changed the way events are handled ?

Regardless - unless the default of 50 is hard coded deep inside the datastage code and is unreachable to us somebody should be able to tell me where to find it and how to change it. sigh !!!

I will also try the DS_FORCE_ABORT_AT_WARN_LIMIT=0 in admin environment variables and see if that makes a difference.