Multi-Instance "Missing Parameters"

jonathanhale · Post by **jonathanhale** » Fri Jul 31, 2009 4:15 am

We are encountering random aborts of multi-instance server jobs .

i.e. a different instance of different server jobs aborts with "Missing Parameters"

In this example there 8 concurrently executing instances of the same sequence/jobs, but only 1 instance fails with the "Missing Parameters" message.

It is definitely the case that the parameters are NOT missing. The parameters are passed to the job by the driving sequence. Checking the log for the driving sequence confirms that the server job was called correctly by the sequence, and that the parameters were correctly passed by the sequence.

However, checking the control start entry for the aborted server job shows that the parameters were not received.

Additionally, the entire entry for the failed instance server job in Director occassionally disappears.

I suspect repository corruption (we know that there is a corrupt project in the repository - a project that can't be deleted, and crashes Administrator when trying to access project properties)

Has anyone seen anything like this before?

ArndW · Post by **ArndW** » Fri Jul 31, 2009 4:48 am

I have seen issues with concurrently running multi-instantiated jobs that occur sporadically. The cause was that there were auto-purge settings enabled for the jobs in question. If you do have auto-purge enabled then try turning it off to see if that might be causing your problems.

jonathanhale · Post by **jonathanhale** » Fri Jul 31, 2009 6:59 am

Hi Arnd, many thanks for the hint. Removing the project level auto purge setting has made the problem go away.

Is this a known bug? (by anyone except you) Is there a fix? Does Fix pack 2 for v8 help?

Is there a workaround available? e.g. script/routine for clearing logs "manually"?

I've read the various posts, and seen Kenneth's code for globally changing the autopurge settings, but that doesn't help if we need to keep Autopurge turned off.

ArndW · Post by **ArndW** » Fri Jul 31, 2009 7:08 am

I submitted an eCase for this problem about a year ago but don't have the case number available at the moment. I seem to recall this having come up here at DSXchange a couple of weeks ago and I have a vague feeling that someone mentioned that it might be fixed at 8.1 but "vague" and "feeling" are not enough, right?
I've also found that the autopurge does not correctly remove instance information from the log files, so have ended up writing code which periodically scans all projects and does the same thing as autopurge but fixed the internal data and also (optionally) renumbers the log files. I can't give out the code to anyone but IBM (since they paid for my time while writing it) and I hope that it is no longer necessary at version 8.1 where one can opt to send log information to a database table and could use normal DML commands to prune log information.

ArndW · Post by **ArndW** » Fri Jul 31, 2009 8:09 am

That looks like the 8.1 version where the data is in the XMETA and not in hashed files. I think that it won't work at any version prior to 8.1

chulett · Post by **chulett** » Fri Jul 31, 2009 8:26 am

And this literally goes in and nukes all entries from all logs in all projects on a server, even if jobs are running?

JRodriguez · Post by **JRodriguez** » Fri Jul 31, 2009 10:09 am

This was intended to remove all job logs from the metadata repository due to a bug in version 8.1

At version 8.1 the jobs logs can be saved in the metadata repository or using the old way having them in files. But there is a bug - I did some post about it a while ago - and Big Blue recommended to revert the logging to files and remove all existing job logs in the metadata repository. This piece of code was provide for that purpose[/b]

chulett · Post by **chulett** » Fri Jul 31, 2009 10:54 am

Ah... so this only affects the XMETA repository then. That makes more sense.

jonathanhale · Post by **jonathanhale** » Thu Aug 13, 2009 3:43 am

So for 8.0.1 the only solution would be Arnd's code? @Arnd, do you have access to that Ecase number yet? (so the client can follow-up with IBM)

ArndW · Post by **ArndW** » Thu Aug 13, 2009 7:03 am

I cannot find the eCase, either in the project docs or under my name at support.

vradzije · Post by **vradzije** » Fri Sep 25, 2009 5:46 pm

Caution: issues after applying fixpack 2

Yesterday I applied fix pack 2 to DS v8.0 and now experiencing Multi-Instance "Missing Parameters" issue.
I had 8.0.1 installed for more than year with auto-purge enabled on project level and never seen it before. Submitted SR...

vradzije · Post by **vradzije** » Fri Sep 25, 2009 6:34 pm

Got email from support:

I have found a patch that should resolve the problem. The patch is JR30015 and it has fixed the missing parameter error from a sequence job. It also addresses other issues with the number of job instances and auto-purge leading to this error.

vradzije · Post by **vradzije** » Tue Sep 29, 2009 9:10 am

PROBLEM:

Note: JR30015 is merging with JR30251/30312

If more than 25 instances were running, it was possible for Status records
to be purged when their Job Started ID was removed from the list of started jobs,
resulting in Status=99 errors.

It is also possible for jobs to appear to be started with no parameters on a heavily
loaded system.

The merge with JR30251/30312 fixes the way that Parallel jobs are run.
Parallel jobs failed with code -99 intermittently.

DETAILS:

Incorporates previous Auto-purging fixes, and now increases the number of stored
Job Started records from 25 to 100, which can be changed manually up to 9999 if
the DSJobStartedMax enviroment variable is used.

Status records were being purged for job instances that did not have the appropriate
wave records on heavily loaded systems.

On a heavily loaded system, parallel jobs fail when the osh return code
is retrieved from the temporary file RT_SC<jobno>/<oshpid>.retval.

RESOLUTION:

JR30015 - Increased maximum number of stored Job Started IDs. Increasing this limit
prevents Status records being purged if more than 25 instances are run simultaneously,
which resulted in Status=99 errors, and means that more than 25 instances can now be
seen in the Director Status View.
To be able to increase the Auto-purge instance limit to more than 25 in the Director
or Administrator it will be necessary to install the latest version of the DataStage
Client part of this patch - patch_JR30015<patchversion>_client_windows_<release>.
In extreme cases it may still be necessary to use the DS_NO_INSTANCE_PURGING=1 environment
variable setting if Status=99 errors still occur.

On a heavily loaded system we will not now purge status records if the wave record has
not yet appeared.

JR30251/30312 - The DSD.RUN process will now correctly capture the contents of
RT_SC<jobno>/<oshpid>.retval. Newly generated OshExecuter.sh scripts will contain
code to ensure that RT_SC<jobno>/<oshpid>.retval gets written.
*** Parallel Jobs must be recompiled and redeployed for this fix to take effect ***

priyadarshikunal · Post by **priyadarshikunal** » Tue Sep 29, 2009 9:29 am

Make sure it works after you apply JR30015 patch. because even after applying this patch it never worked for me and hence forcing me to write some house keeping routines.