DSXchange

Hi all,

We're having a strange issue on one of our environments, where jobs stop running without code or data changes.

We've traced it back to environment variables: dsadmin -listenv <project_name> returns nothing.
Things will be working fine including dsadmin -listenv, then at some point the issue will occur.
Nothing else about the server appears strange, and the setup is exactly the same as the other environments as far as we can see.

The current "solution" is to restart the machine. Restarting InfoSphere doesn't have any effect. Obviously it's not much of a solution.

Anyone come across this before?


Cheers,
Stuart.

Is the dsadmin command causing everything to stop? How long have you waited for the command to return something? Does it ever time out or give an error?

When the jobs stop, what is the job status? What do you find in the job logs?

Is there anything significant in the WAS log files SystemOut.log or SystemErr.log around the time of when jobs stop?

Hi Eric,

The command doesn't seem to cause the stoppages.
The first symptom is the overnights fail with a number of errors about not being able to find various things. Then we type dsadmin -listenv <project_name> to verify and it comes back with nothing.

We try dsadmin -listenv <project_name> on the other environment and it comes back in a few seconds with the environment variables.
We try it on the server in question and it returns straight away with nothing.
Following that up with an echo $? returns 141.

I don't find the dsadmin return code of 141 documented anywhere.

I guess you may be chasing a symptom rather than the root cause.

Check the WAS log files SystemOut.log and SystemErr.log for clues. Post anything that may look suspicious around the time of the problems.

Also try contacting product support. Good luck.

Is the code 141 or -141?

For -141, the reason could be "Failed to log to UV account"

And, if that's the case, it might be that you mis-spelled or mis-cased the project name. Project names are case-sensitive.

stuartjvnorton wrote:The first symptom is the overnights fail with a number of errors about not being able to find various things.

Expand upon this topic, since this is your true issue.

The dsadmin -listenv <project_name> returning nothing is the core issue, from what we can see.

The project is a series of DS jobs and routines to deploy other DS projects, and it's highly configurable ie: it builds a lot of paths and settings itself, based off current environment variable values and what is parameters provided at run-time.
Right after the initial target project creation it calls dsadmin -listenv <project_name> and puts them in a cache. It's this process that returns nothing.
The other errors about not being able to find various files, etc is because this cache file is empty.

Ironically, the log for this job manages to do its standard info message dump of the environment variable settings, then 5 lines down fails where a routine tries to do the dsadmin call.

As I said previously, the dsadmin call works consistently until some point where it begins to fail (regardless of how you call it). Once it fails the first time, it will consistently fail until the server is rebooted. Restarting InfoSphere does not solve the issue.

stuartjvnorton wrote:Once it fails the first time, it will consistently fail until the server is rebooted.

I can think of no other advice than to contact your official support provider, and I would guess that you have already done so.