DataStage corruption

[eldonp]

A few months ago, when we were running DataStage 7.1, we experienced some job corruption. We established, with the help of IBM support, that there was file corruption occurring in the engine. We addressed the symptoms by re-creating the affected projects or re-installing DataStage.

About 2 months ago, we upgraded to (by performing a clean install and importing job design exports) 7.5 and are still having similar problems. This morning, for example, I get the following warning in Director.

Cannot find job XXXXX

This warning occurs for each job, then none of the jobs show up in Director.

In Manager, the jobs show up, but when I try to edit any job, the following warning appears.

Job XXXXXX does not exist in this project

Upon playing around in Manager, I got the following warnings.

Record XXXXXX in file DS_JOBS does not exist
Cannot find job AAAAAA
Cannot find job FFFFFFF
Cannot find job HHHHHH

We are quite exasperated at this stage. Clearly, the IBM recommendation to upgrade has not given the desired result. We're looking for some clear guidelines/instructions to trace the root cause, and how the problems can be permanently eliminated.

In addition, I have performed maintenance on the project by recreating the repository indexes. Now all the jobs are gone!

[chulett]

In addition, I have performed maintenance on the project by recreating the repository indexes. Now all the jobs are gone!

Then you failed to perform this 'maintenance' in an isolated fashion. If anyone or anything is connected to the Project in question, that is the guaranteed end result, which pretty much any conversation here that mentions DS.REINDEX will also mention.

Try again. Arrange for some alone time. I'd suggest stopping DataStage, waiting for all ports in use to release and then restart it. Then log back on and try reindexing again. Everything should be back and (hopefully) accessible again.

These kind of problems are typically a result of space management issues. Are you using Pathed hashed files or are all being stored in an 'account' - i.e. each Project? Are you checking each Project for stray files that may be wasting space there? This happens when developers don't fully path files, for example. All of this leads up to the $64,000 question - have you had space problems with the partition where DataStage is installed? Filling that partition is a guaranteed Agent of Corruption.

[eldonp]

Thanks Craig,

There are no space problems. We have ensured that each project's files are created on seperate disks - away from the Projects and engine. Space is definately no issue.

Resarting DataStage has (even re-installing DataStage) treats the symptom for a few days, but then we're back to the same problem.

We use standard dynamic hashed files - hardly have files that grow larger than 2GB.

[chulett]

Hmmm... this is hardly 'normal' behaviour. If you've continued to have issues across major versions like that, seems to me there's either an environment issue or something (not sure what, however) in how it is being managed.

Never ran DataStage on an AIX box so don't know how much of a factor that may or may not be. Seem to recall issues with limits on the number of files in a directory, do you have a large amount of jobs per projects, or perhaps of projects themselves? Are you having issues running jobs, getting odd messages? Or is everything 'fine' and then suddenly bang the issue crops up? Have you raised or changed any of the uvconfig values or is everything 'out of the box'?

Something is corrupting your Project repository. This would typically be a resource issue, if not space then configuration it seems to me. Regardless, best to get back in touch with Support if you haven't done so already and have them working with you on this issue.

If anyone else has some thoughts on this, don't be shy - please come on down and play along with us. Especially those of you on AIX.

[ArndW]

eldonp - At my last site, a large AIX implementation, I noticed that several times on some of the engine that dynamic files were corrupted, upon further inspection it seems that the DATA.30 files were *poof* gone. Over the course of months this happened several times and I was never able to track down the possible causes and we all assumed that external program was causing it. Do you know if this might have happened at your site?

[asorrell]

Another possible cause - I've seen sites where people have jobs open and/or running and someone decides to do a hard kill (kill -9) from the UNIX level to stop the job.

This is NEVER recommended by the way, mainly because it can also cause various kinds of corruption.

[ray.wurlod]

Please let us know when you 've performed DS.REINDEX ALL having exclusive access to the project, and the results. If there are still issues we can advise further, but prefer not to guess what advice to give.

[aakashahuja]

We are facing similar issues with DS 7.5 on AIX. We reindexed the projects but still get the same errors.

Any further guidance?

Cheers
Aakash

[chulett]

Aakash, you need to start a new thread and explain your problems in detail. Saying you are 'facing similar issues' doesn't really tell us anything.