RT_CONFIGxx error

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 9:40 am

Hi all,

Got a weird one. One of our servers has had something change over the weekend, still trying to track it down. This caused all our jobs to abort with no reason listed. Doing some checking, I found:

1)No access to RT_CONFIGxx. I can't compile jobs, edit jobs or run jobs. Directories are still in place and permissions to the files for the user are fine. User is still part of the group who owns the files.

2)Links have been set up in place of some directories. This caused an error a few months ago when datastage/ was moved to a link. This time, datastage/ is fine but the following are not:

lrwxrwxrwx .dslibs -> /datastage/DataStage/branded_odbc/lib

Other than that things look fine.

kcbland · Post by **kcbland** » Tue Mar 15, 2005 9:42 am

Did your project run out of disk space on the filesystem?

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Mar 15, 2005 9:46 am

Can you log into DataStage?

Were you removed from the dstage group for any reason?

Can you view the log from DataStage Director?

Can you take an export of your project?

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 10:13 am

kcbland wrote:Did your project run out of disk space on the filesystem?

Maybe. I removed some core files and am able to recompile the jobs (except the one I got the error on earlier). I went through and removed all core files and recompiled all the jobs I can. I still have some that will not allow me to recompile them (cannot open executable job file RT_CONFIGxx).

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 10:15 am

Sainath.Srinivasan wrote:Can you log into DataStage?

Were you removed from the dstage group for any reason?

Can you view the log from DataStage Director?

Can you take an export of your project?

Yes, I can log in.
No, I am still in the group.
Yes, I can see the log.
I cannot export the errored jobs. I cannot import a backup over top of the errored job.

The jobs I get the error on are not visible in Director but I can see them in Manager.

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 10:22 am

Something else odd. I was able to rename the errored job in manager (I cannot delete it in Manager or access it in Director). I was then able to move the backup job into the project.

The new jobs has a description of "!!Error getting job status!!' I compiled it, and the Description changed back to the normal description we have written. Log out and log back in and the jobs says description of "!!Error getting job status!!'

kcbland · Post by **kcbland** » Tue Mar 15, 2005 10:26 am

Of all of the bad things that can happen, running a project out of disk space IS THE WORST POSSIBLE THING.

Jobs executing can corrupt their logs, configuration, and status files because of insufficient disk space. If ONE SINGLE DEVELOPER has a job open for design and is saving the job, it can corrupt THE ONE FILE THAT CONTAINS ALL JOB DESIGN INFORMATION.

You must be diligent in your monitoring of project disk space. As you are probably starting to realize, the cleanest solution in front of you is to create a new project and import your jobs into there. This is why you should store your jobs in a tool like PVCS, and why you never build ETL solutions that are tightly embedded with the project (localized hash files versus externally pathed on separate file systems).

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 10:32 am

Yes, I agree. We have NEVER had this problem. I do keep track of disk space (plus we are a large company who pay other groups to do that). These projects have been running since 2002 with no problem...so this is new.

I think you're right though, the cleanest thing to do is to remove the project and import the backup. I'm having issues with some of the projects. "ACCESS DENIED. Cannot get exclusive access to the project. Ensure there are no other DataStage users attached."

I am the only user. ps -fu on the box shows no other instances of DS running.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Mar 15, 2005 10:36 am

From Unix level, try a 'file RT_CONFIGxxx' where xxx is the file number. If it returns a permission error, then it may be a file problem.

As Ken suggested, check file file system and spaces. I do not think that will be a problem as otherwise all your jobs will be affected.

Try to login as root or dsadm and attempt to open the files.

If you want to go further, try giving a 777 permission to all files and folders under the DSEngine and Project.

Can you log into DataStage Administrator? If yes, cleanup will reindex all your pointers. But that will be the last option before reimporting old version.

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 10:38 am

Sainath.Srinivasan wrote:From Unix level, try a 'file RT_CONFIGxxx' where xxx is the file number. If it returns a permission error, then it may be a file problem.

As Ken suggested, check file file system and spaces. I do not think that will be a problem as otherwise all your jobs will be affected.

Try to login as root or dsadm and attempt to open the files.

If you want to go further, try giving a 777 permission to all files and folders under the DSEngine and Project.

Can you log into DataStage Administrator? If yes, cleanup will reindex all your pointers. But that will be the last option before reimporting old version.

Cleanup didn't fix it. Searching for RT_CONFIG256 (one of the errored ones in a project) showed that it did not exist. So, something happened and DS didn't write the RT_CONFIG file (probably while space was low). DS still thinks RT_CONFIG256 holds the info but the RT file doesn't exist.

I stopped/restarted DS and got access to the project. Blew it away and moved the backup in. Will rerun and see if all is well.

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 10:54 am

One quick question:

Is there a list anywhere on the board of files/directories to be routinely cleaned up? DS support mentioned core files and some &TEMP& directory (may have typed that wrong, but it had two &). I'm going to put something together to go out and remove them over the weekend.

kcbland · Post by **kcbland** » Tue Mar 15, 2005 11:17 am

Sainath.Srinivasan wrote: If you want to go further, try giving a 777 permission to all files and folders under the DSEngine and Project.

DON'T EVER DO THIS, YOU WILL CRIPPLE YOUR INSTALLATION OF DATASTAGE.

kcbland · Post by **kcbland** » Tue Mar 15, 2005 11:19 am

Triton46 wrote:Is there a list anywhere on the board of files/directories to be routinely cleaned up?

In your uvconfig file there is a temp directory configuration. There's also the "&PH&" directory in every project. During an operational downtime, maybe once a week, clear these directories.

Triton46 · Post by **Triton46** » Tue Mar 15, 2005 11:25 am

kcbland wrote:
Triton46 wrote:Is there a list anywhere on the board of files/directories to be routinely cleaned up?
In your uvconfig file there is a temp directory configuration. There's also the "&PH&" directory in every project. During an operational downtime, maybe once a week, clear these directories.

That was it &PH&. Thanks

What about DS_TEMPxx?

kcbland · Post by **kcbland** » Tue Mar 15, 2005 11:34 am

Triton46 wrote: What about DS_TEMPxx?

Every job has one of these. Don't worry about them.