issues due to datastage upgrade

ramkarthik · Post by **ramkarthik** » Mon Feb 01, 2010 2:03 am

HI,

We just upgraded datastage 7.5.2 to 7.5.3.I have some jobs that are running for more than two days even though we are processing quite small no of records.We have adequate space in the server too.

Thanks in advance

ArndW · Post by **ArndW** » Mon Feb 01, 2010 4:34 am

You will need to supply some more information in order for anyone to make an analysis. Are the jobs really running or do they just display as running in the director? What sources and targets do these jobs use and can you see if sessions to the appropriate database are actually still open? These two questions are just the beginning but should point towards a possible cause.

ramkarthik · Post by **ramkarthik** » Mon Feb 01, 2010 4:49 am

all the jobs that are being shown status "running" in director, have database sources.I am able to view the data in that stage.....connections are open... but while running from director they stuck up...

ramkarthik · Post by **ramkarthik** » Mon Feb 01, 2010 4:49 am

all the jobs that are being shown status "running" in director, have database sources.I am able to view the data in that stage.....connections are open... but while running from director they stuck up...

ramkarthik · Post by **ramkarthik** » Mon Feb 01, 2010 4:50 am

all the jobs that are being shown status "running" in director, have database sources.I am able to view the data in that stage.....connections are open... but while running from director they stuck up...

ArndW · Post by **ArndW** » Mon Feb 01, 2010 8:34 am

How did you determine that the connections are open? Have you identified the UNIX pids associated with your running job(s)? Once you do that, you can use the "truss" command to see what they are actually doing. Are they using CPU or IO time?

chulett · Post by **chulett** » Mon Feb 01, 2010 8:45 am

Please try to do a better job of identifying what the 'bottleneck' jobs have in common to see where exactly they might be 'hanging up'. Are they all doing database work? If so, what kind? Again if so, check your 'explain plans' (or equivalent) to make sure they haven't changed in any significant fashion. That should also help verify that they are indeed still running and have database connections open rather than not running but still showing a status of Running in the Director.

Have you done a "ps -ef | grep phantom" from the command line to see if indeed there are still processes for your jobs?

munch9 · Post by **munch9** » Mon Feb 01, 2010 10:35 am

Hi,

If your jobs are running on a single node this may be of use.

When we upgraded from V7.5.1a to 7.5.3 we experienced issues with a couple of jobs which showed as running but appeared to hang until we manually stopped them.

IBM identified the issue as being due to the fact the project was running on one node and there had been some changes made to optimize one node performance.

We were advised to specify a user defined variable APT_NO_ONE_NODE_COMBINING_OPTIMIZATION with the value 1.

They also suggested setting APT_JOIN_COMBINATION=yes although this was not essential to fix the problem.

If i remember correctly, disabling operator combination also fixed the problem although the IBM recommendation is probably a better fix.

Hope this helps.

udayk_2007 · Post by **udayk_2007** » Mon Feb 01, 2010 11:48 am

Hello

We also encountered similar issues when we upgraded from 7.5.1 to 7.5.3.
some clues which we came to know while debuging were
1.While doing the update on the table, Datastage job was not using the table indexes implemented and performing full scan
2. Less number of db connections were getting opened while running the job as compared in 7.5.1

Not sure,if this is the same issue in your case but try to get analysis of ur DBA and DataStage Admin team,when this hanging job is running on upgraded environment.