Operation Console 8.7 stopped working

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Operation Console 8.7 stopped working

Post by sangi1981 »

HI gurus,
we installed in test env operation console.
After installation, it worked fine for some time.
Later, for unexplained reasons, it stopped to work fine.
I explain my assertion.

On server, dsAppwatcher -status and -test tell us that :

Code: Select all

/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -status 
AppWatcher:RUNNING 
EngMonApp:RUNNING 
ODBQueryApp:RUNNING 
ResMonApp:RUNNING 

/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -test 
DSODB is turned ON in the DSODBConfig.cfg file. 
       Link Monitoring is OFF. 
       Job Run Usage is ON. 
       Resource Monitoring is ON. 
Checking Database Connection: 
       Successfully loaded the database driver. 
       Successfully connected to the database. 
       DB Schema version number: 1 
Test Successful.
On server-log side there is no evidence of errors.

On browser side, no info is showed, and a generic "Sorry, an error occurred" is showed over all panels.

Some help, or any suggestions, please!?

Thanks all,
Sandro
Top
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

All I can think is the ODBCQueryApp that handles the back-end for those screens is not working properly. What happens when you restart the ODS?
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
Top
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Post by sangi1981 »

Nothing happens.
We restart console more time, and every time DSAppWatcher -status and -test tells us all services are OK.


:? :? :?

Thanks,
Sandro
Top
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Check the logs in IBM/InformationServer/Server/DSODB/logs to see if they have more information / errors.

If there are lots of logs, then you might want to shut down the ODS, clean out all the old logs, then restart the ODS. That way you know the logs you are looking at are relevant.

Also, just to be sure, check and see if you have the semaphore leak that is in some of the ODBC drivers that are on AIX and 8.7. When the system runs out of semaphores most of the java-based applications on the engine stop working.

Do an ipcs -rs | wc -l and post the count (it may take quite a while if you do have the problem).
Last edited by asorrell on Fri Nov 08, 2013 12:45 pm, edited 1 time in total.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
Top
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Contact ray.wurlod

Post by ray.wurlod »

How much later is "later"? Might the passwords have expired?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Top
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Post by sangi1981 »

DB password doesn't expire. DBAs are able to connect to DSODB via sqlplus.
DSODB ( Oracle installation ) is in same local instance of XMETA.
Configuration of server of my customer is all-in-one: engine+repository+AppServ.

Result of ipcs -rs | wc -l is under 500.
Top
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Contact ray.wurlod

Post by ray.wurlod »

Are you trying to monitor multiple machines in the one console, and perhaps using conflicting settings?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Top
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Post by sangi1981 »

Hi Ray,
no I can't. Server Configuration doesn't permitte this kind of test.

Being a java application, do you know if there is some way to view more application log?

Thanks a lot
Sandro
Top
asorrell
Posts: 1707
Joined: Fri Apr 04, 2003 2:00 pm
Location: Colleyville, Texas

Post by asorrell »

Sandro,

Did you check DSODB/logs (per email above)? There should be specific error logs in there for each ODS process.
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2020
Top
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Check file/directory permissions for stuff under DSODB path.
Did you start the console processes as your admin id (dsadm)?
Validate thar your user id configured in your DSIDBConnect.cfg still has the ability to update your dsodb tables.

Does your database exist on the same server as WAS and engine?
Top
Niv0599
Premium Member
Premium Member
Posts: 13
Joined: Tue Apr 10, 2012 11:47 am

Post by Niv0599 »

Did you check this tech note, appears one of the error message is the same you reported.
http://www-01.ibm.com/support/docview.w ... wg21515290
Datastage is the best thing that happened to ETL
Top
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Post by sangi1981 »

Hi all,
thanks a lot for your help and suggestions.
Unfortunately, there is NO error on logs!!!!
Log4j in file DSODBTuning.cfg are set to INFO:

Code: Select all

log4j.logger.com.ibm.datastage=INFO
log4j.logger.com.ibm.datastage.runtime.common=INFO
log4j.logger.com.ibm.datastage.runtime.engmonapp=INFO
log4j.logger.com.ibm.datastage.runtime.resmonapp=INFO
log4j.logger.com.ibm.datastage.runtime.manager=INFO
log4j.logger.com.ibm.datastage.runtime.odbqapp.server=INFO
log4j.logger.com.ibm.datastage.asb.handler=INFO
Thus, the only information on log files are the following:

AppWatcher-20131106111659060.log

Code: Select all

2013-11-06 11:17:00,522  INFO com.ibm.datastage.runtime.appwatcher.AppWatcher.test(AppWatcher.java:93) - AppWatcher started.
 2013-11-06 11:17:04,160  INFO com.ibm.datastage.runtime.common.db.DBConnector.connect(DBConnector.java:141) - Connected to: jdbc:ibm:oracle://server:xxxx;SID=yyyyyyy.
AppWatcher_Svc-20131016143134.log

Code: Select all

2013/10/16 14:31:34: AppWatcher started.
ResTrackApp started, PID = 27132856.
EngMonApp-20131016140235373.log

Code: Select all

2013-10-16 14:02:36,124  INFO com.ibm.datastage.runtime.common.db.DBConnector.connect(DBConnector.java:141) - Connected to: jdbc:ibm:oracle://server:xxxx;SID=yyyyyyy.
 2013-10-16 14:02:36,242  INFO com.ibm.datastage.runtime.engmonapp.FileManager.<init>(FileManager.java:318) - EngMonApp running as user id: uid=203(dsadm) gid=15(dstage) groups=1(staff)
 2013-10-16 14:02:36,243  INFO com.ibm.datastage.runtime.engmonapp.FileManager.<init>(FileManager.java:327) - Java local timezone: Europe/Rome
 2013-10-16 14:02:36,243  INFO com.ibm.datastage.runtime.engmonapp.FileManager.watchForDirectories(FileManager.java:358) - Using events directory: /opt/IBM/InformationServer/Server/DSODB/events
 2013-10-16 14:02:36,278  INFO com.ibm.datastage.runtime.engmonapp.FileManager.processDirectory(FileManager.java:500) - Reading event files with codepage: UTF-8
ResMonApp-20131016143142292.log

Code: Select all

2013-10-16 14:31:42,832  INFO com.ibm.datastage.runtime.common.db.DBConnector.connect(DBConnector.java:141) - Connected to: jdbc:ibm:oracle://server:xxxx;SID=yyyyyyy.
 2013-10-16 14:31:42,928  INFO com.ibm.datastage.runtime.resmonapp.RTManager.run(RTManager.java:360) - Connected to ResTrackApp using ipv4 on port number: 13450
 2013-11-07 10:26:51,759  INFO com.ibm.datastage.runtime.resmonapp.RTManager.checkForStop(RTManager.java:658) - ResMonApp stopping: The stop indicator file has been detected.
odbqapp-20131016143136040.log

Code: Select all

2013-10-16 14:31:36,170  INFO com.ibm.datastage.runtime.odbqapp.server.ODBQueryAppSocketServer.main(ODBQueryAppSocketServer.java:218) - ODBQueryApp server is listening on port :13451

Event and log dirs have rwx permissions for group of dsadm.
Database, engine and WAS are all on the same server.

Thanks all,
Sandro
Top
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

You need to switch to DEBUG, this should be helpful: https://www-304.ibm.com/support/docview ... wg21509567
-craig

"You can never have too many knives" -- Logan Nine Fingers
Top
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Post by sangi1981 »

Thank you for link.
I'm trying to change some settings....
I'll let you know.
Sandro
Top
sangi1981
Participant
Posts: 99
Joined: Fri Jun 13, 2008 8:10 am

Post by sangi1981 »

Hello to all,
I think I have discovered the cause of the problem.

After studying various links on the IBM support:

https://www-304.ibm.com/support/docview ... wg21509567
http://www-01.ibm.com/support/docview.w ... wg21447204
https://www-304.ibm.com/support/docview ... wg21452589

and read a lot of system logs,
I found in the file

Code: Select all

datastage_api_services-server1.log
that the

Code: Select all

ASB agent at the server: 31531 is not running
More, in the agent log I found

Code: Select all

unable to bind port ... 31531
Doing a ps, I found several instances of the agent running.

So I stopped everything: engine, was, agents. Then, I killed any process yet running.
Restarting services and engine, the console works again.

Thanks for the help,
Hello,
Sandro
Top
Post Reply

Return to “General”