orchgeneral,orchsort,orchstats could not load Bad address

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 9:52 am

Parallel job is getting aborted due to the following error.

Warning : main_program : orchgeneral: could not load "orchgeneral": bad address
Warning : main_program : orchsort: could not load "orchsort": bad address
Warning : main_program : orchstats: could not load "orchstats": bad address

fatal : Path search failure
fatal : error loading orchgeneral : Bad address

please suggest

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 9:58 am

Strange error. Seems like the engine is not able to fine the PXEngine path. Look in the dsenv script in $DSHOME and see if $APT_ORCHHOME is properly set. Is this the first time your getting the error with a parallel job? Is this a new installation?

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 10:00 am

We have been using PX for the past one year, but this is the first time we are facing this problem. This happened in Prod, i will request our support guy to check $APT_ORCHHOME.

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 10:07 am

It seems like the libraries could not be loaded. I am on HP-UX so I have liborchsorthpux.sl , liborchstatshpux.sl and liborchgeneralhpux.sl for orchsort, orchstat and orchgeneral respectively. These are present in $APT_ORCHHOME/lib folder. I think your issue might be related to these library files as well.

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 10:27 am

look like LIBPATH changed. The order changed, but dsenv didnot changed recently.

Prior Version :
LIBPATH=/opt/Ascential/DataStage/Projects/mercybi/RT_BP2975.O:/opt/Ascential/DataStage/Projects/mercybi/buildop:/opt/Ascential/DataStage/DSCAPIOp:/opt/Ascential/DataStage/RTIOperators:/opt/Ascential/DataStage/DSParallel:/opt/Ascential/DataStage/PXEngine/user_lib:/opt/Ascential/DataStage/branded_odbc/lib:/opt/Ascential/DataStage/DSEngine/lib:/opt/Ascential/DataStage/DSEngine/uvdlls:/opt/Ascential/DataStage/DSEngine/java/jre/bin/classic:/opt/Ascential/DataStage/DSEngine/java/jre/bin::/u01/oracle/10gclient/lib32:/opt/Ascential/DataStage/PXEngine/bin:/opt/Ascential/DataStage/PXEngine/lib:/usr/lib:/lib

Current Version :
LIBPATH=/opt/Ascential/DataStage/Projects/mercybi/RT_BP2975.O:/opt/Ascential/DataStage/DSCAPIOp:/opt/Ascential/DataStage/RTIOperators:/opt/Ascential/DataStage/DSParallel:/opt/Ascential/DataStage/PXEngine/user_lib:/opt/Ascential/DataStage/Projects/mercybi/buildop:/opt/Ascential/DataStage/branded_odbc/lib:/opt/Ascential/DataStage/DSEngine/lib:/opt/Ascential/DataStage/DSEngine/uvdlls:/opt/Ascential/DataStage/DSEngine/java/jre/bin/classic:/opt/Ascential/DataStage/DSEngine/java/jre/bin::/u01/oracle/10gclient/lib32:/opt/Ascential/DataStage/PXEngine/bin:/opt/Ascential/DataStage/PXEngine/lib:/usr/lib:/lib

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 10:35 am

Clear the scratch space, temp area. Run the job again. Is it only this job or other jobs as well. Run a sample job in prod to see. Also keep the two environments identical, to the extent of comments.

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 10:47 am

We are having the same issue in all the environments, i tried running a different job in development, but we are having the same issue.

we are using AIX server

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 10:51 am

Did you try looking for those files in the lib path. The ones that I mentioned in bold. They wont be the exact but atleast similar.

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 11:01 am

i verified orchsort.o,orchgeneral.o,orchstats.o in lib folder of PXEngine.

/opt/Ascential/DataStage/PXEngine.751.2/lib

But these were not changed recently.

lrwxrwxrwx 1 dsadm dsstage 10 Apr 17 2006 liborchio64aix3.so -> orchio64.o
lrwxrwxrwx 1 dsadm dsstage 13 Apr 17 2006 liborchmonitoraix3.so -> orchmonitor.o
lrwxrwxrwx 1 dsadm dsstage 10 Apr 17 2006 liborchodbcaix3.so -> orchodbc.o
lrwxrwxrwx 1 dsadm dsstage 15 Apr 17 2006 liborchoracle10gaix3.so -> orchoracle10g.o
lrwxrwxrwx 1 dsadm dsstage 14 Apr 17 2006 liborchoracle8iaix3.so -> orchoracle8i.o
lrwxrwxrwx 1 dsadm dsstage 14 Apr 17 2006 liborchoracle9iaix3.so -> orchoracle9i.o
lrwxrwxrwx 1 dsadm dsstage 12 Apr 17 2006 liborchoracleaix3.so -> orchoracle.o
-rw-r--r-- 1 dsadm dsstage 1527361 Mar 22 2005 liborchoslaix3.a
lrwxrwxrwx 1 dsadm dsstage 10 Apr 17 2006 liborchpftpaix3.so -> orchpftp.o
lrwxrwxrwx 1 dsadm dsstage 14 Apr 17 2006 liborchsasop612aix3.so -> orchsasop612.o
lrwxrwxrwx 1 dsadm dsstage 12 Apr 17 2006 liborchsasop8aix3.so -> orchsasop8.o
lrwxrwxrwx 1 dsadm dsstage 12 Apr 17 2006 liborchsasop9aix3.so -> orchsasop9.o
lrwxrwxrwx 1 dsadm dsstage 10 Apr 17 2006 liborchsortaix3.so -> orchsort.o
lrwxrwxrwx 1 dsadm dsstage 11 Apr 17 2006 liborchstatsaix3.so -> orchstats.o
lrwxrwxrwx 1 dsadm dsstage 12 Apr 17 2006 liborchsybaseaix3.so -> orchsybase.o
lrwxrwxrwx 1 dsadm dsstage 14 Apr 17 2006 liborchteradataaix3.so -> orchteradata.o
lrwxrwxrwx 1 dsadm dsstage 5 Apr 17 2006 liboshaix3.so -> osh.o
-rw-r--r-- 1 dsadm dsstage 951673 Mar 22 2005 librwtool.a
lrwxrwxrwx 1 dsadm dsstage 16 Apr 17 2006 libustdio.so -> libustdio22.0.so
-rw-r--r-- 1 dsadm dsstage 117795 Aug 14 2003 libustdio22.0.so
lrwxrwxrwx 1 dsadm dsstage 16 Apr 17 2006 libustdio22.so -> libustdio22.0.so
lrwxrwxrwx 1 dsadm dsstage 25 Apr 17 2006 netstub.a -> ../informix/lib/netstub.a
-rwxr-xr-x 1 dsadm dsstage 2308517 Mar 22 2005 orchbuildop.o
-rwxr-xr-x 1 dsadm dsstage 138525 Mar 22 2005 orchcommondb.o
-rwxr-xr-x 1 dsadm dsstage 1491660 Mar 22 2005 orchdb2op7.o
-rwxr-xr-x 1 dsadm dsstage 1492451 Mar 22 2005 orchdb2op8.o
-rwxr-xr-x 1 dsadm dsstage 3749819 Mar 22 2005 orchgeneral.o
-rwxr-xr-x 1 dsadm dsstage 2669990 Mar 22 2005 orchinformix.o
-rwxr-xr-x 1 dsadm dsstage 2669478 Mar 22 2005 orchinformixnohpl.o
-rwxr-xr-x 1 dsadm dsstage 43070 Mar 22 2005 orchio64.o
-rwxr-xr-x 1 dsadm dsstage 141436 Mar 22 2005 orchmonitor.o
-rwxr-xr-x 1 dsadm dsstage 617014 Mar 22 2005 orchodbc.o
lrwxrwxrwx 1 dsadm dsstage 15 Apr 17 2006 orchoracle.o -> orchoracle10g.o
-rwxr-xr-x 1 dsadm dsstage 1754119 Mar 22 2005 orchoracle10g.o
-rwxr-xr-x 1 dsadm dsstage 1753871 Mar 22 2005 orchoracle8i.o
-rwxr-xr-x 1 dsadm dsstage 1754091 Mar 22 2005 orchoracle9i.o
-rwxr-xr-x 1 dsadm dsstage 489482 Mar 22 2005 orchpftp.o
-rwxr-xr-x 1 dsadm dsstage 1486916 Mar 22 2005 orchsasop612.o
-rwxr-xr-x 1 dsadm dsstage 1487438 Mar 22 2005 orchsasop8.o
-rwxr-xr-x 1 dsadm dsstage 1487490 Mar 22 2005 orchsasop9.o
-rwxr-xr-x 1 dsadm dsstage 915669 Mar 22 2005 orchsort.o
-rwxr-xr-x 1 dsadm dsstage 2358749 Mar 22 2005 orchstats.o
-rwxr-xr-x 1 dsadm dsstage 1122158 Mar 22 2005 orchsybase.o
-rwxr-xr-x 1 dsadm dsstage 897838 Mar 22 2005 orchteradata.o
-rw-r--r-- 1 dsadm dsstage 79642 Mar 22 2005 osh.o

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 11:08 am

I noticed that osh changed today morning and we are facing the problems after that.

/opt/Ascential/DataStage/PXEngine.751.2/bin]ls -l
total 3904
-rwxr-xr-x 1 dsadm dsstage 57646 Mar 22 2005 aptsev2.612
-rwxr-xr-x 1 dsadm dsstage 30262 Mar 22 2005 aptsev2.8.2
-rwxr-xr-x 1 dsadm dsstage 215187 Mar 22 2005 aptsev2.9.1
-rwxr-xr-x 1 dsadm dsstage 54 Mar 22 2005 build.txt
-rwxr-xr-x 1 dsadm dsstage 273018 Mar 22 2005 buildop
-rwxr-xr-x 1 dsadm dsstage 31434 Mar 22 2005 checkport
-rwxr-xr-x 1 dsadm dsstage 5695 Mar 22 2005 db2esql.bnd
-rwxr-xr-x 1 dsadm dsstage 386 Mar 22 2005 db2grant.sh
-rwxr-xr-x 1 dsadm dsstage 578 Mar 22 2005 db2setup.sh
-rwxr-xr-x 1 dsadm dsstage 38814 Mar 22 2005 dsrecords
-rwxr-xr-x 1 dsadm dsstage 22802 Mar 22 2005 forktest
-rwxr-xr-x 1 dsadm dsstage 35297 Mar 22 2005 gencnval
-rwxr-xr-x 1 dsadm dsstage 52331 Mar 22 2005 icu_utils
-rwxr-xr-x 1 dsadm dsstage 28325 Mar 22 2005 makeCatalog
-rwxr-xr-x 1 dsadm dsstage 59786 Mar 22 2005 makeconv
-rwxr-xr-x 1 dsadm dsstage 7787 Mar 22 2005 makerangemap
-rwxr-xr-x 1 dsadm dsstage 237477 Mar 22 2005 orchadmin
-rwxr-xr-x 1 dsadm dsstage 31586 Mar 22 2005 orchconvert
-rwxr-xr-x 1 dsadm dsstage 33224 Mar 22 2005 orchdaemon
-rwxr-xr-x 1 dsadm dsstage 192441 Mar 22 2005 orchdbutil
-rwxr-xr-x 1 dsadm dsstage 2372 Mar 22 2005 orchfscat
-rwxr-xr-x 1 dsadm dsstage 2425 Mar 22 2005 orchfsls
-rwxr-xr-x 1 dsadm dsstage 2047 Mar 22 2005 orchfsrm
-rwxr-xr-x 1 dsadm dsstage 160242 Mar 06 04:55 osh
-rwxr-xr-x 1 dsadm dsstage 35781 Mar 22 2005 oshprinthost
-rwxr-xr-x 1 dsadm dsstage 40698 Mar 22 2005 parseschema
-rwxr-xr-x 1 dsadm dsstage 9916 Mar 22 2005 performance_convert
-rwxr-xr-x 1 dsadm dsstage 25003 Mar 22 2005 quote_propstring
-rwxr-xr-x 1 dsadm dsstage 211386 Mar 22 2005 readcobol
-rwxr-xr-x 1 dsadm dsstage 64956 Mar 22 2005 readstats
-rwxr-xr-x 1 dsadm dsstage 707 Mar 22 2005 register-library
-rwxr-xr-x 1 dsadm dsstage 707 Mar 22 2005 register-operator
-rwxr-xr-x 1 dsadm dsstage 20069 Mar 22 2005 registry-services
-rwxr-xr-x 1 dsadm dsstage 707 Mar 22 2005 unregister-library

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 11:10 am

You might have nailed the problem. Go back to your admin and start questioning.

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 11:40 am

But osh is an executable file, how could this be changed by someone.

I noticed that the same was changed in development at 4:55AM, TEST at 4:57AM and PROD at 5:09 AM.

I believe we cannot change thse executables right.

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 11:42 am

You cannot. But they did get affected. Someone else is having a similar problem today here. He claims that a working job, if copied and renamed was displaying the same error message. Maybe you can give it a test shot. Take a job that works. Double check if it works by compiling and running it. Make a copy, rename and try again.

panchusrao2656 · Post by **panchusrao2656** » Tue Mar 06, 2007 11:53 am

Travis is also working with us, he doenot know that we had this issue in all the environments.

DSGuru can you explore little bit about the files in /bin and /lib of the same parent directory.

osh is in /bin
osh.o is in /lib

osh in bin should refer osh.o in lib right, but osh.o is not changed and it still shows Mar2005, but bin/osh shows today's date.

Can you explore these a little bit

DSguru2B · Post by **DSguru2B** » Tue Mar 06, 2007 12:02 pm

These are executable files. Nothing much I can do at my end. I think its time to give support a call. Something wierd is happening. A bad memory address was being accessed or something.