Library error when running Server job accessing DB2

[gainfomart]

Dear all,

I have found the following problems when moving my jobs from Development to the production servers.

This server job is reading a DB2 tables and write to a text file (log file).

"ds_loadlibrary: error in dlopen of ccpassive.so -/opt/IBM/InformationServer/Server/PXEngine/lib/liborchmonitorx86_64.so: undefined symbol: _ZNK11RWPtrVector11boundsCheckEm"

However, I have tried to do the following:
1. make another simple and similar parallel job and it works
2. make a new server job and it doesn't work again
3. check LD_LIBRARY_PATH for both servers - they are the same
4. check DS versions for both servers - 8.7 FP1
5. both running on SUSE Linux 11 SP2 (64 bit)

What is the next step / possible action? Thanks in advance.

[Mike]

So, two different servers?

1) Try your server job on the server where the parallel job is working.
2) Try your parallel job on the server where the server job is not working.
3) Involve your support provider.

Mike

[Mike]

I've made the assumption that you are using the DB2 Connector in both jobs and connecting to the same DB2 database.

Let us know if that is not the case.

Mike

[gainfomart]

Thanks, let me clarify...
1. Development Server - both server / parallel job fine
2. Production Server - Server job=dead / parallel job fine
3. call but wait for reply...

Yes, both jobs are connecting to DB2 database v9.7 fp 8
But again, 2 different database servers in 2 different networks. So, no quick chance to test the DB2 connections by exchange

[asorrell]

1) When you said LD_LIBRARY_PATH is same on both systems, do you mean exactly the same, including the order of the libraries?
2) Have you checked the library it is complaining about (/opt/IBM/InformationServer/Server/PXEngine/lib/liborchmonitorx86_64.so) to insure it is present on both systems and is identical in size? You can also use the strings command (or the SUSE equivalent) to check version information:

Code: Select all

$ strings liborchmonitoraix64.so | grep Version
Version: AIX64 Torrent 2_3 2012/06/01 23:18:57 GMT aixbld05:/builds/IS_8_7_FP1_RC1/PXEngine.src/orch_master/monitor//builds/IS_8_7_FP1_RC1/PXEngine.src/orch_master/Libs/liborchmonitoraix64.a root Exp 
$

I'd suggest copying the library from the working system to the non-working system, but I'm concerned that any library difference may indicate that the systems aren't as identical as you think they are, so backup the original first just in case IBM needs it for diagnosis.

[gainfomart]

I'm still facing the error. Here is my note.

1. LD_LIBRARY_PATH is the same for DB2(V9) and local library paths. Only difference is using different client version for other databases.

2. I tired to check the version using strings command on SUSE. liborchmonitorx86_64 have same version info. and The size and creation date are all equal. Replace this file with the one in development take no effect.

3. I replaced the whole ASBNode directory and PXEngine directory but it does not work.

4. For server job execution, which libraries will be referred to ?

5. Is it possible to check the library dependence on liborchmonitorx86_64.so ?

[asorrell]

The list of libraries a server job could use is extensive. The tool used to find library dependencies is ldd. You can find lots of tutorials on the web including http://www.itworld.com/nls_unix_lib060727. Run that on /opt/IBM/InformationServer/Server/DSEngine/lib/ccpassive.so

One other note, check the LD_LIBRARY_PATH in the environment variable listing for the running AND failing jobs (director log - first few entries). See if anything is overriding the setting somehow.

[gainfomart]

LD_LIBRARY_PATH inside JOB_LOG is the same as it's in the DSENV.


Below is some foundings after issuing 'ldd' commands.

****************************************************************************
ldd result on Development (cppassive.so)(by root user)
****************************************************************************

linux-vdso.so.1 => (0x00007fff9dbff000)
libdsplugin.so => not found
libfcl.so => not found
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fb73c04a000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb73bdd1000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb73bbba000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb73b846000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb73c49b000)

****************************************************************************
ldd result on Production (cppassive.so)(by root user)
****************************************************************************

linux-vdso.so.1 => (0x00007fffee9fe000)
libdsplugin.so => not found
libfcl.so => not found
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f47e642a000)
libm.so.6 => /lib64/libm.so.6 (0x00007f47e61d4000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f47e5fbd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f47e5c64000)
/lib64/ld-linux-x86-64.so.2 (0x00007f47e6877000)

****************************************************************************

The top 3 libraries are under /DSEngine/lib. They have same size and creation date on both env.
Other libraries are under /lib64 and /usr/lib64. Some differences are found.
As it's not DS directory, will it be able to update the C libraries to the same version by copy & paste?

[libstdc++.so.6]
DEV: libstdc++.so.6 -> libstdc++.so.6.0.16 (981K)
PROD: libstdc++.so.6 -> libstdc++.so.6.0.10 (996K)

[libm.so.6]
DEV: libm.so.6 -> libm-2.11.3.so (530K)
PROD: libm.so.6 -> libm-2.9.so (343K)

[libgcc_s.so.1]
DEV: libgcc_s.so.1 (87K)
PROD: libgcc_s.so.1 (95K)

[libc.so.6]
DEV: libc.so.6 --> libc-2.11.3.so (1.7M)
PROD: libc.so.6 --> libc-2.9.so (1.4M)

[ld-linux-x86-64.so.2]
DEV: libc.so.6 --> ld-2.11.3.so (148K)
PROD: libc.so.6 --> ld-2.9.so (125K)

[asorrell]

Unless your jobs are running as root I'm not certain if those ldd's are going to help. I always test by being logged in as dsadm, so ldd is attempting to snap links to all the libraries that are in the library path I'm trying to debug.

I can tell you that my ldd output has way more libraries listed and none of them are showing up as "not found" or blank. I'm on AIX so posting my output wouldn't help. However, as far as I know, all the listed libraries should be "found".

Can you re-run as dsadm just to be sure it found everything after dsenv was activated?

I'm not certain what you mean by "cut and paste". As long as it finds the required UNIX libraries and they are at the minimal required release or greater, it usually doesn't have issues with minor library differences.

Also - be sure to ldd the liborchmonitorx86_64.so library as well.

I'm also just noticing that even though you said it was a Server job, it's complaining about a PX library, which is also a bit strange...

[gainfomart]

Thanks,

Our current user credential settings allow jobs running as root.
After dsenv activated, I tried the ldd command again. The result of root and dsadm are the same. Same error found while running the job with different credential setting - root and dsadm on OS.

****************************************************************************
result of (ldd ccpassive.so)(by root /dsadm)(prod)
****************************************************************************
linux-vdso.so.1 => (0x00007fff9dbff000)
libdsplugin.so => /opt/IBM/InformationServer/Server/DSEngine/lib/libdsplugin.so (0x00007f85a5aa1000)
libfcl.so => /opt/IBM/InformationServer/Server/PXEngine/lib/libfcl.so (0x00007f85a57b4000)
libstdc++.so.6 => /opt/IBM/InformationServer/ASBNode/apps/proxy/cpp/linux-all-x86_64/libstdc++.so.6 (0x00007f85a55b5000)
libm.so.6 => /lib64/libm.so.6 (0x00007f85a532d000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f85a5117000)
libc.so.6 => /lib64/libc.so.6 (0x00007f85a4dbd000)
libicuuc.so.32 => opt/IBM/InformationServer/Server/PXEngine/lib/libicuuc.so.32
libicudata.so.32 => opt/IBM/InformationServer/Server/PXEngine/lib/libicudata.so.32
libpthread.so.0 => /lib64/libpthread.so.0
libnsl.so.1 => /lib64/libnsl.so.1
libdl.so.2 => /lib64/libdl.so.2
libicutu.so.32 => /opt/IBM/InformationServer/Server/PXEngine/lib/libicutu.so.32
libicuio.so.32 => /opt/IBM/InformationServer/Server/PXEngine/lib/libicuio.so.32
libxml4c.so.56 => /opt/IBM/InformationServer/Server/DSComponents/lib/libxml4c.so.56
liborchx86_64.so => /opt/IBM/InformationServer/Server/PXEngine/lib/liborchx86_64.so
liborchcorex86_64.so => /opt/IBM/InformationServer/Server/PXEngine/lib/liborchcorex86_64.so
/lib64/ld-linux-x86-64.so.2
liborchmonitorx86_64.so => /opt/IBM/InformationServer/Server/PXEngine/lib/liborchmonitorx86_64.so

****************************************************************************
result of (ldd liborchmonitorx86_64.so)(by root /dsadm)(prod)
****************************************************************************
linux-vdso.so.1 => (0x00007fffb0bfe000)
/opt/IBM/InformationServer/Server/DSComponents/lib/libicui18n.so
libicuio.so.32 => /opt/IBM/InformationServer/Server/PXEngine/lib/libicuio.so.32
libicuuc.so.32 =>/opt/IBM/InformationServer/Server/PXEngine/lib/libicuuc.so.32
liborchcorex86_64.so => /opt/IBM/InformationServer/Server/PXEngine/lib/liborchcorex86_64.so
libstdc++.so.6 => /opt/IBM/InformationServer/ASBNode/apps/proxy/cpp/libstdc++.so.6
libm.so.6 => /lib64/libm.so.6
libgcc_s.so.1 => /lib64/libgcc_s.so.1
libc.so.6 => /lib64/libc.so.6
libicudata.so.32 => /opt/IBM/InformationServer/Server/PXEngine/lib/libicudata.so.32
libpthread.so.0 => /lib64/libpthread.so.0
/lib64/ld-linux-x86-64.so.2

****************************************************************************

[chulett]

You really need to involve your official support provider, if you haven't already.

[asorrell]

I'm going to have to agree with Craig. The last ldd doesn't show any missing libraries. You need to get Customer Service to take a look at it. They'll probably have to get engineering to track down the problem. It may be a 32 / 64 bit incompatibility of some sort between one of the non-DataStage libraries.