abt db2 EE

[Kirtikumar]

Hi,

We are using DB2 EE stage to access DB2.

We have a SMP unix called as Server1 and DB2 is installed on Server2.
Our jobs follow ETL methodology i.e. 1 extract job, 1 transform and then 1 load job.

Now the configuration file consists of 4 nodes with fastname a Server1 and one db2 node with fastname as Server2. The db2 server has no PX on it, so it is not the part of default node pool and defined as the member of DB2 pool.
While extract this works fine and dataset has one partition file from dataset on db2 server i.e. Server2 and remaining on Sever1.
Here the DB2 EE stage runs on Server2.

Now during Transform job, it gives one error as:
node_db2node1: Error loading "V194S1_GLEAMMthCMArrIPRelDervationJob_CMArrIpRel_Derivation_Tfp": Could not find "V194S1_GLEAMMthCMArrIPRelDervationJob_CMArrIpRel_Derivation_Tfp"
on the library search path defined by the environment variable LIBPATH; path:
/gcdm/dev/project/GrpCreditDM_DEV/RT_BP5642.O:/DataStageApp/Ascential/DataStage/DSCAPIOp:
/DataStageApp/Ascential/DataStage/RTIOperators:/DataStageApp/Ascential/DataStage/DSParallel:
/DataStageApp/Ascential/DataStage/PXEngine/user_lib:/DataStageApp/Ascential/DataStage/PXEngine/lib:
/gcdm/dev/project/GrpCreditDM_DEV/buildop:/DataStageApp/Ascential/DataStage/branded_odbc/lib:
/DataStageApp/Ascential/DataStage/DSEngine/lib:/DataStageApp/Ascential/DataStage/DSEngine/uvdlls:
/DataStageApp/Ascential/DataStage/DSEngine/java/jre/bin/classic:
/DataStageApp/Ascential/DataStage/DSEngine/java/jre/bin:/db2/cdwhd32/home/sqllib/lib:
/opt/sybase/OCS-12_5/lib:/usr/local/openwin/lib:/usr/lib:/lib; for class "APT_TransformOperatorImplV194S1_GLEAMMthCMArrIPRelDervationJob_CMArrIpRel_Derivation_Tfp"

node_db2bode is Server2 in config file.

It is trying to put one transformer process one Server2 which is not the part of default pool, it is in different pool called 'db2'. Why is it still trying to load one transformer process on this Server2, ideally accorirding to config, it should not put this transformer process on db2 server.

[roy]

Hi,
Can you post your config file?

[vmcburney]

Set the environment variable $APT_DUMP_SCORE to true within your job to get better Director log messages. As previously noted post your config file. Have you tried testing the same job writing to a copy stage (save job under a new name, remove all stages except DB2 and add a copy stage). This is a useful technique for isolating one part of a job.

[Kirtikumar]

The config file is as follows:

Code: Select all

{
  node "node1"
  {
    fastname "Server1"
    pools ""
    resource disk "/tmp/Datasets" {pools ""}
    resource scratchdisk "/Ascential/Scratch" {pools ""}
  }

  node "node2"
  {
    fastname "Server1"
    pools ""
    resource disk "/tmp/Datasets" {pools ""}
    resource scratchdisk "/Ascential/Scratch" {pools ""}
  }

  node "node3"
  {
    fastname "Server1"
    pools ""
    resource disk "/tmp/Datasets" {pools ""}
    resource scratchdisk "/Ascential/Scratch" {pools ""}
  }

  node "node4"
  {
    fastname "Server1"
    pools ""
    resource disk "/tmp/Datasets" {pools ""}
    resource scratchdisk "/Ascential/Scratch" {pools ""}
  }
  node "db2node1"
  {      
    fastname "Server2"
    pools "db2"
    resource disk "/DataStageProject/scratch" {pools ""}
    resource scratchdisk "/DataStageProject/scratch" {pools ""}
  }
}

I tried using copy stage, still the same error thrown.
One more observation regardless of the error, the output is generated properly, but this fatal error is reported for DB2 node. As mentioned DB2 node is not the part of default node pool.
One more thing worth mentioning is if I substitute filter in place of transformer or remove transformer from the job, the error disappears.

[vmcburney]

I could be barking up the wrong river without a pole but it could be that DataStage is combining your DB2 and transformer operators into one process. To quote the manual "Under the covers DataStage can combine the operators that underlie parallel stages so that they run in the same process.".

To avoid this modify the properties of your transformer (top left button inside the transformer form) and on the Stage - Advanced tab change the Combinality Mode from "Auto" to "Don't Combine".

[Kirtikumar]

Vincent,

Your guess about combining is 100% correct!!!
After adding the APT_DUMP_SCORE, when I saw the DS was combining the operators, I added do not combine in both the stages. After doing this also it is throwing same fatal error and generating proper output. From the messages and dump score, it is not putting the transformer process on db2 node as it is not the part of default pool. It just puts the trans process on Server1 in RTI_BPxxx.O directory for that job. While executing it expects the same trans process to be avaiable on Server2 which is db2. But it does no find this execuatable file on Server2. If you observe the error message, in the lib path first dir is the location where it is keepiing the trans executable file. On Server1 it puts that file there, but this does not happen for Server2 as server2 is not the part of default pool. But I am not getting if it is not putting the trans process on Server2, why is it trying to execute it from there?
One more thing: As mentioned I have one extract job where I am cretaing dataset from DB2 EE stage. Then I am using this dataset in transform job where there no DB2 EE stage. So earlier, it was combining first dataset and transformer. Now I have added Dont combine.

Still getting the same error!!!

[Kirtikumar]

I tried using the job in job sequence and despite of the Fatal message reported in DS Director the PX job is returning status RUNOK in sequence.

But still not able to trace the cause of the error!!!

[vmcburney]

Can't help you on that one. Strange that you would get an error and still have all rows processed. Have you mounted the /apps/Ascential and /projects directories to the remote DB2 box? The DataStageEE Remote DB2 Configuration doc states:

Either NFS mount both the /apps/Ascential and /projects to the remote DB2 boxes or use the copy utility provided in the PXHOME directory (NFS mounts are preffered).

[Kirtikumar]

Vincent,

Directories mentioned in config doc are in place on DB2 server.
I think these directories are created/copied instead of mounting them through NFS. The DataStage directory contains all files and directories like PX and DS engine, the executables for osh etc are also there!!!

Do you think that this problem is bcoz of not using common shared mount point?
If the common mount point is used, transformer executable will be on DS EE server. If this project directory is mounted through NFS on DB2 server as well, it will find this executable there as it is project NFS dir used by Server1 and Server2. As it will find this file, fatal error will not be generated.

As the server we are using is remote (in U.K.) and administration is managed by UK DS Admins, I doubt if I can try this or not!!!
But I will ask them to try this out!!!

Thanks a ton for all your inputs, Vincent!!!

[Eric]

1 extract job, 1 transform and then 1 load job.

Just to Note that the extract job will be reading from DB2 in sequential mode and the transform job will be running in parallel mode.

You could try?
In the transformer stage -> properties -> stage tab -> advanced you can set which nodes the transformer will use, so just add node 1 - 4.

[Kirtikumar]

1 extract job, 1 transform and then 1 load job.

You could try?
In the transformer stage -> properties -> stage tab -> advanced you can set which nodes the transformer will use, so just add node 1 - 4.

I tried constraining the node to node1-4. Still getting the same error!!!