Parallel routines

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Parallel routines

Post by abc123 »

I am trying to write a simple parallel routine.I wrote a simple C+ application, compiled it, lined it and created the executable. I created the .so file. In the routine, in the Library Path, I put in the path and filename of the .so file. I wrote a simple job to test this routine. My job
compiles fine but when I go to run it I get the following error:

-------------------------------------------------------------------------------
Transformer_1: Failed to load the library "V0S1_RoutineTest_Transformer_1.so"; either the directory containing the library file
is not on the library search path, or the library was compiled on a system
that is incompatible with this system: Could not load "V0S1_RoutineTest_Transformer_1": /opt/Ascential/DataStage/Projects/MyProject/RT_BP574.O/V0S1_RoutineTest_Transformer_1.so: undefined symbol: _Z13rtOnev.
------------------------------------------------------------------------------

Name of my routine is rtOne. Name of my job is RoutineTest.

My C++ code for the routine is:

-------------------------------------------------------------------------
#include <cstdlib>
#include <iostream>
#include <string>

using namespace std;

int main(int argc, char *argv[])
{

cout << "Hello World" << endl;
return EXIT_SUCCESS;
}
---------------------------------------------------------------------------

This code works fine from Unix. I went through all the related posts in this forum. I could not find a complete example. I would appreciate any help.
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

A few pointers:
-Create the routine as a stand alone function. THat means that it should not have a main function but could be executed if called from a main function.
-Instead of creating the object file and supplying it as a library, supply it as an object file. Choose the option of object file instead of library.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Yuan_Edward
Participant
Posts: 73
Joined: Tue May 10, 2005 6:21 pm
Location: Sydney

Re: Parallel routines

Post by Yuan_Edward »

It may be an issue with the compiler/linker options. Are you using dynamic or static library?

Anyway the object file optin will always work :) . I would recommend to use the same compiler options as DataStage (APT_COMPILEOPT) when you compile the C++ codes into the object file.
abc123 wrote:I am trying to write a simple parallel routine.I wrote a simple C+ application, compiled it, lined it and created the executable. I created the .so file. In the routine, in the Library Path, I put in the path and filename of the .so file. I wrote a simple job to test this routine. My job
compiles fine but when I go to run it I get the following error:

-------------------------------------------------------------------------------
Transformer_1: Failed to load the library "V0S1_RoutineTest_Transformer_1.so"; either the directory containing the library file
is not on the library search path, or the library was compiled on a system
that is incompatible with this system: Could not load "V0S1_RoutineTest_Transformer_1": /opt/Ascential/DataStage/Projects/MyProject/RT_BP574.O/V0S1_RoutineTest_Transformer_1.so: undefined symbol: _Z13rtOnev.
------------------------------------------------------------------------------

Name of my routine is rtOne. Name of my job is RoutineTest.

My C++ code for the routine is:

-------------------------------------------------------------------------
#include <cstdlib>
#include <iostream>
#include <string>

using namespace std;

int main(int argc, char *argv[])
{

cout << "Hello World" << endl;
return EXIT_SUCCESS;
}
---------------------------------------------------------------------------

This code works fine from Unix. I went through all the related posts in this forum. I could not find a complete example. I would appreciate any help.
Edward Yuan
mohdsuf
Participant
Posts: 24
Joined: Mon Jun 18, 2007 3:21 am
Location: India

Post by mohdsuf »

As far as my knowledge and experience goes, you have to have a FUNCTION with any return type required by your C/C++ program and call that function in your MAIN. You must have to write the MAIN after the declaration of your FUNCTION ie FUNCTION first and MAIN body below it.
Also the name of the routine name and external subroutine name in DSMANAGER should always be same as that of FUNCTION name in C++ program, and compile the code in the project directory.
Try "aCC" compiler for compiling
Try this I think with this simple modification you can run your parallel routine. Let me know that it worked for you or not.
I'm also giving a sample c code which you can try with your system.
--------------------------------------------------------------------------------------
/* The function greatest of the three numbers entered*/

#include <stdio.h>


int greatest(int a,int b,int c)
{
if (a>b)
{
if (a>c)
{
return a;
}
else
return c ;
}
else
if(b>c)
{
return b;
}
else
return c;
}
int main()
{
int a1,b1,c1,great;
a1=0;
b1=0;
c1=0;
great=greatest(a1,b1,c1);
return great;
}
Sufiyan
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

DSGuru2B, how do you create a routine as a standalone function? Are you talking about the routine or the C++ code?
As far as the object file, I tried the following:
I checked Object and I put in the .so file and tried. I checked Object again and put in the .o file and tried. I also checked Library, put in the .so file and tried. All failed.
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

Yuan_Edward, my compiler option from APT_COMPILEOPT is:
-O -fPIC -Wno-deprecated -c

I created the object file with that. I also tried:

-shared -O -fPIC -Wno-deprecated -c

However, this does not create the shared object file.
mohdsuf
Participant
Posts: 24
Joined: Mon Jun 18, 2007 3:21 am
Location: India

Post by mohdsuf »

dear
The way gave the example to create the routine is the live one and I have given you the simplest one. Just try the code and compile it with aCC -z option to prepare .o file. I'm sure you will not get the problem at all.
Sufiyan
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

mohdsuf, my compiler is g++32. I put your code in a file called one.cpp. I tried:

g++32 -c one.cpp

I also tried:

g++32 -O -fPIC -Wno-deprecated -c one.cpp

I then made the object file shared as:

g++32 -shared -o libone.so one.o

Then, I tried everything in my previous post. I still get the following error:

Transformer_1: Failed to load the library "V0S1_RoutineTest_Transformer_1.so"; either the directory containing the library file
is not on the library search path, or the library was compiled on a system
that is incompatible with this system: Could not load "V0S1_RoutineTest_Transformer_1": /opt/Ascential/DataStage/Projects/MyProject/RT_BP574.O/V0S1_RoutineTest_Transformer_1.so: undefined symbol: _Z13rtOnev.
mohdsuf
Participant
Posts: 24
Joined: Mon Jun 18, 2007 3:21 am
Location: India

Post by mohdsuf »

Are you giving the name of the routine/subroutine same as that of function name in C++ code. The routine/subroutine name in dsmanager must match with the function name in your code. Also check that the object type is set as "Object".
Sufiyan
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

Thanks mohdsuf for trying to help. I called everything greatest like your function. I always had the Object checked. My return type is int. I still get the exact same error.

Just to clarify every setting that I have:
Routine Name: greatest
Category: MyCategory
External subroutine name: greatest
Object Type: Object
Library Path: /fld1/fld2/fld2/greatest.so
Type: External Function
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

I read on one of the posts here that you have to import the External Function using Datastage Manager. Do I have to do that? Under Datastage Manager, External Functions is disabled.
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

Ok. I figured it out from one of Ray's posts. We can only have the function. We cannot have main()
anywhere. My routine is returning value. However, my original requirement is that I send in 5
string parameters and get 7 back. So there are 5 input parameters and 7 output parameters.
Can we return multiple parameters back from a function? How would we read multiple parameters
back into multiple columns? May be into one column and parses it?
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

I am glad you figured it out. Thats exactly what I said in my first post, no main function is allowed.
Anywho, you can have as many input parameters and only a single return variable. But you could concatenate all your 7 variables together and return the entire string which you can then parse out.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
abc123
Premium Member
Premium Member
Posts: 605
Joined: Fri Aug 25, 2006 8:24 am

Post by abc123 »

Thanks DSGuru2B. I didn't understand it at that time.
Post Reply