Creating a Parallel Routine.

thebird · Post by **thebird** » Wed Oct 05, 2005 10:39 am

Hi,

I am trying to create a Parallel routine using a C++ code. I have the C++ code compiled and saved as a shared object string_one.o.

I have linked this object to the routine TestStripRoutine, but while trying to test this routine in a job, I get the following compilation error in the Transformer -

Output from transformer compilation follows:

##I TFCN 000001 12:14:31(000) <main_program>
Ascential DataStage(tm) Enterprise Edition 7.5.1A
Copyright (c) 2004, 1997-2004 Ascential Software Corporation.
All Rights Reserved

##I TOSH 000002 12:14:31(001) <main_program> orchgeneral: loaded
##I TOSH 000002 12:14:31(002) <main_program> orchsort: loaded
##I TOSH 000002 12:14:31(003) <main_program> orchstats: loaded
##I TFSC 000001 12:14:31(006) <main_program> APT configuration file: /opt/etl/Ascential/DataStage/Configurations/default.apt
##W TCOS 000049 12:14:32(000) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##W TFCP 000000 12:14:32(002) <transform> Error when checking composite operator: The number of reject datasets "0" is less than the number of input datasets "1".
##E TFCP 000003 12:14:32(003) <transform> Error when checking composite operator: Unexpected tokens: string; [line 6, character 38].
##E TFCP 000038 12:14:32(004) <transform> Error when checking composite operator: Expected semi-colon; [line 6, character 44].
##E TFCP 000012 12:14:32(005) <transform> Error when checking composite operator: Invalid local variable declaration: string; [line 6, character 63].
##E TFCP 000038 12:14:32(006) <transform> Error when checking composite operator: Expected semi-colon; [line 6, character 77].
##E TFCP 000038 12:14:32(007) <transform> Error when checking composite operator: Expected semi-colon; [line 9, character -1].
##E TFCP 000038 12:14:32(008) <transform> Error when checking composite operator: Expected semi-colon; [line 9, character 11].
##E TFCP 000038 12:14:32(009) <transform> Error when checking composite operator: Expected semi-colon; [line 10, character 0].
##E TFCP 000038 12:14:32(010) <transform> Error when checking composite operator: Expected semi-colon; [line 10, character 13].
##E TFSR 000019 12:14:32(011) <main_program> Could not check all operators because of previous error(s)
##E TCOS 000029 12:14:32(012) <main_program> Creation of a step finished with status = FAILED. (TestBuildOp.Transformer_21)

*** Internal Generated Transformer Code follows:
0001: //
0002: // Generated file to implement the V0S21_TestBuildOp_Transformer_21 transform operator.
0003: //
0004:
0005: // define external functions used
0006: extern string stripinvalidchars(string string,string termstring,string replace);
0007:
0008: // define our input/output link names
0009: inputname 0 DSLink2;
0010: outputname 0 DSLink4;
0011:
0012: initialize {
0013: // define our row rejected variable
0014: int8 RowRejected0;
0015:
0016: // define our null set variable
0017: int8 NullSetVar0;
0018:
0019: // declare our intermediate variables for this section
0020: string InterVar0_0;
0021: string InterVar0_1;
0022:
0023: // initialise constant values which require conversion
0024: InterVar0_0 = "#";
0025: InterVar0_1 = "_";
0026: }
0027:
0028: mainloop {
0029: // initialise our row rejected variable
0030: RowRejected0 = 1;
0031:
0032: // declare our intermediate variables for this section
0033: string InterVar0_2;
0034:
0035: // evaluate columns (no constraints) for link: DSLink4
0036: InterVar0_2 = stripinvalidchars(DSLink2.Name , InterVar0_0 , InterVar0_1);
0037: DSLink4.Name = InterVar0_2;
0038: writerecord 0;
0039: RowRejected0 = 0;
0040: }
0041:
0042: finish {
0043: }
0044:
*** End of Internal Generated Transformer Code

In the C++ code, there are 3 arguements - string,termstring and replace.
The input field that I am trying to pass to this routine is "Name".

Can anyone tell what is happening? Or rather can someone give me a small detailing on how I can get this routine to be up and working?

I did try the manuals, but there very little infor provided there regarding this.

Thanks in advance.

Regards,

The Bird.

ray.wurlod · Post by **ray.wurlod** » Wed Oct 05, 2005 4:41 pm

None of the messages seems to relate to the routine, but rather they appear to relate to unexpected tokens in the data. Remove the routine from the job and see if they persist. If so, fix that issue before reinstating the routine in the job.

thebird · Post by **thebird** » Thu Oct 06, 2005 11:59 am

Ray,

The job that I am using to test the routine is a simple one -

Seq File-------->Transformer------------->Seq File.

The sequential file has a single field, which is passed to the routine along with 2 other arguements. When I remove the routine, the job compiles fine. So, I believe that the problem lies wiht the routine that I am trying to test.

The C++ code that I have accepts 3 arguements - string,termstring and replace. This scans the first arguement for invalid characters, and replaces the invalid characters with the third arguement. The list of invalid characters is provided as the second arguement. The routine has to return a string, after replacing the invalid characters.

Please find below the code I am using -

#include "cstring"
#include "cstdio"
#include "cctype"
#include "cstdlib"

#define PTR_INIT (char *) 1

char *stripinvalidchars(char *string, char *termstring, char *replace)
{
int i, termlen, stringlen;
char *ptr = PTR_INIT;
char *dup_string;

stringlen = strlen(string);
termlen = strlen(termstring);
dup_string = (char *) malloc(stringlen+1);

strcpy(dup_string, string);

for(i = 0; i < termlen ; i++) {
while(ptr) {
if((ptr = strchr(dup_string, (int) termstring)))
*ptr = *replace;
}
ptr = PTR_INIT;
}
return dup_string;
}

I have compiled and created a shared object by the name mystring.o and linked it to the routine. The shared object is in the same path as the one in $SHLIB_PATH.

But I am unable to even compile the job, when I call the routine. I even tried hard coding the arguements as -

RoutineName("Rogues$","$#","_")

But when I compile this, I get the following error (not much different from the one I posted yesterday) -

Output from transformer compilation follows:

##I TFCN 000001 13:44:57(000) <main_program>
Ascential DataStage(tm) Enterprise Edition 7.5.1A
Copyright (c) 2004, 1997-2004 Ascential Software Corporation.
All Rights Reserved

##I TOSH 000002 13:44:57(001) <main_program> orchgeneral: loaded
##I TOSH 000002 13:44:57(002) <main_program> orchsort: loaded
##I TOSH 000002 13:44:57(003) <main_program> orchstats: loaded
##I TFSC 000001 13:44:57(006) <main_program> APT configuration file: /opt/etl/Ascential/DataStage/Configurations/4NodeConfig.apt
##W TCOS 000049 13:44:58(000) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##W TFCP 000000 13:44:58(002) <transform> Error when checking composite operator: The number of reject datasets "0" is less than the number of input datasets "1".
##E TFCP 000003 13:44:58(003) <transform> Error when checking composite operator: Unexpected tokens: string; [line 6, character 38].
##E TFCP 000038 13:44:58(004) <transform> Error when checking composite operator: Expected semi-colon; [line 6, character 44].
##E TFCP 000012 13:44:58(005) <transform> Error when checking composite operator: Invalid local variable declaration: string; [line 6, character 63].
##E TFCP 000038 13:44:58(006) <transform> Error when checking composite operator: Expected semi-colon; [line 6, character 77].
##E TFCP 000038 13:44:58(007) <transform> Error when checking composite operator: Expected semi-colon; [line 9, character -1].
##E TFCP 000038 13:44:58(008) <transform> Error when checking composite operator: Expected semi-colon; [line 9, character 11].
##E TFCP 000038 13:44:58(009) <transform> Error when checking composite operator: Expected semi-colon; [line 10, character 0].
##E TFCP 000038 13:44:58(010) <transform> Error when checking composite operator: Expected semi-colon; [line 10, character 13].
##E TFSR 000019 13:44:58(011) <main_program> Could not check all operators because of previous error(s)
##E TCOS 000029 13:44:58(012) <main_program> Creation of a step finished with status = FAILED. (TestRoutineJob.Transformer_21)

*** Internal Generated Transformer Code follows:
0001: //
0002: // Generated file to implement the V1S1_TestRoutineJob_Transformer_21 transform operator.
0003: //
0004:
0005: // define external functions used
0006: extern string stripinvalidchars(string string,string termstring,string replace);
0007:
0008: // define our input/output link names
0009: inputname 0 DSLink2;
0010: outputname 0 DSLink4;
0011:
0012: initialize {
0013: // define our row rejected variable
0014: int8 RowRejected0;
0015:
0016: // define our null set variable
0017: int8 NullSetVar0;
0018:
0019: // declare our intermediate variables for this section
0020: string InterVar0_0;
0021: string InterVar0_1;
0022: string InterVar0_2;
0023:
0024: // initialise constant values which require conversion
0025: InterVar0_0 = "Rogues$";
0026: InterVar0_1 = "$#";
0027: InterVar0_2 = "_";
0028: // Stage variable declaration and initialisation
0029: string[20] StageVar0_Hi;
0030: StageVar0_Hi = "";
0031: }
0032:
0033: mainloop {
0034: // initialise our row rejected variable
0035: RowRejected0 = 1;
0036:
0037: // declare our intermediate variables for this section
0038: string InterVar0_3;
0039:
0040: // evaluate the stage variables first
0041: InterVar0_3 = stripinvalidchars(InterVar0_0 , InterVar0_1 , InterVar0_2);
0042: StageVar0_Hi = InterVar0_3;
0043:
0044: // evaluate columns (no constraints) for link: DSLink4
0045: DSLink4.Name = StageVar0_Hi;
0046: writerecord 0;
0047: RowRejected0 = 0;
0048: }
0049:
0050: finish {
0051: }
0052:
*** End of Internal Generated Transformer Code

I have called the Routine in a stage variable named "Hi", and then passing this to the output field "Name".

I am unable to figure out what exactly the issue is over here. What are the unexpected tokens that the error code is talking about? And what does it mean by "Expected semi-colon" and "Invalid local variable declaration: string" ?

Is there something that I have to do other than linking the routine to the shared object, to get the routine up and running?

Any help regarding this would be really appreciated.

Thanks in advance.

Regards,
The Bird.

ameyvaidya · Post by **ameyvaidya** » Thu Oct 06, 2005 1:09 pm

One possible issue may be this line in the transformer output:

0006: extern string stripinvalidchars(string string,string termstring,string replace);

Try to rename the argument name from "string to something else("inpstr" perhaps).

HTH
Amey Vaidya

Eric · Post by **Eric** » Fri Oct 07, 2005 7:08 am

You cannot run PX jobs (incuding compiling transformers and PX routines) under "Ascential DataStage(tm) Enterprise Edition 7.5.1A " with the DataStage Server installed on to windows platform.

If you are running the server on a unix platform please specify which one as any help for PX routine compilation is platform dependent.

pavanns · Post by **pavanns** » Sat Oct 08, 2005 7:41 pm

i agree px jobs can be developed on windows but can be run only on the unix server..pl sref to the PARjDEV doc for further inf abt this..

thebird · Post by **thebird** » Sun Oct 09, 2005 4:55 am

Hi Eric,

THe server is indeed running on a Unix platform. It is HP-UX.

As Amey, had mentioned in the post above, by renaming the variable string to someother name,I was able to successfully compile the job, but when running, it is throwing the error -

Transformer_21: Failed to load the library "V1S1_CopyOfTestRoutineJob_Transformer_21.sl"; either the directory containing the library file
is not on the library search path, or the library was compiled on a system
that is incompatible with this system: Could not load "V1S1_CopyOfTestRoutineJob_Transformer_21": Unresolved external.

I have searched the forum, but could not find a solution to this.

I have compiled the C code using the compiler mentioned in $APT_COMPILER. The $SHLIB_PATH is pointing to the directory containing the object file.

a) Is there any options that I have to give when compiling the code?

b) Is there anything else that has to be taken care of?

Eric wrote:You cannot run PX jobs (incuding compiling transformers and PX routines) under "Ascential DataStage(tm) Enterprise Edition 7.5.1A " with the DataStage Server installed on to windows platform.

If you are running the server on a unix platform please specify which one as any help for PX routine compilation is platform dependent.

Thanks in advance.

Regards,
The Bird.

Eric · Post by **Eric** » Tue Oct 11, 2005 6:36 am

Have you looked in the "samples\TrxExternalFunctions\HPUX" folder in your client install media?

This gives an example which include a makefile with the correct compiler options. You can then check that you are compiling correctly.

Then ensure you re-compile your job using the FORCE compile option. This makes sure that the transformer library is re-generated.

thebird · Post by **thebird** » Fri Oct 14, 2005 8:56 am

Hi,

I modified the code a little for my routine as -

Code: Select all

#include "cstring"
#include "cstdio"
#include "cctype"
#include "cstdlib"

#define PTR_INIT (char *) 1

char *stripinvalidchars(char *inpstr, char *termstring, char *replace)
{
	int i, termlen, stringlen;
	char *ptr = PTR_INIT;
	char *dup_string;
	
	stringlen = strlen(inpstr);
	termlen =  strlen(termstring);
	dup_string = (char *) malloc(stringlen+1);
	
	strcpy(dup_string, inpstr);

	for(i = 0; i < termlen ; i++) {
		while(ptr) {
			if((ptr = strchr(dup_string, (int) termstring[i]))) 
				*ptr = *replace;
		}
		ptr = PTR_INIT;
	}
	return dup_string;
}

When I tried to run the test job, again this job aborted with a fatal error -

Transformer_21: Failed to load the library "V1S1_CopyOfTestRoutineJob_Transformer_21.sl"; either the directory containing the library file
is not on the library search path, or the library was compiled on a system
that is incompatible with this system: Could not load "V1S1_CopyOfTestRoutineJob_Transformer_21": Unresolved external.

My server is HP-UX and the client OS that I work on is Windows XP Pro.

The $SHLIB_PATH points to : /scripts/customer/offshore/customfunctions/objectcode

and this is where I have my object file.

The $APT_COMPILER points to: /opt/aCC/bin/aCC, and it is using this that I compiled my code and created the shared object.

While compiling I had set the compile option as set in $APT_COMPILEOPT, which -O -c -ext -z +Z.

After doing all this, when I create the routine and use this in a job, I get the above mentioned fatal error.

In the routine, I have set the librarypath to point to my objectfile and have mention the object type as "Object".

I thought that this was enough, but obviously not.

Is there anything else that I have to do, to get the routine up and working?

As there is not much documentation given regarding this, I am at a loss.

Thanks in advance.

Regards,

The Bird.

DSXchange

Creating a Parallel Routine.

Creating a Parallel Routine.

Modified the code, but still giving error.