DataStage Parallel Routine -- Compile Location

Tornquist · Post by **Tornquist** » Tue May 28, 2013 2:55 pm

I am working on taking a column that has a date as well as some other assorted string values and converting that into a Julian Day so that I can use the date functions within DataStage.

Right now I do not have anything more than the ability to copy files to the UNIX machine and use DataStage designer on the DataStage server. Because of this I've been compiling my C code in a UNIX environment and then copying that logic over with WinSCP.

The following is my C code:

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

char *replace_str(char *str, char *orig, char *rep);

int ReplaceToJulian(char *stringIn);

int ReplaceToJulian(char *stringIn)
{
	char *stringInput = replace_str(stringIn, "Rate Effective ","");

	char *tokenOne = strtok( stringInput, "/");
	char *tokenTwo = strtok( NULL, "/");
	char *tokenThree = strtok( NULL, "/");
	int month = (int) strtol(tokenOne, (char **)NULL, 10);
	int day = (int) strtol(tokenTwo, (char **)NULL, 10);
	int year = (int) strtol(tokenThree, (char **)NULL, 10);
	int a = (14-month)/12;
	int y = year+4800-a+2000;
	int m = month+12*a-3;
	int JDN = day + (153*m+2)/5 + 365*y + y/4 - y/100 + y/400 - 32045;
	
	return JDN;
}

char *replace_str(char *str, char *orig, char *rep)
{
  static char buffer[4096];
  char *p;

  if(!(p = strstr(str, orig)))  // Is 'orig' even in 'str'?
    return str;

  strncpy(buffer, str, p-str); // Copy characters from 'str' start to 'orig' st$
  buffer[p-str] = '\0';

  sprintf(buffer+(p-str), "%s%s", rep, p+strlen(orig));

  return buffer;
}

I've been compiling this logic with:

Code: Select all

gcc -fpic -c ReplaceToJulian.c
gcc -shared -o ProphixLib.so ReplaceToJulian.o

As you can see, I compiled both an object file and a library but I was unable to get either to work.

The error thrown from DataStage is as follows:

Code: Select all

Output from transformer compilation follows:

##I IIS-DSEE-TFCN-00001 15:36:39(000) <main_program> 
IBM InfoSphere DataStage Enterprise Edition 9.1.0.6525 
Copyright (c) 2001, 2005-2012 IBM Corporation. All rights reserved
 


##I IIS-DSEE-TFCN-00006 15:36:39(001) <main_program> conductor uname: -s=AIX; -r=1; -v=6; -n=cadapp06bao; -m=00F608DE4C00
##I IIS-DSEE-TOSH-00002 15:36:39(002) <main_program> orchgeneral: loaded
##I IIS-DSEE-TOSH-00002 15:36:39(003) <main_program> orchsort: loaded
##I IIS-DSEE-TOSH-00002 15:36:39(004) <main_program> orchstats: loaded
##W IIS-DSEE-TOSH-00049 15:36:39(007) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##E IIS-DSEE-TBLD-00076 15:37:13(000) <main_program> Error when checking composite operator: Subprocess command failed with exit status 2,048.
##E IIS-DSEE-TFSR-00019 15:37:13(001) <main_program> Could not check all operators because of previous error(s)
##W IIS-DSEE-TFTM-00012 15:37:13(002) <transform> Error when checking composite operator: The number of reject datasets "0" is less than the number of input datasets "1".
##W IIS-DSEE-TFEV-00025 15:37:13(003) <transform> Error when checking composite operator: Converting number to string.
##W IIS-DSEE-TFEV-00023 15:37:13(004) <transform> Error when checking composite operator: Implicit conversion from source type "Int32" to result type "String".
##W IIS-DSEE-TFEV-00025 15:37:13(005) <transform> Error when checking composite operator: Converting number to string.
##W IIS-DSEE-TFEV-00023 15:37:13(006) <transform> Error when checking composite operator: Implicit conversion from source type "Int32" to result type "String".
##W IIS-DSEE-TBLD-00000 15:37:13(007) <main_program> Error when checking composite operator: Output from subprocess:     1500-030: (I) INFORMATION: std::_Tree<std::_Tmap_traits<APT_UString,APT_OutputAccess
##W IIS-DSEE-TBLD-00000 15:37:13(008) <main_program> Error when checking composite operator: Output from subprocess: orBase *,std::less<APT_UString>,std::allocator<std::pair<const APT_UString,APT_OutputAccessorBase *> >,0> >::_Insert(bool, _Nodeptr, const value_type &): Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.

##W IIS-DSEE-TBLD-00000 15:37:13(009) <main_program> Error when checking composite operator: Output from subprocess: ld: 0711-715 ERROR: File /datafiles/PROPHIX/ReplaceToJulian.o cannot be processed.
	The file must be an object file, an import file, or an archive.

##I IIS-DSEE-TBLD-00079 15:37:13(010) <transform> Error when checking composite operator: /usr/vacpp/bin/xlC_r    -L/opt/IBM/InformationServer/Server/Projects/ETL/RT_BP39.O/ -L/opt/IBM/InformationServer/Server/PXEngine/lib -L/opt/IBM/InformationServer/Server/PXEngine/user_lib -G -q64 -lorchaix64 -lorchcoreaix64 -lorchbuildopaix64 /datafiles/PROPHIX/ReplaceToJulian.o /opt/IBM/InformationServer/Server/Projects/ETL/RT_BP39.O/V3S0_BurdenRate_Load_Strip_Null_Dates.tmp.o -o /opt/IBM/InformationServer/Server/Projects/ETL/RT_BP39.O/V3S0_BurdenRate_Load_Strip_Null_Dates.o.
##E IIS-DSEE-TCOS-00029 15:37:13(011) <main_program> Creation of a step finished with status = FAILED. (BurdenRate_Load.Strip_Null_Dates)

*** Internal Generated Transformer Code follows:
0001: //
0002: // Generated file to implement the V3S0_BurdenRate_Load_Strip_Null_Dates transform operator.
0003: //
0004: 
0005: // define external functions used
0006: extern int32 ReplaceToJulian(string stringIn);
0007: 
0008: // define our input/output link names
0009: inputname 0 Split_US_Dates;
0010: outputname 0 String_Dates;
0011: 
0012: initialize {
0013: 	// define our control variables
0014: 	int8 RowRejected0;
0015: 	int8 NullSetVar0;
0016: 
0017: 	// declare our intermediate variables for this section
0018: 	string InterVar0_0;
0019: 
0020: 	// initialise constant values which require conversion
0021: 	InterVar0_0 = 0;
0022: }
0023: 
0024: mainloop {
0025: 
0026: 	// initialise the rejected row variable
0027: 	RowRejected0 = 1;
0028: 
0029: 	// evaluate constraint and columns for link: String_Dates
0030: 	if ((Split_US_Dates.Date != InterVar0_0))
0031: 	{
0032: 		String_Dates.Date = ReplaceToJulian(Split_US_Dates.Date);
0033: 		writerecord 0;
0034: 		RowRejected0 = 0;
0035: 	}
0036: }
0037: 
0038: finish {
0039: }
0040: 
*** End of Internal Generated Transformer Code

Which to me looks like I need to compile the function differently. I do not have any idea what the specific difference needed is.

For the object file my setup was as follows:

Routine Name: ReplaceToJulian
External Name: ReplaceToJulian
Object type: Object
Return Type: int
Library path: /datafiles/ReplaceToJulian.o
Arguments:
name=stringIn
type=char*

Are there any suggestions on how to remedy this? Any help or direction would be much appreciated.

Thank You,

Nathan

ray.wurlod · Post by **ray.wurlod** » Tue May 28, 2013 3:23 pm

Not only compilers but also byte-order are likely to be different (I'm assuming your UNIX is not Intel-based here). As a general rule binary code compiled on one platform simply won't work on another.

Tornquist · Post by **Tornquist** » Tue May 28, 2013 3:29 pm

That's what I was afraid of. Is there any way to compile logic that will work on that machine from a different one?

ArndW · Post by **ArndW** » Wed May 29, 2013 12:51 am

No, unfortunately not. The object code generated on a UNIX system is not transportable to your Windows machine. You will need to compile on that machine or another Windows machine with the same OS version. The c++ compiler is free so that shouldn't be a problem for you.

Tornquist · Post by **Tornquist** » Wed May 29, 2013 8:10 am

I was slightly unclear. I am simply using Windows to transfer the logic between a UNIX virtual machine and the UNIX server. I understand that the compiled object is not movable between multiple systems though.

Should I get gcc installed on the server, is the rest of my logic sound? I can use C, correct? It doesn't have to be c++?

eph · Post by **eph** » Wed May 29, 2013 9:46 am

Hi,

I've seen this :

##I IIS-DSEE-TBLD-00079 15:37:13(010) <transform> Error when checking composite operator: /usr/vacpp/bin/xlC_r

which make me think that you are using a gcc compiled object on a non-gcc environment (linux=>AIX?).

Your compilator and your compilation options should match DS server ones (you can see them in the administrator). Once you've compiled with the same compilator and options, you can use your library.

You can use C instead of c++ as long as your program compiles (which was my case using a c code and g++).

Eric

jwiles · Post by **jwiles** » Wed May 29, 2013 10:06 am

You selected Windows as the operating system when you originally started the thread, thus people's answers will generally be with regards to that choice. Just be as clear as possible with accurate information when you start a new thread in the future

Why not simply use SFTP directly from one server to the other without going through a third machine?

Within WinSCP (or SFTP), verify that you are performing a BINARY transfer both from the compiling server and to the target server. I would recommend forcing that option rather than assuming it will be chosen automatically.

From the log output you provided:

Code: Select all

##I IIS-DSEE-TFCN-00006 15:36:39(001) <main_program> conductor uname: -s=AIX; -r=1; -v=6; -n=cadapp06bao; -m=00F608DE4C00

, it appears that your O/S is actually AIX. Per the IS 9.1 system requirements, available from here, you may need to be using the XL C/C++ compiler to compile your code rather than the gcc compiler (DataStage is using XL C/C++ to compile the transformers).

Regards,