Problem with compilation of ds parallel routine on Windows.

Minhajuddin · Post by **Minhajuddin** » Mon Dec 17, 2007 7:25 am

Hi all,

I have created a small datastage parallel routine following the instructions in the blog posted by Josh <a href = "http://blogs.ittoolbox.com/dw/soa/archi ... easy-20926"> [link] </a>

But, I am unable to compile the job. I am clueless as to what's going on with datastage. Here are the details of the parallel routine:

code for the routine(stored in a file called objTest2.cpp):

Code: Select all

#include <stdlib.h> 
#include <stdio.h> 

char * ObjTestOne() 
{ 
char* OutStr; 
OutStr="Hello World - Object Testing"; 
return OutStr; 
}

commands used to setup the environment for building the object:

Code: Select all

@echo off
call "%VS71COMNTOOLS%vsvars32.bat"

SET APT_OPERATOR_REGISTRY_PATH=D:\Minhaj_Test\osh\

SET APT_LINKOPT= -dll -base:0x50000000
SET APT_COMPILEOPT=-TP -EHa -DAPT_USE_ANSI_IOSTREAMS -MD -DNDEBUG -DIDM_LITTLE_ENDIAN -c
SET APT_LINKER=cxx

SET APT_ORCHHOME=D:\IBM\InformationServer\Server\PXEngine

SET APT_COMPILER=cxx
SET APT_CONFIG_FILE=D:\IBM\InformationServer\Server\Configurations\default.apt

SET PATH=%PATH%;D:\IBM\InformationServer\Server\PXEngine\bin;.;

set INCLUDE=D:\Program Files\Microsoft Visual Studio .NET 2003\VC7\PlatformSDK\include;D:\IBM\InformationServer\Server\PXEngine\include;c:\Program Files\MKS Toolkit\include;D:\Program Files\Microsoft Visual Studio .NET 2003\VC7\ATLMFC\INCLUDE;D:\Program Files\Microsoft Visual Studio .NET 2003\VC7\INCLUDE;D:\Program Files\Microsoft Visual Studio .NET 2003\VC7\PlatformSDK\include\prerelease;D:\Program Files\Microsoft.NET\SDK\v1.1\include;D:\Program Files\Microsoft.NET\SDK\v1.1\include\;
set LIB=%LIB%;D:\IBM\InformationServer\Server\PXEngine\lib;c:\Program Files\MKS Toolkit\lib;

commands to create the object file (*.obj):

Code: Select all

cl %APT_COMPILEOPT% objTest2.cpp

This gave me the ouptut file objTest.obj.
Now to I created a parallel routine, called objTest2 with the following settings:

Code: Select all

Routine Name: objTest2
Type: External Function
Object Type: Object
Return type: char*
Library path: D:\Minhaj_Test\osh\objTest2.obj

I have created a simple job to test this.
Row_generator---> Transformer ---> peek stage

When I try to compile this, I am getting this warning:

Code: Select all

Output from transformer compilation follows:

##I IIS-DSEE-TFCN-00001 08:25:06(000) <main_program> 
IBM WebSphere DataStage Enterprise Edition 8.0.1.4458 
Copyright (c) 2001, 2005-2007 IBM Corporation. All rights reserved
 


##I IIS-DSEE-TOSH-00002 08:25:06(001) <main_program> orchgeneral: loaded
##I IIS-DSEE-TOSH-00002 08:25:06(002) <main_program> orchsort: loaded
##I IIS-DSEE-TOSH-00002 08:25:06(003) <main_program> orchstats: loaded
##W IIS-DSEE-TOSH-00049 08:25:06(006) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##I IIS-DSEE-TCOS-00021 08:25:06(007) <main_program> Echo:
transform -inputschema record
(
  test_column:string[max=1];
  inRec:*;
)
-outputschema record
(
  test_column:string[max=1];
  test_col_Routine:string[max=10];
  outRec:*;
)
-reject
-expressionfile RT_SC104/V0S2_test_parallel_routine_Transformer_2.trx
-flag compile
-staticobj 'D:/Minhaj_Test/osh/objTest2.obj'
-compiler 'cxx'
-compileopt '-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t-'
-linker 'cxx'
-linkopt '-s -W/dll -W/base:0x50000000 -W/Zc:wchar_t-'
-name V0S2_test_parallel_routine_Transformer_2
-dir d:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O
.
##W IIS-DSEE-TOSH-00049 08:25:06(008) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##I IIS-DSEE-TCOS-00022 08:25:06(010) <main_program> Explanation:
Step has 1 operator.
???, invoked with args: -inputschema record ( test_column: string[max=1];  inRec: *; ) -outputschema record ( test_column: string[max=1];  test_col_Routine: string[max=10];  outRec: *; ) -reject -expressionfile RT_SC104/V0S2_test_parallel_routine_Transformer_2.trx -flag compile -staticobj D:/Minhaj_Test/osh/objTest2.obj -compiler cxx -compileopt -W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t- -linker cxx -linkopt -s -W/dll -W/base:0x50000000 -W/Zc:wchar_t- -name V0S2_test_parallel_routine_Transformer_2 -dir d:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O 

Step has 0 data entities.
.
##I IIS-DSEE-TCOS-00023 08:25:06(011) <main_program> Dump:
{ 
  text="transform -inputschema record ( test_column: string[max=1];  inRec: *; )\r\n-outputschema record ( test_column: string[max=1];  test_col_Routine: string[max=10];  outRec: *; )\r\n-reject\r\n-expressionfile RT_SC104/V0S2_test_parallel_routine_Transformer_2.trx\r\n-flag compile\r\n-staticobj 'D:/Minhaj_Test/osh/objTest2.obj'\r\n-compiler 'cxx'\r\n-compileopt '-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t-'\r\n-linker 'cxx'\r\n-linkopt '-s -W/dll -W/base:0x50000000 -W/Zc:wchar_t-'\r\n-name V0S2_test_parallel_routine_Transformer_2\r\n-dir d:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O", 
  line=1, column=1, name="", qualname="", 
  op={ 
       text="transform -inputschema record ( test_column: string[max=1];  inRec: *; )\r\n-outputschema record ( test_column: string[max=1];  test_col_Routine: string[max=10];  outRec: *; )\r\n-reject\r\n-expressionfile RT_SC104/V0S2_test_parallel_routine_Transformer_2.trx\r\n-flag compile\r\n-staticobj 'D:/Minhaj_Test/osh/objTest2.obj'\r\n-compiler 'cxx'\r\n-compileopt '-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t-'\r\n-linker 'cxx'\r\n-linkopt '-s -W/dll -W/base:0x50000000 -W/Zc:wchar_t-'\r\n-name V0S2_test_parallel_routine_Transformer_2\r\n-dir d:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O", 
       line=1, column=1, name=transform, qualname=transform, 
       wrapout={},
       wrapperfile=transform, kind=non_wrapper_cdi_op, exec_mode=none, 
       args="'record ( test_column: string[max=1];  inRec: *; )'-outputschema'record ( test_column: string[max=1];  test_col_Routine: string[max=10];  outRec: *; )'-reject'-expressionfile'RT_SC104/V0S2_test_parallel_routine_Transformer_2.trx'-flag'compile'-staticobj'D:/Minhaj_Test/osh/objTest2.obj'-compiler'cxx'-compileopt'-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t-'-linker'cxx'-linkopt'-s -W/dll -W/base:0x50000000 -W/Zc:wchar_t-'-name'V0S2_test_parallel_routine_Transformer_2'-dir'd:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O'"
     }
}
.
##E IIS-DSEE-TBLD-00076 08:25:09(000) <main_program> Error when checking composite operator: Subprocess command failed with exit status 40,192.
##E IIS-DSEE-TFSR-00019 08:25:09(001) <main_program> Could not check all operators because of previous error(s)
##I IIS-DSEE-TBLD-00000 08:25:09(002) <main_program> Error when checking composite operator: Output from subprocess: C:/PROGRA~1/MKSTOO~1/include\nutc/stdarg.h(55) : warning C4005: '_APALIGN' : macro redefinition
        C:/Program Files/Microsoft Visual Studio 8/VC/include\vadefs.h(84) : see previous definition of '_APALIGN'
C:/PROGRA~1/MKSTOO~1/include\nutc/stdarg.h(61) : warning C4005: '_INTSIZEOF' : macro redefinition
        C:/Program Files/Microsoft Visual Studio 8/VC/include\vadefs.h(101) : see previous definition of '_INTSIZEOF'
D:\IBM\InformationServer\Server\PXEngine\include\apt_util/keylookup.h(1148) : war
##I IIS-DSEE-TBLD-00000 08:25:09(003) <main_program> Error when checking composite operator: Output from subprocess: ning C4251: 'APT_KeyLookupRange::rangeOptions_' : class 'std::vector<_Ty>' needs to have dll-interface to be used by clients of class 'APT_KeyLookupRange'
        with
        [
            _Ty=APT_KeyLookupRange::rangeOption
        ]
D:\IBM\InformationServer\Server\Projects\MATCH_ENGINE\RT_BP104.O\V0S2_test_parallel_routine_Transformer_2.C(315) : warning C4244: '=' : conversion from 'APT_Int64' to 'int', possible loss of data

##I IIS-DSEE-TBLD-00079 08:25:09(004) <transform> Error when checking composite operator: cxx    -Ld:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O/ -LD:/IBM/InformationServer/Server/PXEngine/lib -LD:/IBM/InformationServer/Server/PXEngine/user_lib -s -W/dll -W/base:0x50000000 -W/Zc:wchar_t- -lliborchnt -lliborchcorent -lliborchbuildopnt D:/Minhaj_Test/osh/objTest2.obj.o D:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O/V0S2_test_parallel_routine_Transformer_2.tmp.o -o D:/IBM/InformationServer/Server/Projects/MATCH_ENGINE/RT_BP104.O/V0S2_test_parallel_routine_Transformer_2.dll.
##I IIS-DSEE-TBLD-00000 08:25:09(005) <main_program> Error when checking composite operator: Output from subprocess: LINK : fatal error LNK1181: cannot open input file 'D:\Minhaj_Test\osh\objTest2.obj.o'

##E IIS-DSEE-TCOS-00029 08:25:09(006) <main_program> Creation of a step finished with status = FAILED. (test_parallel_routine.Transformer_2)

*** Internal Generated Transformer Code follows:
0001: //
0002: // Generated file to implement the V0S2_test_parallel_routine_Transformer_2 transform operator.
0003: //
0004: 
0005: // define external functions used
0006: extern string ObjTestOne();
0007: 
0008: // define our input/output link names
0009: inputname 0 DSLink3;
0010: outputname 0 DSLink4;
0011: 
0012: initialize {
0013: 	// define our row rejected variable
0014: 	int8 RowRejected0;
0015: 
0016: 	// define our null set variable
0017: 	int8 NullSetVar0;
0018: 
0019: }
0020: 
0021: mainloop {
0022: 	// initialise our row rejected variable
0023: 	RowRejected0 = 1;
0024: 
0025: 	// declare our intermediate variables for this section
0026: 	string InterVar0_0;
0027: 
0028: 	// evaluate columns (no constraints) for link: DSLink4
0029: 	InterVar0_0 = ObjTestOne();
0030: 	DSLink4.test_col_Routine = InterVar0_0;
0031: 	writerecord 0;
0032: 	RowRejected0 = 0;
0033: }
0034: 
0035: finish {
0036: }
0037: 
*** End of Internal Generated Transformer Code

Any help is greatly appreciated!

Thank you.

Minhajuddin · Post by **Minhajuddin** » Mon Dec 17, 2007 10:35 am

I have gone through a lot of posts, But I am unable to fix this

Most of them deal about issues on Unix

ray.wurlod · Post by **ray.wurlod** » Mon Dec 17, 2007 1:59 pm

Try creating a routine with at least one input argument. DataStage expects to pass at least one value to a routine and get something back.

lstsaur · Post by **lstsaur** » Mon Dec 17, 2007 2:49 pm

Double check your External Subroutine Name in your parallel routine's properties. Did you have "ObjTestOne" in there? Also, you have "D:\Minhaj_Test\osh\objTest2.obj " in your Library path, but DS couldn't open D:\Minhaj_Test\osh\objTest2.obj.o.

JoshGeorge · Post by **JoshGeorge** » Mon Dec 17, 2007 9:51 pm

This is not mandatory. A parallel routine can be without any input parameters.

ray.wurlod wrote:Try creating a routine with at least one input argument. DataStage expects to pass at least one value to a routine and get something back. ...

AmeyJoshi14 · Post by **AmeyJoshi14** » Tue Dec 18, 2007 4:50 am

Hi,
Can you post the values for following parameter:
$APT_COMPILE
$APT_COMPILEOPT.

The syntax for the compliation of .c file is
for eg:
$APT_COMPLIE $APT_COMPILEOPT some.c some.o

Minhajuddin · Post by **Minhajuddin** » Tue Dec 18, 2007 5:33 am

@lstsaur:

The name of my the object file is D:\Minhaj_Test\osh\objTest2.obj

@All
These are the environment variable settings in the administrator for the compiler.

Code: Select all

APT_COMPILEOPT	-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t-
APT_COMPILER	cxx
APT_LINKER	cxx
APT_LINKOPT	-s -W/dll -W/base:0x50000000 -W/Zc:wchar_t-

Minhajuddin · Post by **Minhajuddin** » Tue Dec 18, 2007 5:56 am

AmeyJoshi14 wrote:Hi,
Can you post the values for following parameter:
$APT_COMPILE
$APT_COMPILEOPT.

The syntax for the compliation of .c file is
for eg:
$APT_COMPLIE $APT_COMPILEOPT some.c some.o

I tried with

Code: Select all

D:\Minhaj_Test\osh>cxx -W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c -W/Zc:wchar_t- O
bjTest2.cpp
cxx: file 'C:/PROGRA~1/MKSTOO~1/etc/cxx.ccg': The system cannot find the file sp
ecified.


D:\Minhaj_Test\osh>

I think cl.exe is the compiler for Windows.
I am lost