Getting error compiling transformer in PX Job

sunRoy68 · Post by **sunRoy68** » Wed Jul 30, 2008 3:29 am

Hi to all,
I'm having problems compiling transformer in a PX Job.

The output link is very wide (562 columns!!!) ... does it depend from this or something else?

PLEASE HELP!

Output from transformer compilation follows:

##I TFCN 000001 10:38:14(000) <main_program>
Ascential DataStage(tm) Enterprise Edition 7.5.2
Copyright (c) 2004, 1997-2004 Ascential Software Corporation.
All Rights Reserved

##I TUTL 000031 10:38:14(001) <main_program> The open files limit is 2048; raising to 4096.
##I TOSH 000002 10:38:14(002) <main_program> orchgeneral: loaded
##I TOSH 000002 10:38:14(003) <main_program> orchsort: loaded
##I TOSH 000002 10:38:15(000) <main_program> orchstats: loaded
##I TFSC 000001 10:38:15(003) <main_program> APT configuration file: /software/dsadm/Ascential/DataStage/Configurations/default.apt
##I TUTL 000031 10:33:22(000) <node_node2> The open files limit is 2048; raising to 4096.
##W TCOS 000049 10:38:17(000) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##E TBLD 000000 10:39:20(000) <main_program> Error when checking composite operator: Subprocess command failed with exit status 768.
##E TFSR 000019 10:39:20(001) <main_program> Could not check all operators because of previous error(s)
##W TFCP 000000 10:39:20(002) <transform> Error when checking composite operator: The number of reject datasets "0" is less than the number of input datasets "1".
##W TFCP 000025 10:39:20(003) <transform> Error when checking composite operator: Converting string to number.
##W TFCP 000000 10:39:20(004) <transform> Error when checking composite operator: Implicit conversion from source type "String" to result type "Int32".
##W TFCP 000025 10:39:20(005) <transform> Error when checking composite operator: Converting string to decimal.
##W TFCP 000000 10:39:20(006) <transform> Error when checking composite operator: Implicit conversion from source type "String" to result type "Decimal".
##W TFCP 000025 10:39:20(007) <transform> Error when checking composite operator: Possible range limitation.
##W TFCP 000000 10:39:20(008) <transform> Error when checking composite operator: Implicit conversion from source type "Int32" to result type "Decimal".
##W TFCP 000025 10:39:20(009) <transform> Error when checking composite operator: Converting string to number.
##W TFCP 000000 10:39:20(010) <transform> Error when checking composite operator: Implicit conversion from source type "String" to result type "Int32".
##W TFCP 000025 10:39:20(011) <transform> Error when checking composite operator: Converting string to number.
##W TFCP 000000 10:39:20(012) <transform> Error when checking composite operator: Implicit conversion from source type "String" to result type "Int64".
##W TFCP 000025 10:39:20(013) <transform> Error when checking composite operator: Converting string to number.
##W TFCP 000000 10:39:20(014) <transform> Error when checking composite operator: Implicit conversion from source type "String" to result type "Int64".
##W TBLD 000000 10:39:20(015) <main_program> Error when checking composite operator: Output from subprocess: aCC: Not enough memory is available to finish the compilation.

##I TFCP 000000 10:39:20(016) <transform> Error when checking composite operator: /opt/aCC/bin/aCC +DD64 +W890 +Z -O -I/software/dsadm/Ascential/DataStage/PXEngine/include +DD64 -O -c -ext -z +Z /data/Progetti/BASE_NELLO/RT_BP1467.O/V18S1_CopyOfTcpOdsConDhbBss020BssMonthlyDownload_TR_BSS.C -o /data/Progetti/BASE_NELLO/RT_BP1467.O/V18S1_CopyOfTcpOdsConDhbBss020BssMonthlyDownload_TR_BSS.tmp.o.
##I TBLD 000000 10:39:20(017) <main_program> Error when checking composite operator: Output from subprocess: MSTAT af
##I TBLD 000000 10:39:20(018) <main_program> Error when checking composite operator: Output from subprocess: ter OutOfMemory error Bytes total=1052760064, in use=1049078160, free=3681904
Pages after OutOfMemory error: Text=8560 Data=256046 Stack
##I TBLD 000000 10:39:20(019) <main_program> Error when checking composite operator: Output from subprocess: =244

##E TCOS 000029 10:39:20(020) <main_program> Creation of a step finished with status = FAILED. (CopyOfTcpOdsConDhbBss020BssMonthlyDownload.TR_BSS)

ArndW · Post by **ArndW** » Wed Jul 30, 2008 3:41 am

You can edit your post to remove all the details, they are not required in this case.
Most likely the code is too big for your installation's optimizer. I would recommend turning off optimization either via your ADMIN tool for all jobs, or to change the compiler options for this particular transform stage. Each UNIX has different options, but I think that "-o0" usually does the trick.

mahadev.v · Post by **mahadev.v** » Wed Jul 30, 2008 3:45 am

##W TBLD 000000 10:39:20(015) <main_program> Error when checking composite operator: Output from subprocess: aCC: Not enough memory is available to finish the compilation.

First few warnings are related to String to decimal, String to number conversions. i think these will have to be handled explicitly. Not sure of about this last warning though.[/code]

ArndW · Post by **ArndW** » Wed Jul 30, 2008 3:47 am

Ignore the other errors, the abort is due to memory. I have had to turn off optimization in large transform stages before.

sunRoy68 · Post by **sunRoy68** » Wed Jul 30, 2008 3:52 am

ArndW wrote:Ignore the other errors, the abort is due to memory. I have had to turn off optimization in large transform stages before. ...

excuse me ...what do you mean with "turn off optimization" ?!?

ArndW · Post by **ArndW** » Wed Jul 30, 2008 3:56 am

The c++ compiler optimization chews up large amounts of resources. For typical transform stage functionality the compiler's optimization won't bring that much performance improvement, so turning off or lowering the optimization level is acceptable.

sunRoy68 · Post by **sunRoy68** » Wed Jul 30, 2008 8:28 am

ArndW wrote:The c++ compiler optimization chews up large amounts of resources. For typical transform stage functionality the compiler's optimization won't bring that much performance improvement, so turning off o ...

Too much for me ... I can't make changes on the compiler.

I solved treating the columns in 4 transformers dividing the flow in "blocks". I know, I'm the first to say that it's absolutely ugly putting 4 transformers in row, but in this case I'm forced because I didn't find other solution.
I hope it's the first and last time that I have to treat a table of 562 columns!!!

Have a nice day.

sunRoy68 ^__^

ArndW · Post by **ArndW** » Wed Jul 30, 2008 9:55 am

Sunroy68 - you aren't changing the compiler, just the "compiler options". Depending upon your platform, the standard options will contain something reading "-O" or "-o" and that defaults to some level of optimization. Changing that to "-O0" or "-o0" turns off optimization.

jsrikanth · Post by **jsrikanth** » Wed Apr 29, 2009 7:24 am

I had a similar issue. I added null handling to the stage variables and the compile error disappeared!

ray.wurlod · Post by **ray.wurlod** » Wed Apr 29, 2009 5:07 pm

The Transformer stage properties page has a Build tab where you can provide replacement compiler options to override those given for the job.