Parallel Extender TransformerStage Issue

[Veni]

Do we have any limitation on writing expressions in Parallel Extender Transformation Stage ,because when i am using 300 colums with same expressions my job is working fine and compliation is also fast but when i am using all 1220 colums in single transformer , its throwing lot of errors -(Error when checking composite operator :Function Decimal_from_String)

Please help me in understanding the issue.

[salil]

Its not clear whether u have tested the whole derivations in chunks of 300 or only some of it to conclude that -
The derivetions used are same through out every column.
which functions might have thrown the error.

Can u pls paste the full output from the transformer compiler error log..
That may help in understanding more on what really is generated.

[Veni]

When i try with the 300 fields , its working fine
but when i am trying to use the same expression with 1220 filed in a singletransformer .I am getting this error

The expression which i used is (if trim(field_Name)='.' then SetNull() else
StringtoDecimal(field_Name)
--------------------------------------------------------
Output from transformer compilation follows:

Output from transformer compilation follows:

##I TFCN 000001 09:45:29(000) <main_program>
Ascential DataStage(tm) Enterprise Edition 7.5.1
Copyright (c) 2004, 1997-2004 Ascential Software Corporation.
All Rights Reserved


##I TUTL 000031 09:45:29(001) <main_program> The open files limit is 100; raising to 10000.
##I TOSH 000002 09:45:30(000) <main_program> orchgeneral: loaded
##I TOSH 000002 09:45:30(001) <main_program> orchsort: loaded
##I TOSH 000002 09:45:30(002) <main_program> orchstats: loaded
##I TFSC 000001 09:45:30(005) <main_program> APT configuration file: /opt/tools/datastage7/Ascential/DataStage/Configurations/af2-1Node.apt
##W TFPA 000017 09:45:30(006) <main_program> Warning: the value of the PWD environment variable (/opt/tools/datastage7/Ascential/DataStage/DSEngine) does not appear to be a synonym for the current working directory (/data/dw/af2/projectlib/dsp_af2_dev). The current working directory will be used, but if your ORCHESTRATE job does not start up correctly, you should set your PWD environment variable to a value that will work on all nodes of your system.
##W TCOS 000049 09:45:33(000) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##E TBLD 000000 10:30:29(000) <main_program> Error when checking composite operator: Subprocess command failed with exit status 256.
##E TFSR 000019 10:30:29(001) <main_program> Could not check all operators because of previous error(s)
##W TFCP 000000 10:30:29(002) <transform> Error when checking composite operator: The number of reject datasets "0" is less than the number of input datasets "1".
##W TFCP 000048 10:30:29(003) <transform> Error when checking composite operator: Function decimal_from_string, [line 103, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(004) <transform> Error when checking composite operator: Function decimal_from_string, [line 111, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(005) <transform> Error when checking composite operator: Function decimal_from_string, [line 119, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(006) <transform> Error when checking composite operator: Function decimal_from_string, [line 127, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(007) <transform> Error when checking composite operator: Function decimal_from_string, [line 135, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(008) <transform> Error when checking composite operator: Function decimal_from_string, [line 143, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(009) <transform> Error when checking composite operator: Function decimal_from_string, [line 151, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(010) <transform> Error when checking composite operator: Function decimal_from_string, [line 159, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(011) <transform> Error when checking composite operator: Function decimal_from_string, [line 167, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(012) <transform> Error when checking composite operator: Function decimal_from_string, [line 175, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(013) <transform> Error when checking composite operator: Function decimal_from_string, [line 183, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(014) <transform> Error when checking composite operator: Function decimal_from_string, [line 191, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(015) <transform> Error when checking composite operator: Function decimal_from_string, [line 199, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(016) <transform> Error when checking composite operator: Function decimal_from_string, [line 207, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(017) <transform> Error when checking composite operator: Function decimal_from_string, [line 215, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(018) <transform> Error when checking composite operator: Function decimal_from_string, [line 223, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(019) <transform> Error when checking composite operator: Function decimal_from_string, [line 231, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(020) <transform> Error when checking composite operator: Function decimal_from_string, [line 239, character 16], returns a decimal; default precision is used.
##W TFCP 000048 10:30:29(021) <transform> Error when checking composite operator: Function de

[salil]

We'll go on iteratively:
In the metadata of all output columns(for decimal),ensure:
datatype:decimal
nullable:yes
precision&scale>0

[Veni]

In the metadata of all output columns those who are decimals with precision&scale=0 DataStage has converted automatically those in Integer by default and other those are Decimal with precision&scale >0
thats is in Decimal format.

Is this is the problem with Meta data ?

Thanks in advance

[salil]

I dont think datastage by itself changes the type of data,apart from considering a decimal without precision as integer on the fly.
pls check whether any of the derivations have syntax prob(red mark),like any braces not closed or so.
Minor mistakes like that can happen as the no. of fields are more..

[salil]

I dont think datastage by itself changes the type of data,apart from considering a decimal without precision as integer on the fly.
pls check whether any of the derivations have syntax prob(red mark),like any braces not closed or so.
Minor mistakes like that can happen as the no. of fields are more..

I hope ur issue is resolved,if not pls paste the 3-4 lines before the line
" *** Internal Generated Transformer Code follows:"
so that we'll come to know which properties have been violated..

[Veni]

Still i am facing same problem and i check in my transformer all the expressions are correct (no red mark)

[salil]

Still i am facing same problem and i check in my transformer all the expressions are correct (no red mark)

I tried and generated the same error on my system,as per my conclusion its bcos:
" Trying to setnull() a column i.e not nullable."Pls confirm that everycolumns are nullable:'yes'
And the compiled log says it specifically

Code: Select all

##I TUTL 000031 17:31:11(001) <main_program> The open files limit is 100; raising to 8192.
##I TOSH 000002 17:31:11(002) <main_program> orchgeneral: loaded
##I TOSH 000002 17:31:11(003) <main_program> orchsort: loaded
##I TOSH 000002 17:31:11(004) <main_program> orchstats: loaded
##I TFSC 000001 17:31:11(007) <main_program> APT configuration file: /gmodc03/dsadm/Ascential/DataStage/Configurations/default.apt
##W TFPA 000017 17:31:11(008) <main_program> Warning: the value of the PWD environment variable (/gmodc03/dsadm/Ascential/DataStage/DSEngine/bin) does not appear to be a synonym for the current working directory (/gmodc03/dsadm/Ascential/DataStage/Projects/IARPROJ).  The current working directory will be used, but if your ORCHESTRATE job does not start up correctly, you should set your PWD environment variable to a value that will work on all nodes of your system.
##W TCOS 000049 17:31:12(000) <main_program> Parameter specified but not used in flow: DSPXWorkingDir
##W TFCP 000000 17:31:12(002) <transform> Error when checking composite operator: The number of reject datasets "0" is less than the number of input datasets "1".
##W TFCP 000048 17:31:12(003) <transform> Error when checking composite operator:  Function decimal_from_string, [line 36, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(004) <transform> Error when checking composite operator:  Function decimal_from_string, [line 44, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(005) <transform> Error when checking composite operator:  Function decimal_from_string, [line 52, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(006) <transform> Error when checking composite operator:  Function decimal_from_string, [line 60, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(007) <transform> Error when checking composite operator:  Function decimal_from_string, [line 68, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(008) <transform> Error when checking composite operator:  Function decimal_from_string, [line 76, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(009) <transform> Error when checking composite operator:  Function decimal_from_string, [line 84, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(010) <transform> Error when checking composite operator:  Function decimal_from_string, [line 92, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(011) <transform> Error when checking composite operator:  Function decimal_from_string, [line 100, character 16], returns a decimal; default precision is used.
##W TFCP 000048 17:31:12(012) <transform> Error when checking composite operator:  Function decimal_from_string, [line 108, character 16], returns a decimal; default precision is used.
##W TFCP 000000 17:31:12(013) <transform> Error when checking composite operator: Implicit conversion from source type "Decimal" to result type "SFloat".
##E TFCP 000026 17:31:12(014) <transform> Error when checking composite operator: Setting null to this non-nullable field: i.
##E TFSR 000019 17:31:12(015) <main_program> Could not check all operators because of previous error(s)
##E TCOS 000029 17:31:12(016) <main_program> Creation of a step finished with status = FAILED. (abc.Transformer_1)

*** Internal Generated Transformer Code follows:

[/code][/quote]

[Veni]

After making all Nullable ='Yes' also i am getting same compilation error.

Please suggest me .

[salil]

There's no single step resolution for these issues.what u can probably try next is to make copies of ur existing job so that it has less no. of derivations,say 300 cols * 4 jobs (=1220 cols),so that all the columns are there.If all of these compiles individually,we can suspect other issues,and if it doesn't we'll be able to isolate the problem area.