Hi,
I just wondered how many stage variage can we have in a single transformer.I have almost 75 stage variable need to use?
Thanks,
Gowri Shankar H
Is there a maximum number of stage variables?
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 42
- Joined: Wed Dec 26, 2012 1:13 pm
-
- Participant
- Posts: 246
- Joined: Mon Jun 30, 2008 3:22 am
- Location: New York
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 42
- Joined: Wed Dec 26, 2012 1:13 pm
Since transform stages are converted into c++ programs in parallel jobs. A quick look at the standards didn't show a limit, but often the symbol table will overflow when too many objects are used - but this is probably an unsigned long integer so you've got a lot of headroom.
Most likely the DataStage designer front-end will have a limit on the number of entries you can have in the stage variable box, but as others have stated that they've used 100s you shouldn't reach that limit. If you do, it might be worth thinking about either the approach or in using a custom build-op instead.
Since the stage variables are generally simple assignments and not complex or lengthy loops, the performance impact is going to be small - but that deepends upon how you are assigning and using the values. Doing a "SVtemp = In.ColumnA" is going to be much faster than a long expression with type conversions and the like.
Most likely the DataStage designer front-end will have a limit on the number of entries you can have in the stage variable box, but as others have stated that they've used 100s you shouldn't reach that limit. If you do, it might be worth thinking about either the approach or in using a custom build-op instead.
Since the stage variables are generally simple assignments and not complex or lengthy loops, the performance impact is going to be small - but that deepends upon how you are assigning and using the values. Doing a "SVtemp = In.ColumnA" is going to be much faster than a long expression with type conversions and the like.
Last edited by ArndW on Tue Aug 27, 2013 12:39 am, edited 1 time in total.
-
- Participant
- Posts: 8
- Joined: Wed Jul 10, 2013 11:30 pm
- Location: Nagpur
Components Vs Backend Code Vs Performance
Hi,
In the above post you informed us that the trasformer is converted into C++ code in parallel job.
I really very anxious to know about following:
# As per my knowledge Data Stage on the back end converts the job components in to Orchestrate & C++ code for processing.
So is here, logic in trasformer will get converted in to C++ code.
# Are not all components get converted in to C++ & Orchestrate lang?
# If not which components only get converted into C++?
# Are the components which gets converted in to C++ expensive, in terms of performance?
Flashing light in to these points will be really helpful!!!
In the above post you informed us that the trasformer is converted into C++ code in parallel job.
I really very anxious to know about following:
# As per my knowledge Data Stage on the back end converts the job components in to Orchestrate & C++ code for processing.
So is here, logic in trasformer will get converted in to C++ code.
# Are not all components get converted in to C++ & Orchestrate lang?
# If not which components only get converted into C++?
# Are the components which gets converted in to C++ expensive, in terms of performance?
Flashing light in to these points will be really helpful!!!
Best Regards,
Mayur
Mayur
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The entire Transformer stage is rewritten in C++ at compile time, and this code is compiled and linked into a library/object (according to current settings).
The performance hit is taken at compile time, not at run time. You may notice that a job with a Transformer stage will take longer to compile than a job with none. If a job is changed, but its Transformer stages are unchanged, then recompiling does not recompile the Transformer stages, in the interests of efficiency (unless Force Compile is specified).
Generated C++ source code for job number nnn can be found in the project subdirectory RT_SCnnn along with osh scripts for executing the compiled version.
The performance hit is taken at compile time, not at run time. You may notice that a job with a Transformer stage will take longer to compile than a job with none. If a job is changed, but its Transformer stages are unchanged, then recompiling does not recompile the Transformer stages, in the interests of efficiency (unless Force Compile is specified).
Generated C++ source code for job number nnn can be found in the project subdirectory RT_SCnnn along with osh scripts for executing the compiled version.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.