i have a question. many people said notto use a parallel transformer becuase of performance issues. is this true. i am talking about the parallel xformer not basic xformer.
Another question instead of using a xformer, if i build a wrapper (custom stage) and use, will there be any performance improvement.
Replies will be appreciated.
why the performance degrades when we use a parallel xformer.(is this true)
parallel stage transformer
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 236
- Joined: Sun Apr 01, 2007 7:41 am
- Location: Michigan
The only real performance issue with a PX transform stage, in my opinion, is that it seems to take an eternity to compile; runtime performance is quite good as it is executing c++ code.
Could you cite some specific examples of cases where people state that the PX transform is slow?
PX has many stages to perform specific functions. I think it makes sense to use those stages where possible. Some things (i.e. changing a column data type) could be done either in a purpose-built stage (in this case, a MODIFY stage) or in a transform stage. I think that both will perform at similar speeds and you can test this if you wish; but it does make sense to use a MODIFY stage rather than a transform stage. If you not only need to convert a datatype but also check it's value you would need 2 modify stages - even though these might be combined at runtime it makes sense to perform this type of transformation in a transform stage, since it would only take one derivation expression to do both actions.
A specific wrapper stage to perform an action might be faster if well-written - but there really is no way to give a correct answer to this question as there are far too many variables.
Once somebody can show a performance degradation with the parallel transformer stage (degradation compared to what?) then it might be explained.
Could you cite some specific examples of cases where people state that the PX transform is slow?
PX has many stages to perform specific functions. I think it makes sense to use those stages where possible. Some things (i.e. changing a column data type) could be done either in a purpose-built stage (in this case, a MODIFY stage) or in a transform stage. I think that both will perform at similar speeds and you can test this if you wish; but it does make sense to use a MODIFY stage rather than a transform stage. If you not only need to convert a datatype but also check it's value you would need 2 modify stages - even though these might be combined at runtime it makes sense to perform this type of transformation in a transform stage, since it would only take one derivation expression to do both actions.
A specific wrapper stage to perform an action might be faster if well-written - but there really is no way to give a correct answer to this question as there are far too many variables.
Once somebody can show a performance degradation with the parallel transformer stage (degradation compared to what?) then it might be explained.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It used to be true. In the late 7.1 versions and subsequently the Transformer stage was completely re-vamped and is quite efficient - it now generates a transform operator (in osh terminology).
Yes, it takes longer to compile than other stages, but that's because more things have to happen - generation of C++ source code, compiling and linking of the result - but that means that the run-time stage is efficient. Also, if you make a change to the job but do not change the Transformer stage, recompiling the job does not re-compile the Transformer (unless you select Force Compile from the menu).
Yes, it takes longer to compile than other stages, but that's because more things have to happen - generation of C++ source code, compiling and linking of the result - but that means that the run-time stage is efficient. Also, if you make a change to the job but do not change the Transformer stage, recompiling the job does not re-compile the Transformer (unless you select Force Compile from the menu).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Charter Member
- Posts: 166
- Joined: Wed Mar 16, 2005 6:52 am
- Location: Mumbai, India
Also if you build a custom stage, the onus is on you to ensure that it is properly coded, tested and does not have any memory leaks.. etc.
Also the person who will support it afterwards better understand C++ and what exactly the stage does.
Too much work/risk IMO. Especially when there is a standard alternative available.
Also the person who will support it afterwards better understand C++ and what exactly the stage does.
Too much work/risk IMO. Especially when there is a standard alternative available.
Amey Vaidya<i>
I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.</i>
<i>- Douglas Adams</i>
I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.</i>
<i>- Douglas Adams</i>
Re: parallel stage transformer
no one will tell u that not to use transformer stage because of performance.just4u_sharath wrote:i have a question. many people said notto use a parallel transformer becuase of performance issues. is this true. i am talking about the parallel xformer not basic xformer.
Another question instead of using a xformer, if i build a wrapper (custom stage) and use, will there be any performance improvement.
Replies will be appreciated.
why the performance degrades when we use a parallel xformer.(is this true)
i agree with you....transformer has its own interface , so it decreases the perfomance
SRINIVAS
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: