Never use basic transformers!!!
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 291
- Joined: Wed Sep 26, 2007 11:23 am
- Location: Madrid, Spain
Never use basic transformers!!!
Following on my documentation study (for certification), it insists over and over again in not using basic transformer
Can anyone tell me what is a basic transformer???????
Can anyone tell me what is a basic transformer???????
A Server job Transformer stage. The concept of partitioned parallelism is defeated when you place serialized operations into it. A Server job BASIC Transformer stage will be invoked within an instance of a job passing data in and out thru pipes. It's horrifically slow, but lets you exploit existing BASIC logic and functions that you don't have access to in PX.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Sometimes you need it...and in those sometimes, a "25 mph speed limit" may be perfectly acceptable. As Ken says, you may have some great logic to exploit, making it worth doing. To say "never use it" out of hand throws away the value of whatever effective solution you may have built previously, or built more easily using uv/basic. Just use it wisely.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
I'd like to defend on the method using Basic Transformer in PX job via following observations:
1. Place Basic Transformer in a PX job won't break Partitioned Parallelism inscribed by PX engine, since it can be run in Parallel to map with Partitioned method declared in its Predecessor stage and its Successor stage. I once monitor this type of job running in Unix, and it spawns multiple physical processes in the system to fit in the parallelism requested by DataStage job.
2. I agree the performance penalty for using Basic Transformer cannot be avoided due to the high resource usage overhead(Mutilple Physcial Processes), unlike the generic PX transformer smart enough to combine the operator in runtime. However, if the data volume is not dramatically huge, and you need to use PX job to access certain useful stages(such as Join, Difference) that are not available in Server job, and you need to use some great Server functions such as IConv/OConv that are not available in PX job, then why not using this method. You won't see too much of the performance penalty and it's not a sin
3. As I was told by Ascential Advanced Engineer, the Basic Transformer is designed this way with the purpose listed in 1. and 2. I have designed some jobs fit in the condition as 2. for my client, and they works just fine with customer's satisfaction. As a matter of fact, my client turned down my offer to build them the C routines(hard to support in the future) that can be called from PX Transformer, and they prefer to have Basic Routines that can be called in Basic Transformer for the purpose of easier support than C code.
That's my personal opinion, please feel free to comment on this.
1. Place Basic Transformer in a PX job won't break Partitioned Parallelism inscribed by PX engine, since it can be run in Parallel to map with Partitioned method declared in its Predecessor stage and its Successor stage. I once monitor this type of job running in Unix, and it spawns multiple physical processes in the system to fit in the parallelism requested by DataStage job.
2. I agree the performance penalty for using Basic Transformer cannot be avoided due to the high resource usage overhead(Mutilple Physcial Processes), unlike the generic PX transformer smart enough to combine the operator in runtime. However, if the data volume is not dramatically huge, and you need to use PX job to access certain useful stages(such as Join, Difference) that are not available in Server job, and you need to use some great Server functions such as IConv/OConv that are not available in PX job, then why not using this method. You won't see too much of the performance penalty and it's not a sin
![Smile :)](./images/smilies/icon_smile.gif)
3. As I was told by Ascential Advanced Engineer, the Basic Transformer is designed this way with the purpose listed in 1. and 2. I have designed some jobs fit in the condition as 2. for my client, and they works just fine with customer's satisfaction. As a matter of fact, my client turned down my offer to build them the C routines(hard to support in the future) that can be called from PX Transformer, and they prefer to have Basic Routines that can be called in Basic Transformer for the purpose of easier support than C code.
That's my personal opinion, please feel free to comment on this.
Pneuma Lin.
pneumalin@yahoo.com
pneumalin@yahoo.com
Partitioned Parallelism and either a mixed MPP and SMP environment or clustered environment won't work with DS BASIC Transformers. Then DS BASIC Transformer is only available on the one SMP server running the DS Engine.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Only on each SMP machine in the cluster - the MPP are out of consideration. I doubt anyone will pay to license a DS Engine on every phyiscal machine in the SMP cluster just to satisfy having a BASIC Transformer stage operational in full parallelism.ray.wurlod wrote:Unless, of course, you install a full DataStage Engine on every machine. Talk to your account rep - they'd LOVE to hear from you ($$$$$$).
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
MPP is for sure not in the case I described in my previous post. My clients has already paid the premium price upfront upon purchasing the product, so the license issues never come across to their attention. It's good to know the limitation for helping out the new clients in the future, Thanks for the info.
I still believe there is a valid reason for Basic Transformer usage to exist in the product, or we sould advise IBM to take away this feature in future release.
Appreciated if someone can find out this feature still existing in 8.0.1 or not?
I still believe there is a valid reason for Basic Transformer usage to exist in the product, or we sould advise IBM to take away this feature in future release.
Appreciated if someone can find out this feature still existing in 8.0.1 or not?
Pneuma Lin.
pneumalin@yahoo.com
pneumalin@yahoo.com
-
- Premium Member
- Posts: 291
- Joined: Wed Sep 26, 2007 11:23 am
- Location: Madrid, Spain
Great having started such an interesting debatepneumalin wrote: Appreciated if someone can find out this feature still existing in 8.0.1 or not?
As far as I am trying, in version 8 is not possible to use basic transformers any more.
Not avaliable in tool palete, and I tried Ctrl+Cing from a server job and then pasting, but pasted nothing.
Could anybody use it?
-
- Premium Member
- Posts: 291
- Joined: Wed Sep 26, 2007 11:23 am
- Location: Madrid, Spain
-
- Premium Member
- Posts: 291
- Joined: Wed Sep 26, 2007 11:23 am
- Location: Madrid, Spain
You may need to do the following to add Basic Transformer to your palette:
1. Create a PX job in designer.
2. Right Click on Palette window and select "Customization->Customize"
3. On the Top-Left window, click on StageType->Parallel->Process to select "Basic_Transformer"
4. Click on right arrow button to add to your PX job palette.
Let me know if you run into any problem using it in PX job.
1. Create a PX job in designer.
2. Right Click on Palette window and select "Customization->Customize"
3. On the Top-Left window, click on StageType->Parallel->Process to select "Basic_Transformer"
4. Click on right arrow button to add to your PX job palette.
Let me know if you run into any problem using it in PX job.
Pneuma Lin.
pneumalin@yahoo.com
pneumalin@yahoo.com