Is there any limit for Stage variables in one transformer
Moderators: chulett, rschirm, roy
-
- Charter Member
- Posts: 64
- Joined: Sat Sep 17, 2005 10:42 am
Is there any limit for Stage variables in one transformer
How many stage variables one can use in one transformer, i had a case where i am using 40+ stage variables.I got compilation errors then i used two tarnsformers(with 20 stage variables each) with a copy stage in between.
Can i use back to back trasformers(or Active stages like Lookup,Join) with out having a dummy stage in between. Does this create any performance problems when the #of records is huge.
Can i use back to back trasformers(or Active stages like Lookup,Join) with out having a dummy stage in between. Does this create any performance problems when the #of records is huge.
-
- Charter Member
- Posts: 560
- Joined: Wed Jul 13, 2005 5:36 am
- Location: Ohio
-
- Charter Member
- Posts: 822
- Joined: Sat Sep 17, 2005 5:25 pm
- Location: USA
-
- Charter Member
- Posts: 64
- Joined: Sat Sep 17, 2005 10:42 am
-
- Charter Member
- Posts: 64
- Joined: Sat Sep 17, 2005 10:42 am
-
- Charter Member
- Posts: 560
- Joined: Wed Jul 13, 2005 5:36 am
- Location: Ohio
-
- Charter Member
- Posts: 822
- Joined: Sat Sep 17, 2005 5:25 pm
- Location: USA
Rao,panchusrao2656 wrote:Hi us1aslam1us,
Can you explain briefly what happens when we use multiple active stages without having any dummy stages in between.
For example, i need to extract data based on keys which i cannot use them in one lookup stage or join stage. Don't we run out of memory.
As far as i know when u process two active stages, in the background a passive stage will be defaulted in these two active stages.
If you can elaborate more on your design and what problem you are facing, Atleast some one from the premium posters can throw some light on memory and performance issues.
Sam
Not to mention that it is probably poor form to have 60 stage variables in a single transformer! Maybe look at separating into two (2) or possibly more.
Regards,
Regards,
Mike Hester
mhester@petra-ps.com
mhester@petra-ps.com
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Just because you can does not mean you shouldThe most stage variables I have used in a Transformer stage is 220. It was successful.
Mike Hester
mhester@petra-ps.com
mhester@petra-ps.com
The fact that you are asking this question motivates me to ask: do you really think this is a good design? How about moving a lot of that logic into a custom function or buildop?
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Yes the many stage you use, the more the memory it occupies.panchusrao2656 wrote:Hi us1aslam1us,
Can you explain briefly what happens when we use multiple active stages without having any dummy stages in between.
For example, i need to extract data based on keys which i cannot use them in one lookup stage or join stage. Don't we run out of memory.
By default there are some staeges which can combine during compilating and runtime to act as a single process, and this is enabled by COMBINALBE option avaialble both in project level and stage levels.
It is not nessary that each stage will contibute to single stage, there are some case where one can lead to two or more and where in some case, it will be very light in such a way that it wont impose any process at all.
You can find more in detail in Operators guide/Orchestrate guide.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
-
- Charter Member
- Posts: 64
- Joined: Sat Sep 17, 2005 10:42 am
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Charter Member
- Posts: 64
- Joined: Sat Sep 17, 2005 10:42 am
The fields we were using are Decimals ( In Oracle they were defined as Numbers, so by default it is treating them as Decimal (38,10) ). Besides these we have a total of 300+ input fields & nearly 30+ million records.
We had all kinds of troubles before splitting the stage variables among multiple transformers.
Ray,
Do you suggest using multiple active stages like Join,Lookup,Transformer without any passive stages in between.Can you explain theimpact of using those. I am trying to avoid back to back active stages by keeping a copy stage in between.
Thank You
We had all kinds of troubles before splitting the stage variables among multiple transformers.
Ray,
Do you suggest using multiple active stages like Join,Lookup,Transformer without any passive stages in between.Can you explain theimpact of using those. I am trying to avoid back to back active stages by keeping a copy stage in between.
Thank You