Is there any limit for Stage variables in one transformer

panchusrao2656 · Post by **panchusrao2656** » Wed Jul 12, 2006 2:02 pm

How many stage variables one can use in one transformer, i had a case where i am using 40+ stage variables.I got compilation errors then i used two tarnsformers(with 20 stage variables each) with a copy stage in between.

Can i use back to back trasformers(or Active stages like Lookup,Join) with out having a dummy stage in between. Does this create any performance problems when the #of records is huge.

Krazykoolrohit · Post by **Krazykoolrohit** » Wed Jul 12, 2006 2:07 pm

can you post the error?

I am not sure about the limit but 20 is certainly not that number.

us1aslam1us · Post by **us1aslam1us** » Wed Jul 12, 2006 2:08 pm

Hi Rao,

There is no such limit on the use of stage variables but it all depends on what conditions or constraints you are working with. You can use two active stages together.

Thanks
Sam

panchusrao2656 · Post by **panchusrao2656** » Wed Jul 12, 2006 2:16 pm

I donot have the log now, one of my friend is using 60+ stage variables in one transformer, then we split them by adding two more transformers. It worked fine later.

panchusrao2656 · Post by **panchusrao2656** » Wed Jul 12, 2006 2:20 pm

Hi us1aslam1us,

Can you explain briefly what happens when we use multiple active stages without having any dummy stages in between.

For example, i need to extract data based on keys which i cannot use them in one lookup stage or join stage. Don't we run out of memory.

Krazykoolrohit · Post by **Krazykoolrohit** » Wed Jul 12, 2006 2:22 pm

I have used almost 30 stage variables some time in a single transformer.

you must be sure about how the stage variabels work ie they get compiled before the columns from top to bottom.

If you can post the error it would have been a great help in understanding your issue

us1aslam1us · Post by **us1aslam1us** » Wed Jul 12, 2006 2:38 pm

panchusrao2656 wrote:Hi us1aslam1us,

Can you explain briefly what happens when we use multiple active stages without having any dummy stages in between.

For example, i need to extract data based on keys which i cannot use them in one lookup stage or join stage. Don't we run out of memory.

Rao,

As far as i know when u process two active stages, in the background a passive stage will be defaulted in these two active stages.

If you can elaborate more on your design and what problem you are facing, Atleast some one from the premium posters can throw some light on memory and performance issues.

Sam

mhester · Post by **mhester** » Wed Jul 12, 2006 2:59 pm

Not to mention that it is probably poor form to have 60 stage variables in a single transformer! Maybe look at separating into two (2) or possibly more.

Regards,

ray.wurlod · Post by **ray.wurlod** » Wed Jul 12, 2006 3:32 pm

The most stage variables I have used in a Transformer stage is 220. It was successful.

mhester · Post by **mhester** » Wed Jul 12, 2006 4:07 pm

The most stage variables I have used in a Transformer stage is 220. It was successful.

Just because you can does not mean you should

kcbland · Post by **kcbland** » Wed Jul 12, 2006 4:13 pm

The fact that you are asking this question motivates me to ask: do you really think this is a good design? How about moving a lot of that logic into a custom function or buildop?

kumar_s · Post by **kumar_s** » Wed Jul 12, 2006 5:58 pm

panchusrao2656 wrote:Hi us1aslam1us,

Can you explain briefly what happens when we use multiple active stages without having any dummy stages in between.

For example, i need to extract data based on keys which i cannot use them in one lookup stage or join stage. Don't we run out of memory.

Yes the many stage you use, the more the memory it occupies.
By default there are some staeges which can combine during compilating and runtime to act as a single process, and this is enabled by COMBINALBE option avaialble both in project level and stage levels.
It is not nessary that each stage will contibute to single stage, there are some case where one can lead to two or more and where in some case, it will be very light in such a way that it wont impose any process at all.
You can find more in detail in Operators guide/Orchestrate guide.

panchusrao2656 · Post by **panchusrao2656** » Thu Jul 13, 2006 7:30 am

Thank you for sharing your ideas

ray.wurlod · Post by **ray.wurlod** » Thu Jul 13, 2006 10:01 am

int8 stage variables consume only one byte - most of my 220 were int8

panchusrao2656 · Post by **panchusrao2656** » Thu Jul 13, 2006 10:43 am

The fields we were using are Decimals ( In Oracle they were defined as Numbers, so by default it is treating them as Decimal (38,10) ). Besides these we have a total of 300+ input fields & nearly 30+ million records.

We had all kinds of troubles before splitting the stage variables among multiple transformers.

Ray,

Do you suggest using multiple active stages like Join,Lookup,Transformer without any passive stages in between.Can you explain theimpact of using those. I am trying to avoid back to back active stages by keeping a copy stage in between.

Thank You