Best practices for 2 consecutives transforms

kjaouhari · Post by **kjaouhari** » Tue May 30, 2006 3:38 am

Hi all,

My question is just on the best practices for two consecutives transforms.
Transforms are used to do look up.

What is the best 1 or 2 ?
1. Put a Hash File or Sequential between the two Look up
2. Nothing between the transforms

Thanks !

ashwin141 · Post by **ashwin141** » Tue May 30, 2006 4:47 am

Hi

I will suggest that you should try avoiding the two transformers. Try implementing all the transformations in one transformer.

If at all you can't avoid then I thing directly linking this transformer to next transformer will be better that using a stage between them and then linking it to second transformer.

I hope that answers your question.

Regards
Ashwin

balajisr · Post by **balajisr** » Tue May 30, 2006 5:21 am

Can you tell exactly what you are trying to do?

As mentioned by ashwin it is better to use only one transfomer rather than two.

kumar_s · Post by **kumar_s** » Tue May 30, 2006 6:00 am

Two transformed next to next will be clubbed as a single process during runtime unless interprocess is introduced. And placeing a intermediate passive stage will increase a unecessary IO.

loveojha2 · Post by **loveojha2** » Tue May 30, 2006 6:11 am

kumar_s wrote:Two transformed next to next will be clubbed as a single process during runtime unless interprocess is introduced. And placeing a intermediate passive stage will increase a unecessary IO.

Kumar, I have one doubt here?
I am aware of the thing that if you put consecutive transformers it would be executed only by a single process, until you have the row buffering enabled.

The doubt is on how is it actually executed?
1. Do all the transformation clubbed together, then it is executed.
2. Or, each single transformer related transformations are executed first, then the row follows the transformations of the succeding transformers in that order.

Thanks

kumar_s · Post by **kumar_s** » Tue May 30, 2006 6:31 am

Not only transformer, all active stages are merged to a single process during compilation. Passive stages frame the boundary. All the transformation logic will be built in to a single process. There wont be pipeline parallelism among the stages.

kjaouhari · Post by **kjaouhari** » Tue May 30, 2006 6:32 am

In a first time, the first transformers (look up) is used to get the ColA in FileA
Then in a second time, ColA is used in the second transformers(look up) to get ColB in FileB.

So the process has 2 sequenced steps.

Thanks for your help !

ray.wurlod · Post by **ray.wurlod** » Tue May 30, 2006 6:56 am

kjaouhari wrote:Hi all,

My question is just on the best practices for two consecutives transforms.
Transforms are used to do look up.

What is the best 1 or 2 ?
1. Put a Hash File or Sequential between the two Look up
2. Nothing between the transforms

Thanks !

Nothing between the Transformer stages. If you have multiple CPUs, or the job in its current form consumes less than 40% of the single CPU, enable inter-process row buffering.

DSXchange