How to avoid using flat files & connect transformers out

vinaymanchinila · Post by **vinaymanchinila** » Wed May 04, 2005 9:39 am

Hi,
I have 3 transformers generating data with same metadata. Instead of staging it into 3 flat files and merge them and load into Target, is there a way I can avoid the flat files and merge the output of the transformers . I did use the link collector but it doesnt take active inputs!

Thanks

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Wed May 04, 2005 9:42 am

Use 'Append' option for the output files.

vinaymanchinila · Post by **vinaymanchinila** » Wed May 04, 2005 9:45 am

I did use append still when I use the files I get error saying required column missing when that particular column is nullable and not a key!.
So if I can avoid the files altogether and somehow link the 3 Xfms to the target.
Thanks,

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Wed May 04, 2005 9:48 am

You can use link-collector to merge. But the issue you are raising is not because of 'append'. So whatever way you go, you will face the same problem.

vinaymanchinila · Post by **vinaymanchinila** » Wed May 04, 2005 10:12 am

When I use Link collector and drag the 3 output links from transformers , when I compile it says , likn collector does not support active stages.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Wed May 04, 2005 10:14 am

Enable row buffering or introduce an IPC stage. You can search this forum for details info about this.

vinaymanchinila · Post by **vinaymanchinila** » Wed May 04, 2005 10:16 am

How would that solve the issue about connecting 3 transformers to a link collector. You mean use IPC in between the 3 ransformers and the link collector?

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Wed May 04, 2005 10:22 am

Exactly Correct.

vinaymanchinila · Post by **vinaymanchinila** » Wed May 04, 2005 10:52 am

The IPC doe not support more than one input dude!

chulett · Post by **chulett** » Wed May 04, 2005 11:20 am

Use three of them.

vinaymanchinila · Post by **vinaymanchinila** » Wed May 04, 2005 12:21 pm

I have used 3 IPC stages and then directly connected them to an Link collector and then to the target.
The Issue is all the 3 transformers run fine and when the link comes to the IPC stage it just shows running. I stopped the job and increased the IPC connection time out to 999999. Still the job does not execute!.
Any help would be appreciated.

rsaliah · Post by **rsaliah** » Thu May 05, 2005 2:39 am

How about loading to a hash file. You can have multle inputs to the same file. Obviously make sure your keys are unique and since you don't know which link will write first ensure that the file is cleared prior to the job running.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu May 05, 2005 4:54 am

Why can't you split the job into two? First creating 3 seq files followed by a 'post-job' concatenate and the second doing the load.

vinaymanchinila · Post by **vinaymanchinila** » Thu May 05, 2005 5:14 am

The whole point is of making the job more efficient, and not use the files. I can concatenate teh files int eh same job but why to go for it when I am able to connect the transformers throguh IPC stage to Link Collector.
The issue is the link collector doesnt look like it is working.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu May 05, 2005 5:19 am

In your old job design (without the IPC stages), switch on the row buffering properties for that job. This will also do the trick.