How to funnel multiple links into single output link ?

coehcl · Post by **coehcl** » Sat Mar 24, 2007 12:31 am

Hi,
I have a transformer which provides 5 output links each having the same metadata.
I need to horinatally join all the records into a single link. Basically i am trying to achieve the funnel stage functionality of PX flavour,

I tried

TRANFORMER ===> Link Collector ==> Seq. File

, but obviously it will not work as both are active stage.

Could some one give me a pointer on this,

Warm Regards,
Coehcl

us1aslam1us · Post by **us1aslam1us** » Sat Mar 24, 2007 1:04 am

Check this reply by Ray.Wurlod from a previous post :

Code: Select all

The Link Collector stage creates a process of its own; therefore to communicate with other active stages some form of inter-process communication mechanism is required - inter-process row buffering must be enabled (which is made very clear in the manual). If you want visibility of the buffering, you can add Inter Process Communication (IPC) stages to the job design.

IHTH
sAM

coehcl · Post by **coehcl** » Sat Mar 24, 2007 1:14 am

Hi,
Thanks for the response.
But IPC does cannot take more than 1 input,
hence

TXF ==>(Cannot take more than 1 Input) IPC ==> Link Collector ==> Seq. File

will not work,

Any other suggestion pls.

Thanks

rejith.ry · Post by **rejith.ry** » Sat Mar 24, 2007 2:15 am

Use hash files before ans after the link collector. Use one hash file/per each input & output link of link collector.

coehcl · Post by **coehcl** » Sat Mar 24, 2007 2:20 am

Hi Rejith, Thanks for your thoughts, I am not sure if it will work, but I feel that the usage of hashfile (which is gonna occupy physical memory) for this scenario is not an apt and optimal solution,

can't we derive the output on the fly ??

ray.wurlod · Post by **ray.wurlod** » Sat Mar 24, 2007 5:18 am

Five IPC stages. But these also take physical memory (for the buffers). Why can't your design take physical memory?

coehcl · Post by **coehcl** » Sat Mar 24, 2007 12:56 pm

We have these jobs for almost 40 countries(hence 40 jobs/module, we have number of modules and design is generic) and considering the volume of records, i personally felt that its better we have a design which occupies less phyical space, please correct me if i am wrong in my thoughts. Even though we can purge these hash files, the house keeping activity becomes an over head.

Thanks for the response.

ray.wurlod · Post by **ray.wurlod** » Sat Mar 24, 2007 1:03 pm

There will always be an overhead. You cannot process large (or even small) volumes of data without demanding some resources to do so.
Simple economics.

Try a number of techniques, benchmark and measure them in your environment, and make your choice based upon those results.

"There ain't no such thing as a free lunch."

chulett · Post by **chulett** » Sat Mar 24, 2007 1:51 pm

I'd be curious what 'house keeping' you are worried about. And if you really were worried about hashed files, they can be easily purged automagically - heck, you could even make that the trailing end of your job stream if you so desired.

DSguru2B · Post by **DSguru2B** » Sat Mar 24, 2007 9:32 pm

You can also load the 5 links into 5 different flat files and in an after job subroutine, cat them all together.

chulett · Post by **chulett** » Sun Mar 25, 2007 7:52 am

More house keeping.

coehcl · Post by **coehcl** » Sun Mar 25, 2007 9:10 am

Thank You everyone for your expert suggestions,

I am closing this thread with the below solution as the optimal one,

TXF (5 o/p links)==>5 IPC ==> Link Collector ==> Seq. File TXF

Warm regards,