Hi,
I have a transformer which provides 5 output links each having the same metadata.
I need to horinatally join all the records into a single link. Basically i am trying to achieve the funnel stage functionality of PX flavour,
I tried
TRANFORMER ===> Link Collector ==> Seq. File
, but obviously it will not work as both are active stage.
The Link Collector stage creates a process of its own; therefore to communicate with other active stages some form of inter-process communication mechanism is required - inter-process row buffering must be enabled (which is made very clear in the manual). If you want visibility of the buffering, you can add Inter Process Communication (IPC) stages to the job design.
IHTH
sAM
I haven't failed, I've found 10,000 ways that don't work.
Thomas Alva Edison(1847-1931)
Hi Rejith, Thanks for your thoughts, I am not sure if it will work, but I feel that the usage of hashfile (which is gonna occupy physical memory) for this scenario is not an apt and optimal solution,
We have these jobs for almost 40 countries(hence 40 jobs/module, we have number of modules and design is generic) and considering the volume of records, i personally felt that its better we have a design which occupies less phyical space, please correct me if i am wrong in my thoughts. Even though we can purge these hash files, the house keeping activity becomes an over head.
There will always be an overhead. You cannot process large (or even small) volumes of data without demanding some resources to do so.
Simple economics.
Try a number of techniques, benchmark and measure them in your environment, and make your choice based upon those results.
"There ain't no such thing as a free lunch."
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
I'd be curious what 'house keeping' you are worried about. And if you really were worried about hashed files, they can be easily purged automagically - heck, you could even make that the trailing end of your job stream if you so desired.
-craig
"You can never have too many knives" -- Logan Nine Fingers