Funnel and Merge Stage

dgokulakrishnan · Post by **dgokulakrishnan** » Wed May 01, 2013 1:22 pm

Can anyone please explain Why Funnel and Why not Merge stage to combine several files?

ray.wurlod · Post by **ray.wurlod** » Wed May 01, 2013 2:30 pm

It depends what you mean by "combine". The analogies that you need are that the Funnel stage effects a UNION ALL of its input rows, while the parallel Merge stage effects a "master and updates" approach (which means that unmatched masters and unmatched updates may not make it into the result, depending on how you configure the stage).

kduke · Post by **kduke** » Wed May 08, 2013 8:26 pm

We used a funnel stage because we had multiple record types. Say you have a sequential file you want to output. It needs a header record and a trailing record. You can use a funnel stage that inputs all three as one column says 255 bytes. So you build your header record as one big string. Same on all your detail records. Then maybe the trailer wants total record count. You can build this and funnel all these together and output to one sequential file stage. Mike Stafford taught me this trick. Used to output 3 files then concatenate after job was over. This way is cleaner.