Aggregator Scenario

vijayrc · Post by **vijayrc** » Tue Oct 10, 2006 4:29 pm

Hi,
My Input Fields are FLD-1,FLD-2,FLD-3,FLD-4,FLD5,FLD-6,FLD-7
AGG1 Groups on FLD-1 sums on FLD-4 & FLD-5
AGG2 Groups on FLD-1,FLD-2 sums on FLD-4 & FLD-5
AGG3 Groups on FLD-1,FLD-2,FLD-3 sums on FLD-4,FLD-5,FL7
Can I FUNNEL the AGG1,AGG2,AGG3 output into one as Output File layout [similar to Input File layout]

...........--->AGG1--->
Seq IN -->AGG2--->Funnel---> Seq Out
...........-->AGG3--->

Above isn't possible, as the meta data of all input data sets must be identical. So what other inexpensive, simple STAGE after each AGGR and before FUNNEL can I use to get the varying fields from diff AGGR into one common layout for the funnel??.

Thanks
Vijay

kris007 · Post by **kris007** » Tue Oct 10, 2006 5:21 pm

The inputs to the Funnel stage needs to have identical(exact) metadata. So, your job desing won't work unless you are bringing out all the columns from all the aggregators which I doubt it. You could use a join stage or Merge stage followed by a Transformer.

vijayrc · Post by **vijayrc** » Tue Oct 10, 2006 5:28 pm

kris007 wrote:The inputs to the Funnel stage needs to have identical(exact) metadata. So, your job desing won't work unless you are bringing out all the columns from all the aggregators which I doubt it. You could use a join stage or Merge stage followed by a Transformer.

Thanks. I have bunch of Aggregators, and I don't want to have a transformer[for performance issue] for each Aggregator to get to the same metadata for the funnel. Any other Stage, that's inexpensive, and simple to get to a solution for this ??
[1]Would a Colum Generator take the AGGR output fields, and initialize the other fields for the funnel input ?? I'm more inclined towards this
[2] Can I use a COPY stage ?? Would this work and be simple

ray.wurlod · Post by **ray.wurlod** » Tue Oct 10, 2006 6:43 pm

Copy stage can not add columns to the record schema. Column Generator stage can.

kris007 · Post by **kris007** » Tue Oct 10, 2006 6:45 pm

vijayrc wrote: I have bunch of Aggregators, and I don't want to have a transformer[for performance issue] for each Aggregator to get to the same metadata for the funnel. Any other Stage, that's inexpensive, and simple to get to a solution for this ??
[1]Would a Colum Generator take the AGGR output fields, and initialize the other fields for the funnel input ?? I'm more inclined towards this

From what I understand about that stages, it should work fine. I have never tried it though.

vijayrc wrote: [2] Can I use a COPY stage ?? Would this work and be simple

I dont think it is possible.

pigpen · Post by **pigpen** » Tue Oct 10, 2006 7:40 pm

What about that? Add transformer after Agg and add dummy columns inside to sync all inputs before the Funl.

vijayrc · Post by **vijayrc** » Wed Oct 11, 2006 3:41 am

pigpen wrote:What about that? Add transformer after Agg and add dummy columns inside to sync all inputs before the Funl.

Thanks Kris and Ray..and I would try out the Col Gen, though I was skeptical abt the COPY stage.

Pigpen, I have bunch of Aggregators[20+] for different purposes, and don't want to burn myself out w/ a Transformer in each line just to get those output fields, with no value in them. Thanks anyways