Is the output of the aggregator stage sorted ?

[Nsg]

Hello,

Does anyone know if sorted data is input to aggregator / funnel / join / merge stage, will the output data from these stages be also sorted ?

I have a very basic aggregator job with following details -

Infile - 40 fields
Key - 1 fields
aggr clms - 2 fields
pass through - 37 fields.

to achieve this, I am doing these steps -

1) Sort the infile on "Key" and generating "Key Change column"
2) Passing the sorted data through the copy stage ---
a) Aggregator stage with key_clm and 2 rollup fields
b) Filter stage with Key_clm and 37 pass through fields
3) Filter records with "Key Change column =1" .. thus getting distinct on key_clm
3) Aggregate the 2 fields.
4) sort the output data from aggregator
4) sort teh output data from filter stage
5) join the data using the Key field in Join stage.

Is it really needed to use the sort in step (4) or the aggrgator and filter stage will output the data sorted and hence will not need to be sorted for the "Join" stage.


Please help me understand the outputs from the stages a little better.

Thanks,
Neha

[ray.wurlod]

if sorted data is input to aggregator / funnel / join / merge stage, will the output data from these stages be also sorted ?a

Yes. Per node. Provided conditions are met such as Sort mode for Aggregator, Sort mode for Funnel.

[Nsg]

Thanks, Ray.

I thought the sort was just to tell how the incomming data is and how the stage should aggregate.

So based on what you say, following two will provide the same results (with $APT_NO_SORT_INSERTION=True and $APT_NO_PART_INSERTION=True)

Code: Select all

Dataset ------ Sort ------ Copy ------- Aggregator -------- Sort  
                             |                               |
                             |                               |
                           Filter -------- Sort ------------Join 


Dataset --------- Sort ---------- Copy ---------- Aggregator
                                     |                    |
                                     |                    |
                                 Filter ----------------- Join 

Thanks, Nsg

[chulett]

ps. I added the

Code: Select all

 tags around your ASCII art so it's closer to what you intended but I'm not about to spend my time cleaning it up so everything lines up perfectly.

You're welcome to do that, however. Edit the post and make liberal use of the 'Preview' option.

[Nsg]

ps. I added the

Code: Select all

 tags around your ASCII art so it's closer to what you intended but I'm not about to spend my time cleaning it up so everything lines up perfectly.

You're welcome to do that, however. Edit the post and make liberal use of the 'Preview' option.[/quote]

I have modified the formating .. 

please see the earlier post for details

Thanks, Nsg

[ray.wurlod]

Try it. Let us know.