Aggregator stage internal processing

sendmkpk · Post by **sendmkpk** » Tue May 14, 2013 3:43 am

Hi,

I want to know, how internally Aggregator stage does count opertaion?

does it do the count on each partition and sum it out to the output?

or does it do it in a sequential mode?

ray.wurlod · Post by **ray.wurlod** » Tue May 14, 2013 3:48 am

Neither. It does a count on each node. End of story.

sendmkpk · Post by **sendmkpk** » Tue May 14, 2013 8:33 am

Thx again for reply, its so simple for you ray, its confusing For me

So that means it counts at each node and sum it up at the end to give one value, yes?

jwiles · Post by **jwiles** » Tue May 14, 2013 9:45 am

No, it means just as Ray indicated: It does a count on each node that Agreggator is executing on.

If you need a single count across all nodes, you must run an Aggregator (or other stage) in a single node (i.e. sequential) in order to combine the counts. Aggregator is no different from other parallel stages in this regard.

Regards,

ray.wurlod · Post by **ray.wurlod** » Tue May 14, 2013 6:22 pm

While the job is running in parallel (let's assume on four nodes) you will have four counts. Nothing will sum those together unless you create more stages to do so, or unless you run your Aggregator on a single node (that is, in Sequential mode).