Problem with aggregator Stage

mdtauseefhussain · Post by **mdtauseefhussain** » Tue Jan 09, 2007 12:50 am

HI! All

Iam facing a problem with aggregator stage in my job,Ian selecting a records from source and doing a group by with two key fileds and aggrergatting with max() function with the rest of the fields.
The JOb is working fine when the record count is below 8 lakhs ,if the records count is more than 8 lakhs iam getting the following error

"Abnormal termination of stage EdwLocalListPricesJobNew..Agg_ProductIdCountryId detected"

This looks strange to us ,

waiting for your valuable suggestions

Regards

Tausif

bakul · Post by **bakul** » Tue Jan 09, 2007 1:17 am

Are there no other messages in the log?
What is the aggregation type that you are using in the aggregator?

mdtauseefhussain · Post by **mdtauseefhussain** » Tue Jan 09, 2007 1:27 am

Selecting the maximum records and group by on two key fields are types of aggregation we are doing .

ray.wurlod · Post by **ray.wurlod** » Tue Jan 09, 2007 4:20 am

The Aggregator stage does have memory limits. Sort the input by the grouping keys and assert on the Aggregator's input link that the input is sorted. This will mean that the Aggregator does not run out of memory, because it can free memory every time a grouping column value changes (because they are sorted, a change in value means that the previous value will never be seen again).