Error in Aggregator Stage

shiva459 · Post by **shiva459** » Thu Jan 20, 2005 12:27 pm

Hi

I have a job in which I am using aggregator stage.Hash file is the input for aggregator and I group by two columns and take min of the third column.I have 2.5 million records in hash file and the job gets aborted at 1.51M every time i run.It aborts at the same row num and the log does not have any information except for " ABNORMAL TERMINATION OF AGGREGATOR STAGE DETECTED".If anyone has insight into this problem please let me know.

Thanks
S

chulett · Post by **chulett** » Thu Jan 20, 2005 12:47 pm

It generally means you are processing more rows than the stage can handle. It would help if you can presort your data on the grouped fields and then assert that in the stage by marking those columns as sorted.

Is that a possibility?

rasi · Post by **rasi** » Sat Jan 22, 2005 2:44 am

I won't advise to use datastage aggregator plug-in if have millions of records unsorted. The efficient way is to put your records into a Temp Stage table and use the power of your database engine to do the aggregation, max, min, group. You will be amazed to see the difference between those two.

Thanks

andru · Post by **andru** » Mon Jan 24, 2005 1:17 am

I agree with Rasi on avoiding aggregator stage for huge data. As sugessted you can dump the data into a temp table and do a group by query on the table to find the min value.

Another alternative is that u can dump the data into a hash file with a transofmer in between. Let the key for your target hash file be you group by columns. In the transformer you will have stage variables to find the minimum value between the current and the previous values. Always the minimum value will keep on overwritting on the same record, since the key of the hash file is same as that of your group variable.

rasi · Post by **rasi** » Tue Jan 25, 2005 1:51 am

If you are trying Andru's suggestion then your source has to be sorted in order to use Stage Variables

Thanks