Abnormal termination of stage Error In Aggregator stage

csc.datastage · Post by **csc.datastage** » Thu Apr 30, 2009 12:39 pm

Hi,

My Job design is:

Seq File-Transformer-Aggregator-seq file.

I am getting this error while running:

Abnormal termination of stage TestZOCTRHS2_ChkDuplicates_Job1_v1..Aggregator_26 detected

After reseting the job: getting following message:

From previous run
DataStage Job 89 Phantom 16362
jobnotify: Unknown error
Job Aborted after Fatal Error logged.
Program "DSD.WriteLog": Line 239, Abort.
[16654] DSD.StageRun TestZOCTRHS2_ChkDuplicates_Job1_v1. TestZOCTRHS2_ChkDuplicates_Job1_v1.Aggregator_26 1 0/0 - core dumped.
Attempting to Cleanup after ABORT raised in stage TestZOCTRHS2_ChkDuplicates_Job1_v1..Transformer_23

DataStage Phantom Aborting with @ABORT.CODE = 1

Other Key items are:

Input data file size is 703 MB.

Thanks for any help in advance.

ray.wurlod · Post by **ray.wurlod** » Thu Apr 30, 2009 4:20 pm

Are your data sorted on the grouping keys?

Most Aggregator problems in server jobs relate to running out of memory because they have to keep the entire result set in memory. When input data are sorted this is not the case, and jobs run orders of magnitude faster as a result. You do, however, have to tell the Aggregator stage that its inputs are sorted in addition to providing sorted input.

csc.datastage · Post by **csc.datastage** » Thu Apr 30, 2009 6:46 pm

Thanks Ray!
You are right. Input to Aggregator stage is not sorted.
However I am confused about how to sort the input data.
By UNIX sort or SyncSort or sort Stage.
Which method will provide best performance.

ray.wurlod · Post by **ray.wurlod** » Thu Apr 30, 2009 6:51 pm

The third party sorters, like SyncSort and CoSort, cost money because they sort fast. The UNIX sort command is acceptable in many cases. Server Sort stage, however, is notoriously slow.

csc.datastage · Post by **csc.datastage** » Fri May 01, 2009 7:15 am

ray.wurlod wrote:You do, however, have to tell the Aggregator stage that its inputs are sorted in addition to providing sorted input.

Thanks a lot Ray!
I am exploring different options.
How we can tell Aggregator stage that Inputs are already sorted.

Thanks!

chulett · Post by **chulett** » Fri May 01, 2009 7:24 am

There are "Sort" columns in the Aggregator, one for order and another for direction. So, you can say "Fred" is the 1st field sorted ascending, etc. Make sure you don't lie to it, it will bust you!

And make sure you sort to match your grouping columns or the Aggregator will throw it all away and still do its own thing like it is doing now.

csc.datastage · Post by **csc.datastage** » Fri May 01, 2009 7:56 am

Thanks Craig!
I am working on 2 options: Sort Stage and SyncSort Utility.
I will keep updating how it goes.

chulett · Post by **chulett** » Fri May 01, 2009 7:58 am

You can tell your sorting is "working" when rows flow through the Aggregator rather than none coming out until all have gone in.