Aggregation Unknown Failure

[ippie02]

I am trying to perform an aggregation of a 2M record flat file and fail everytime around the 1.5M mark with an unknown error:

"Abnormal termination of stage srcDmSalesRepCanvPerfFDeltaTrAgg..Aggregate_Metrics detected"

I added a pre-exec sort which helped me go from 1.2M to 1.5 but when I added the sort order in the aggregator, it failed on this error

"srcDmSalesRepCanvPerfFDeltaTrAgg..Aggregate_Metrics: Row out of sequence
At row 60, link "toTempSrc"
Row out of sequence"

Now I group by the first 6 fields and do a combination of max() and sum() for my last 40 columns.

I have validated that the source file is not corrupted and fields are all ok. Is this a performance issue? What steps should I take to resolve this issue?

thank you

[chulett]

I added a pre-exec sort which helped me go from 1.2M to 1.5 but when I added the sort order in the aggregator, it failed on this error

"srcDmSalesRepCanvPerfFDeltaTrAgg..Aggregate_Metrics: Row out of sequence
At row 60, link "toTempSrc"
Row out of sequence"

This means the sort order you declared in the Aggregator doesn't match how you are actually sorting the file. Make sure they match and you'll be able to handle your volume.

[ippie02]

Thx for the reply. I do believe they match...

pre-exec sort:



in the input, i specified my sort for the first 6 fields then group by the first 6 fields in the output

Code: Select all

sort  -k 1 -k 2 -k 3 -k 4 -k 5 -k 6  -T #tempDir#DM_SALES_REP_CANV_PERF_F_DELTA -o #sourceDir#DM_SALES_REP_CANV_PERF_F_DELTA/DM_SALES_REP_CANV_PERF_F.tr #sourceDir#DM_SALES_REP_CANV_PERF_F_DELTA/DM_SALES_REP_CANV_PERF_F.tr

input columns:

sorts ascending on rows 1 to 6

output columns:

group by checked for first 6 fields
[/img][/code]

[chulett]

If they matched, you wouldn't get that particular error.

Presorting is pretty much the only way to have the Aggregator support that volume. It allows it to push rows through at each group change, rather than having to hold onto them all before outputting anything.

It can be tough to track down issues like this. Try looking at the sorted input file at the line it mentioned and see if you can tell where the sorting goes awry. It could be something like doing string sorts (which is the default) on numeric fields that you haven't zero padded to the same length. At least one of your fields isn't 'ascending' properly.

[kcbland]

Your issue is probably that the keys need the "n" to force numeric as Craig mentioned. They will left justify instead of right justify sort. "sort -k 1,1n 2,2n" would sort fields 1 and 2 numerically.