DSXchange

I receive the following error during aggregation;

Aggregator_37,0: Failure during execution of operator logic.
Aggregator_37,0: Input 0 consumed 9614492 records.
Aggregator_37,0: Output 0 produced 9614492 records.
Aggregator_37,0: Fatal Error: pipe write failed: Broken pipe
Aggregator_37,0: Failure during execution of operator logic.
Aggregator_37,0: Input 0 consumed 9614046 records.
Aggregator_37,0: Output 0 produced 6683230 records.
Aggregator_37,0: Fatal Error: sendReadAck(): write failed on node SASHQOKWSDA Broken pipe
node_node2: Player 18 terminated unexpectedly.

I assume it's a space issue as the program worked fine with a 5 million row output. I ran it multiple time and it keeps aborting around the same amount of records. Is there a limit that I should change.

Thanks,

Have you monitored your scratch space while the job was running? Perhaps the broken pipe was caused by space issues.

Are you doing some pre-sorting before aggregation ?

Definitely its an space/memory issue. Temporary fix can be to run the job as sequential or make aggregator stage as sequential.

The aggregator is known to choke under large volumes of data. Check space and rearchitect the job if necessary.

If it's anything like the Server Aggregator stage, you can help it out tremendously by presorting your data in a manner than support the grouping being done. This should, in essence, remove any 'large volumes' issues.

I think in the aggregate method you are using hash, use "sort" instead.

Follow-up:

We have two servers. DEV and PROD If I run the same job on each server, I get two different results. On the DEV server, the job completes normally. On the PROD server, the job aborts. The only DS difference is a hot patch IBM offered to fix another problem. I have a suspicion that the patch has caused the problem. I will keep you posted.

I will also try some of your suggestions.

Thanks

Well after 5 weeks of grueling pain, I think we have solved this issue. Today June 18th, IBM will be releasing a patch regarding the MKS Toolkit which solves many broken pipe problems. If you are a Windows client and receive many broken pipe issues, I suggest getting the patch from IBM.

Raftsman, do you have a patch number that you can give me?

Five weeks? Looks more like a year and a half to me.