Broken Pipe

[din_datastage]

We are facing issue with our production env with broken pipe problem.i know usually broken pipe is related to network issue,but this seems to be wierd.This issue is recurring issue now when two application run parallely.when aplication1 is running it uses heavy operators.application 2 is getting failed with broken pipe because of thiis.


Write to dataset failed: Broken pipe
The error occurred on Orchestrate node node12 (hostname awdxm2p)

[arvind_ds]

Check disk space on awdxm2p host for dataset directory/file system.

[din_datastage]

I have checked disk space and there seems no issues with that..file system is just filled at 1%.there is another error along with broken pipe.


Fatal Error: Virtual data set.; output of "{natural="/hrdata/hr/prod/datasets/fact/HR_WRK_FEMPRFDEM_Load.ds", synthetic="input repartition(0)"}": DM getOutputRecord error.

[g_rkrish]

I think the problem is with your Dataset,it got corrupted so Please use the Orchestrate command to delete the dataset and recreate them by running the same job.

[ray.wurlod]

I think the problem is with your Dataset...

Why do you think that? I see no evidence for it in the original post.

[g_rkrish]

I have been through this before and checked with the space as it was alright, the option i went in to delete the Dataset through commands and recreate them it went through fine.

[din_datastage]

I have been through this before and checked with the space as it was alright, the option i went in to delete the Dataset through commands and recreate them it went through fine.

How do we know if dataset is corrupted or not...strange thing is when one application which uses heavey operators runs or completed then only dataset with particualr application abends with broken pipe and when we restart them it doens'nt work.

does datastage operator blocks any memory if there are heavy operators and doesn't go off??

[GJ_Stage]

Hi ,

It depends on the stages used in your job and sometime it will slowdown the job performance. When we use more number of transformer stage , lookup stage this error will occur.

[vijaykumarpj]

Sometimes, this may happen due to the space overflow in the ramdisk, if you have mentioned it the config file.