Memory problem with Funnel and Transformer

[lisamel888]

Hello there,

Our job used up all production server memory and failed. The design is :

Sequential file-->transfomer---100 links---->funnel----->sequential file.

The 100 links and funnel are used as a workaround of pivot because using pivot would fail the job in a much lower volume with a heap memory issue.

Does anyone know the memory problem is due to too many links in transformer or the funnel? Any suggestion/work around?

Many thanks in advance,

Lisa

[ArndW]

Reduce the number of nodes in your configuration file and/or sort your input data.

[girija]

Are you using any sorting logic in the transformer or funnel? Why its using memory. Please tell us little bit detail of your design.

[girija]

Are you using any sorting logic in the transformer or funnel? Why its using memory. Please tell us little bit detail of your design.

[lisamel888]

Hi there,

No sorting in the transformer. The transformer and funnel is used as an alternative of pivot. Is the funnel using up all the memory or the 100 ouput linds in the transformer? Any help is appreciated.

Th job failed on memory with only a couple of thousands rows. The same job on another server survives but still very memory/cpu intensive.

Thanks,
Lisa

[mahadev.v]

DataStage inserts a buffer operator on each input link to the funnel stage. So for 100 input links, 100 buffer operators, 100 processes. If it is a continuous funnel try using the APT_BUFFERING_POLICY set to "NO_BUFFERING".

[Pagadrai]

Hi Lisa,
Can you explain what Pivot logic are you trying to accomplish.
We can find a way to do it instaed of 100 links from the transformer.

[lisamel888]

Thanks for all the replies.

The original files are form mainframe Cobol Occurs Depending On (99 times).

We are currently working on a workaround to break the job down to 25 links... I will post our result......

thanks,
Lisa