Slower downstream stage affects performance of upstream stag

chetan.c · Post by **chetan.c** » Sun Apr 29, 2012 7:05 am

Hi,

My job design looks like below.

Ext_Source---transformer---Java Transaformer---Java Transaformer---Transformer--dataset.

When i run the Code only from

Code: Select all

External source --Transformer---Dataset

, the rows/second from the stages are around 1400rows/sec.

But if the Complete design is included the External source_stage and transformer output slows down to 250rows/sec!!.

Is my inference correct that a slower perforrming downstream stage affects faster performing upstream stages?

Thanks,
Chetan.C

chulett · Post by **chulett** » Sun Apr 29, 2012 7:19 am

Of course. Which is why they are known as bottlenecks.

chetan.c · Post by **chetan.c** » Sun Apr 29, 2012 7:26 am

Thanks Craig.

chulett · Post by **chulett** » Sun Apr 29, 2012 8:23 am

Identification of them is the first step, after that you can work on ameliorating them. What's going on in the Java transformers? Any way to replace them with native DataStage functionality?

chetan.c · Post by **chetan.c** » Mon Apr 30, 2012 12:18 am

chulett wrote: What's going on in the Java transformers? Any way to replace them with native DataStage functionality? ...

Hi Craig,

The java transformer has the code to generate XPATH,element name and value.We are using this as the xsd of the XML's that we get comes from over 1000 applications and all have their own structures.

The other java transformer does cleansing of XML.

So its not possible to parse even with XML assembly stage also.

That's why we had to go for the java code.

Thanks,
Chetan.C

chulett · Post by **chulett** » Mon Apr 30, 2012 6:48 am

Well... then it sounds like you're stuck with the neck of that particular bottle.

jerome_rajan · Post by **jerome_rajan** » Mon Apr 30, 2012 9:01 am

The lag is being caused by the way buffers work in DataStage and is an essential.

You might want to have a look at the "score" to ensure that no unnecessary repartitioning is happening.

Also, if your Transformer is not implementing any business logic and is just performing minor conversions, you might want to have a look at the Modify stage.

If performance is a very critical facet and you have substantial memory at your disposal, try to increase the buffer size.

Not too sure if it would be a wise idea to increase the APT_BUFFER_FREE_RUN but it might help. Trial & error will give you a good idea

chetan.c · Post by **chetan.c** » Mon Apr 30, 2012 9:26 am

chulett wrote:Well... then it sounds like you're stuck with the neck of that particular bottle. ...

yes right at the neck ..we are trying to improve performance of the Java code right now..

Hi Jerome,
Thanks . I did think about buffer memory but thought first let us tune the java code as much as possible and then look into that.

Thanks,
Chetan.C