Page 1 of 1
Slower downstream stage affects performance of upstream stag
Posted: Sun Apr 29, 2012 7:05 am
by chetan.c
Hi,
My job design looks like below.
Code: Select all
Ext_Source---transformer---Java Transaformer---Java Transaformer---Transformer--dataset.
When i run the Code only from
Code: Select all
External source --Transformer---Dataset
, the rows/second from the stages are around 1400rows/sec.
But if the Complete design is included the
External source_stage and transformer output slows down to 250rows/sec!!.
Is my inference correct that a slower perforrming downstream stage affects faster performing upstream stages?
Thanks,
Chetan.C
Posted: Sun Apr 29, 2012 7:19 am
by chulett
Of course. Which is why they are known as
bottlenecks.
Posted: Sun Apr 29, 2012 7:26 am
by chetan.c
Thanks Craig.
Posted: Sun Apr 29, 2012 8:23 am
by chulett
Identification of them is the first step, after that you can work on ameliorating them. What's going on in the Java transformers? Any way to replace them with native DataStage functionality?
Posted: Mon Apr 30, 2012 12:18 am
by chetan.c
chulett wrote: What's going on in the Java transformers? Any way to replace them with native DataStage functionality? ...
Hi Craig,
The java transformer has the code to generate XPATH,element name and value.We are using this as the xsd of the XML's that we get comes from over 1000 applications and all have their own structures.
The other java transformer does cleansing of XML.
So its not possible to parse even with XML assembly stage also.
That's why we had to go for the java code.
Thanks,
Chetan.C
Posted: Mon Apr 30, 2012 6:48 am
by chulett
Well... then it sounds like you're stuck with the neck of that particular bottle.
Posted: Mon Apr 30, 2012 9:01 am
by jerome_rajan
The lag is being caused by the way buffers work in DataStage and is an essential.
You might want to have a look at the "score" to ensure that no unnecessary repartitioning is happening.
Also, if your Transformer is not implementing any business logic and is just performing minor conversions, you might want to have a look at the Modify stage.
If performance is a very critical facet and you have substantial memory at your disposal, try to increase the buffer size.
Not too sure if it would be a wise idea to increase the APT_BUFFER_FREE_RUN but it might help. Trial & error will give you a good idea
Posted: Mon Apr 30, 2012 9:26 am
by chetan.c
chulett wrote:Well... then it sounds like you're stuck with the neck of that particular bottle. ...
yes right at the neck ..we are trying to improve performance of the Java code right now..
Hi Jerome,
Thanks . I did think about buffer memory but thought first let us tune the java code as much as possible and then look into that.
Thanks,
Chetan.C