Performace Issue

ag_ram · Post by **ag_ram** » Wed Mar 12, 2008 4:44 pm

My job is designed as below.

MQ --> XML input stage --> transfrmer --> funnel --> XML output --> transformer --> MQ

We get around 2million records into the queue almost in less time. Our job not able to consume fast enough and MQ getting overloaded. Finally it job got aborted. When I checked the processing speed, job able to process 1000 records/second. What type of tuning can i do in the job? Can i know which stage might be creating this hit?
Our job has transaction count as 1. It does not process in batch mode. Any pointers are invited.

kumar_s · Post by **kumar_s** » Wed Mar 12, 2008 4:55 pm

What is the error message?
Try to increase the transaction count too.

ag_ram · Post by **ag_ram** » Thu Mar 13, 2008 12:58 pm

It is not ginving any error message. Our concern is slowness. Job is working fine. We want the job to process faster. I cant increase transaction count as it is message by message processing. It will create the duplicates at end if i do process in batch mode.

kcbland · Post by **kcbland** » Thu Mar 13, 2008 2:56 pm

Are you able to quantify the slowest part of the transformation process? Can you be sure that act of writing back to the MQ queue is not what limits you to 1000 records/second? What about the Transformer? You didn't even begin to describe what's going on in there.

Imagine if the job looked like this:

MQ --> SEQ

and

SEQ --> MQ

You could measure the speed of reading the queue and the spead of writing to the queue. Whatever logic you then put between

MQ --> ........whatever..............--> MQ

is only going to slow your job down further. You're going to need to determine what stages impact the performance the most. I would start with MQ --> XML input stage and see what the rows/second achieves. Then add the Transformer and output to /dev/null and see what speed reduction occurs. I don't know the purpose of the funnel, but I keep going. Figure out where performance becomes unacceptable.

bkumar103 · Post by **bkumar103** » Fri Mar 14, 2008 6:15 am

Could you please specify where and how to specify the Transaction Count property.

anshumangupta1206 · Post by **anshumangupta1206** » Wed Mar 19, 2008 3:44 am

Hi,

Trasnaction size is present in the database stage. Check the db stage where you are writing the data.