Performance Issue with Multiple Instance

yalamanchili · Post by **yalamanchili** » Sun Feb 15, 2009 1:55 pm

Hi,

We had an single Interface code in production scheduled to run for Multiple Countries at the Same time. When scheduled at same time, the interface sequence is taking much more time than the normal Execution time.
After Intial analysis some problem areas were identified. I could see that in a sequence from Comepletion of One job to calling of the Next job based on trigger conditions it is waiting for 4-5 mins. This suming up and creating a large time diference at the End of Completion of sequence.

We had Ensured that no Locking is happening because of the Multiple Instance. When Interface is run for Single country it is taking 20 Mins, when run for morethan 4 countries at same time it is taking 4-5 hours for completion. In the taskmanager of server, I could able to see around 40 uvsh process generated. Does this had anything to do with the performance?

Can you please let me know your thoughts on this as i need to fix this issue in Production at the earliest

ray.wurlod · Post by **ray.wurlod** » Sun Feb 15, 2009 2:42 pm

Everyone KNOWS that a DataStage server can deliver an infinite supply of resources, right?

No-one can help you with the scant information you have provided. Are you monitoring your system? Do you know what the metrics of resource usage are (for example %Idle, PF/Sec)?

yalamanchili · Post by **yalamanchili** » Sun Feb 15, 2009 4:03 pm

Hi ray,

Thanks for Replying. Let me put you simple. When I am running the Interface for Single Country, it is taking 20 Mins. If I am running the same for 15 countries, It is taking 7-8 Hrs for Completion for each country.

In the sequence i had checked timings of all Individual jobs. But all jobs are taking normal timings. Only one Observation found was from job to job calling in sequence, there was 4-5 mins difference. As the Number of Countries to be run at same time is increasing, this time is also increasing. Can you provide me any help on this.

Please do let me know if any further information is required.

chulett · Post by **chulett** » Sun Feb 15, 2009 5:34 pm

Match your demand to what can be supplied. If your server doesn't have the resources to run those jobs in parallel, sounds like you'd be better off running serial - at least those 15 would finish in 5 hours that way. Or run fewer simultaneous instances.

yalamanchili · Post by **yalamanchili** » Sun Feb 15, 2009 5:56 pm

Hi,

Previously we had 15 different projects for 15 countries in production. At that point of time, we used to Schedule all the 15 countries at same time. we Faced no such issues with the Interface run. But due to maintanace problems we are asked to move to single datastage project for all 15 countries.

Now as we have limited bandwidth window to run interface for all the countries. We had to schedule interface run for some countries in parallel, which is causing problems.

Can you please suggest what needs to be done. Will the single project appraoch will have any impact on performance.

ray.wurlod · Post by **ray.wurlod** » Sun Feb 15, 2009 8:39 pm

The single approach is immaterial. You are trying to run 15 sets of jobs in parallel (whether from one project or from 15) and don't have the server resources to do so. Experiment and monitor; maybe you have the resources to run two or three simultaneously.

yalamanchili · Post by **yalamanchili** » Thu Mar 05, 2009 3:38 pm

Hi,

I have been monitoring the CPU utilization on server. when Interface sequence is running for multiple countries at same time, i could see CPU utilization is less than 10 percent.

So i can say it is not problem with server handling the resources. Can anyone suggest performance tuning steps like deletion of log files etc that need to performed before running the interface for multiple countries at same time.

Also please let me know what could be probable reason for Interface sequence becoming slow.

sima79 · Post by **sima79** » Thu Mar 05, 2009 5:03 pm

I think you need to identify the bottleneck first before looking for possible solutions. What is your job doing? Are you connecting to any remote databases? What is your network configuration?

Kryt0n · Post by **Kryt0n** » Thu Mar 05, 2009 5:32 pm

I think what they are trying to ask is whether there are any DataStage (or UNIX?) settings that restrict how much resources one project can use.

If 15 projects can run concurrently with no performance issues then why can't 15 jobs within one project...

yalamanchili · Post by **yalamanchili** » Thu Mar 05, 2009 5:37 pm

Hi Seema,

Thanks for the Reply. The only problem i have identified is calling of Individual jobs in sequence.

Detail Description of Sequence:

1. Sequence has around 80 jobs with Multiple Instance property checked
2. Sequence when executed for 1 country taking 20 Min ( Specifies job execution is not taking much time)
3. Sequence when run for more than 4 countries at same time is taking around 7 hours for each country.
-- Actual job runs in sequence is taking only 30 Min out of these 7 hours per country
--After completion of One job in sequence, and before starting of next job there is a gap of around 5 Min when Multiple country Execution is going on.
---- This gap is summing up and creating 6.5 hours of blank time in sequence execution.

Can you please share your thoughts on this issue.

Please do let me know if any further information is required from my side.

Regards,
Y.Kiran

Mike · Post by **Mike** » Fri Mar 06, 2009 9:21 am

Go back and reread Ray's post... You don't have the server resources to support the demanded workload. CPU is but one type of resource. You need to find the resource that is your bottleneck.

Mike