Performance Issue after migrating to 7.5.3

krishnakumar_m16 · Post by **krishnakumar_m16** » Sun Feb 19, 2012 8:18 am

Recently we migrated to Datastage 7.5.3 from Datastage 7.5.1 (EOSL on 30th April 2012).

Post migration we are facing Performance problems.

Background information on our project

All our jobs in our projects are RTI (Realtime integration)jobs. They are accessed by Java application through RTI agent/RTI Server.

Hardware configuration
System Model: IBM,9119-590
Processor Type: PowerPC_POWER5
Processor Implementation Mode: POWER 5
Processor Version: PV_5
Number Of Processors: 4
Processor Clock Speed: 1654 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
Memory Size: 32256 MB
Good Memory Size: 32256 MB
Platform Firmware level: SF240_358
Firmware Version: IBM,SF240_358
Paging Space Information
Total Paging Space: 51328MB

All the jobs are higly complex with multiple shared containers, if we explode the shared containers in any of the RTI job on average it would contain 800 - 1000 stages with sparse lookup, joins, transformers,funnels, aggregators etc.

We have on total 20 RTI jobs with two instances for each job. It would take 30 minutes for us to enable all the jobs in the RTI console. we would enable each job wait for CPU idle to come back to 85 to 90% and would enable the next job.

Behaviour in 7.5.1
After all the jobs are enabled paging file utilization would be 18-19%.
After running the load test, the jobs would process messages, and the paging file utilization would reach 22% and remain in 22% through out the load (4 hours). load would ramp up to full load in 15 minutes.
Average CPU utilization 30%
CPU idle time would be around 70 to 80%

Behaviour in 7.5.3
After all the jobs are enabled paging file utilization would be 18-19%.
After running the load test, the jobs would process messages, and the paging file utilization would reach
23% in 5 mins
25% in 15 mins
30% in 30 mins and would go upto 45 %.
As and when the paging file utilization goes up the response of the RTI jobs starts to degrade.
Average CPU utilization is 60 - 70%
CPU idle would be 30 - 45

Is there any way to reduce the paging file utilization of the Datstage engine.
All configuration are identical between 7.5.1 and 7.5.3
identical uvconfig file
only instance tag is the difference, rest all are same. Both run on the same server. [Performance tests were conducted by shutting down the other server]

uvconfig used for both 7.5.1 and 7.5.3

DMEMOFF 0x90000000
PMEMOFF 0xA0000000
CMEMOFF 0xB0000000
NMEMOFF 0x70000000
MFILES 200
GLTABSZ 300
RLTABSZ 300
MAXRLOCK 299

is there anyway to tweak uvconfig file or any other parameter and get same performance in 7.5.3?

vmcburney · Post by **vmcburney** » Sun Feb 19, 2012 4:57 pm

What about dsparams, is it the same between the two environments? Someone may have tweaked some memory settings in the DataStage Administrator in 7.5.1 that haven't carried over to 7.5.3.

qt_ky · Post by **qt_ky** » Sun Feb 19, 2012 6:52 pm

Speaking of tweaks, I've had to adjust several scripts' java memory settings in 8.x to prevent java out of memory problems. The point here would be to revisit any patches from 7.5.1 and any local fixes you may have applied to double check whether or not you need to apply the same into 7.5.3.

How about comparing all environment variables from each job run (2nd entry in job log)?

Maybe some fix in 7.5.3 simply requires more memory. Have you checked with support about it?

Since you're teetering on the very edge of what your server can handle, have you all considered adding more memory, or is it maxed out? I think it's time to have that discussion...

We have a Power7 system with 128 GB RAM. It's divided into LPARs, but I think it can go up to 1 TB RAM per unit.

Not that you asked, but considering your stage counts, I would guess there's room to optimize your job designs. Do you think they've already been fully optimized?

krishnakumar_m16 · Post by **krishnakumar_m16** » Mon Feb 20, 2012 10:58 am

Hi checked the DSParams file, both the instatllion have same values for all the environment variables.

chulett · Post by **chulett** » Mon Feb 20, 2012 11:05 am

Then it would seem you are firmly in the hands of your support provider.

Mike · Post by **Mike** » Mon Feb 20, 2012 12:52 pm

Your move to version 7.5.3 only buys you one extra year before you run up against the EOSL once more. You should start testing / planning for your move to version 8.7 or later. That move will be much more significant given the new 8.x architecture.

Mike

qt_ky · Post by **qt_ky** » Tue Feb 21, 2012 2:14 pm

And while you're at it, add more memory.