Performance Issues with DS7.x

Precious · Post by **Precious** » Fri Apr 22, 2005 5:33 am

Are there any known issues with running DataStage 7.x on the big machines, i.e. HP SuperDomes and equivalent?

This question is based on the fact that we have sites that have purchased these monsters, and instead of experiencing an improvement in performance, the opposite is happening. We are seeing performance degradation of up to 80% in some cases.

The one is running on Win2000 Advanced Server and the other is running on HP/UX 11i.

kduke · Post by **kduke** » Fri Apr 22, 2005 6:06 am

If you run top then it may not be a CPU issue. It could be disk IO is not as good. It could be the network is setup poorly. It could be that these machines are designed to run lots of users and not a few large processes.

Precious · Post by **Precious** » Fri Apr 22, 2005 6:16 am

Thanks, now I know what kind of questions to ask.

chulett · Post by **chulett** » Fri Apr 22, 2005 7:14 am

I have yet to see much of anything 'super' about our SuperDomes.

What are the specs for your machine - CPUs, ram, etc? What other things are running at the same time as your DataStage jobs? And as Kim notes, there are many factors involved and disk I/O issues can really put a kink in your hose.

Precious · Post by **Precious** » Fri Apr 22, 2005 7:56 am

At the one site, we have the following:

The machine is dedicated to DS.
All the data is extracted from source systems and dumped on this box for processing. The resulting files are copied to the machine where the DB is sitting, and then scripts are run on that machine to load the data.

Hardware Platform - Netra-T12
Disk Size - 1.4 Terabytes
Disk set - Raid 5
Memory Size - 16384 MB
Operating System - SunOS 5.9
Type of CPU(s)/Number of CPU(s) - 4 x 1200 mHz (Dual-Core)
DataStage Version - 7.1.0.13

mhester · Post by **mhester** » Fri Apr 22, 2005 8:06 am

It sounds, from your description, that the network is not the issue since you are processing files that are deposited on the box and then DS processes these files as load files on the same server and then some other process moves them to the DB server. Is this correct? Give us some info about the types of files you are processing - like size, width, format etc...

Precious · Post by **Precious** » Fri Apr 22, 2005 8:25 am

[quote="mhester"]Give us some info about the types of files you are processing - like size, width, format etc...[/quote]

Most of the files are EBCIDIC Binary. We use CFFs as source stages.
Even though some of the files are very wide > 200 cols, we only extract the fields we need. One of the files has on average 2mil recs everyday.

All the other files are straight sequential files.

The biggest issue we have is speed. Its inconsistent. One day you process at 7000 rows/sec next day you process at 200 rows/sec.

kcbland · Post by **kcbland** » Fri Apr 22, 2005 8:38 am

From a unix command line execute:

Code: Select all

$ prstat -a

Monitor your processes, see if your cpus are totally utilized. Until you can qualify your resource availabilities, you can't talk about why things are slow. If all cpus are being used, then there's little you can do to speed things up except remove cohabitating processes/applications.

Precious · Post by **Precious** » Fri Apr 22, 2005 8:41 am

This is a quote from the client

"Jobs running slow even when the CPU is 98% idle"

What would cause this?

mhester · Post by **mhester** » Fri Apr 22, 2005 9:27 am

You need to follow Ken's advice to determine CPU usage. There may be other processes/services running on this server that are consuming resources. When you state that one day it runs 7000 rows/sec and the next day 200 rows/sec this tells me that DS is not likely the culprit, but rather something else is happening that is affecting server performance (like backups etc...)

kduke · Post by **kduke** » Fri Apr 22, 2005 10:13 am

I agree with Michael. Describe your disk storage. Do you have Veritas or some kind of virtual filesystem?

Describe your network and the network path from your server to source systems and target systems.

Precious · Post by **Precious** » Mon Apr 25, 2005 1:15 am

Will get back to you when I have answers to these questions.

Thanx,

palmeal · Post by **palmeal** » Mon Apr 25, 2005 1:49 am

Raid 5 is not the best storage set up for writing - Raid 0 + 1 would be the best set up. Are your files spread around the disks or are the controllers trying to access the same disks at the same time. Again Raid 0 + 1 will randomise this and reduce disk contention.

scottopizza · Post by **scottopizza** » Fri May 13, 2005 12:40 pm

Hi Precious,

This is a long shot, but...
One thing you may want to check is your hashed files. When we migrated from Windows NT to 2000, we used a CompileAll utitlity but it didn't seem to work for us on DS 5.1 when it came to hashed files. When we went to run a job on Win2000, instead of creating three files for a hashed file (like usual), the jobs started to create one hash file for each record. Needless to say, we saw a great reduction in response time. But, when we individually compiled the jobs and removed the hash files (so they would start fresh), then everthing worked fine - and with even better performance than we had on NT.

Scott

alhamilton · Post by **alhamilton** » Fri May 13, 2005 1:20 pm

You might consider that some characteristic of the data is changing from day to day. The more complex your transformation job the more likely a change in the data could impact performance. Have you tried rerunning the slow performing jobs. If you find that a single job runs inconsistently from one time to another, then it is resource contention. However if it consistently runs slow, then you have something to work with. In the past I have seen database loads run along fine and then slow down because the data went from being very ordered to completely random. The load slowed because the database buffers were constantly swapping out data. I've also seen a load slow because a specific patch of records in the data just required much more work than the average record.