DSXchange

Are there any known issues with running DataStage 7.x on the big machines, i.e. HP SuperDomes and equivalent?

This question is based on the fact that we have sites that have purchased these monsters, and instead of experiencing an improvement in performance, the opposite is happening. We are seeing performance degradation of up to 80% in some cases.

The one is running on Win2000 Advanced Server and the other is running on HP/UX 11i.

If you run top then it may not be a CPU issue. It could be disk IO is not as good. It could be the network is setup poorly. It could be that these machines are designed to run lots of users and not a few large processes.

Thanks, now I know what kind of questions to ask.

I have yet to see much of anything 'super' about our SuperDomes.

What are the specs for your machine - CPUs, ram, etc? What other things are running at the same time as your DataStage jobs? And as Kim notes, there are many factors involved and disk I/O issues can really put a kink in your hose.

At the one site, we have the following:

The machine is dedicated to DS.
All the data is extracted from source systems and dumped on this box for processing. The resulting files are copied to the machine where the DB is sitting, and then scripts are run on that machine to load the data.

Hardware Platform - Netra-T12
Disk Size - 1.4 Terabytes
Disk set - Raid 5
Memory Size - 16384 MB
Operating System - SunOS 5.9
Type of CPU(s)/Number of CPU(s) - 4 x 1200 mHz (Dual-Core)
DataStage Version - 7.1.0.13

It sounds, from your description, that the network is not the issue since you are processing files that are deposited on the box and then DS processes these files as load files on the same server and then some other process moves them to the DB server. Is this correct? Give us some info about the types of files you are processing - like size, width, format etc...

[quote="mhester"]Give us some info about the types of files you are processing - like size, width, format etc...[/quote]

Most of the files are EBCIDIC Binary. We use CFFs as source stages.
Even though some of the files are very wide > 200 cols, we only extract the fields we need. One of the files has on average 2mil recs everyday.

All the other files are straight sequential files.

The biggest issue we have is speed. Its inconsistent. One day you process at 7000 rows/sec next day you process at 200 rows/sec.

From a unix command line execute:
Monitor your processes, see if your cpus are totally utilized. Until you can qualify your resource availabilities, you can't talk about why things are slow. If all cpus are being used, then there's little you can do to speed things up except remove cohabitating processes/applications.

This is a quote from the client

"Jobs running slow even when the CPU is 98% idle"

What would cause this?

You need to follow Ken's advice to determine CPU usage. There may be other processes/services running on this server that are consuming resources. When you state that one day it runs 7000 rows/sec and the next day 200 rows/sec this tells me that DS is not likely the culprit, but rather something else is happening that is affecting server performance (like backups etc...)

I agree with Michael. Describe your disk storage. Do you have Veritas or some kind of virtual filesystem?

Describe your network and the network path from your server to source systems and target systems.

Will get back to you when I have answers to these questions.

Thanx,

Raid 5 is not the best storage set up for writing - Raid 0 + 1 would be the best set up. Are your files spread around the disks or are the controllers trying to access the same disks at the same time. Again Raid 0 + 1 will randomise this and reduce disk contention.

Hi Precious,

This is a long shot, but...
One thing you may want to check is your hashed files. When we migrated from Windows NT to 2000, we used a CompileAll utitlity but it didn't seem to work for us on DS 5.1 when it came to hashed files. When we went to run a job on Win2000, instead of creating three files for a hashed file (like usual), the jobs started to create one hash file for each record. Needless to say, we saw a great reduction in response time. But, when we individually compiled the jobs and removed the hash files (so they would start fresh), then everthing worked fine - and with even better performance than we had on NT.

Scott

You might consider that some characteristic of the data is changing from day to day. The more complex your transformation job the more likely a change in the data could impact performance. Have you tried rerunning the slow performing jobs. If you find that a single job runs inconsistently from one time to another, then it is resource contention. However if it consistently runs slow, then you have something to work with. In the past I have seen database loads run along fine and then slow down because the data went from being very ordered to completely random. The load slowed because the database buffers were constantly swapping out data. I've also seen a load slow because a specific patch of records in the data just required much more work than the average record.