DSXchange

We are migrating DataStage 7.5 server on Windows to 8.5 Enterprise on Solaris. This is a phased project. At phase one, we only do migration from 7.5 server jobs on windows to 8.5 server jobs on solaris, without any change of job design. At phase two, we will do performance tuning and rewrite server jobs into parallel jobs.

Now it's the end of phase one and we are suffering from worse performance of server jobs on solaris compared to windows. The unix server is Sun M4000, 64GB RAM, 1 * CPU Board (2 * 4 core, 2.66GHz SPARC64 VII+ CPUs). Windows server has 8 cores AMD opteron 2.4GHz CPUs on HP Proliant 880, 16GB RAM, a 5-year model. Our unix admin comments that because the jobs we run are mostly single-thread design, hence they are not taking advantage of parallel power from unix server. Furthermore, x64 CPUs perform better than Sun CPUs on single-thread jobs, even it's a 5-year model, regardless of CPU speed. It sounds like CISK vs RISC issue,

According to unix admin's comments, we are making wrong decision of migration, or says, IBM & consultants gave us wrong recommendation of migration before we started this project. We shouldn't do phase one becaues there is no benefit, however, due to limitation of resource, we do phase one as IBM & consultants said new Solaris server must be faster than windows server in terms of running server jobs.

Can we say, DataStage server jobs run faster on windows than solaris, based on similar spec of hardware, because of single-thread job natural?

No, because there are just too many other variables. Disk topology (mix of local and remote) and disk device speed, to name just one category.

Have you tried to narrow down exactly why your performance is poor on the Solaris box? What aspect(s) of the jobs you're running have been affected - read speed? Write speed? It is specific to your hashed files, for instance or has access times to remote relational databases taken a hit? You really need to quantify the nature of the 'performance issue' as (as Ray notes) there are ton of factors at play here.

There is also the significant architectural change with the 8.5 release, so realize you're not just simply comparing 'Windows v. UNIX' here.

Have you compared any settings across servers, like the uvconfig file or the DataStage Administrator settings per project? If someone tweaked settings on Windows and you have default settings on Unix, seems like that is another factor that could slow down server jobs, not because of the OS difference.

I understand it's not simply windows vs unix scenario. We have done heaps of testing and monitoring. All stats show that disk /IO or performance is great, network performace is great, and other indicators say unix server is in very good condition. However, it just runs slower than windows in terms of server jobs.

Oracle(Sun) also help us to diagnose but there is no encouraging result. Only thing we assure is CPU utilisation is 100% on 1 single CPU while other 7 ones are nearly idle when we are running testing jobs.

The same jobs run on windows show the same CPU utlisation but runing faster. And it's very hard for us to convince management we are making the right decision.

If i am right this is relates to configuration in DS. I heard, while do the installation you can configure (Even after i guess) in this regards.

You can raise it as a PMR. You will get solution.


DS User

Thanks for all replies. This issue has bothered us for couple weeks. We did any necessary tweak and also raised PMR to IBM, but so far, no much improvement. We did get some performance gain from switching ZFS to UFS for hashed files location. However, the improvement is not good enough to fill the gap.

Now we don't expect server jobs can run faster on solaris than windows but at least, shuoldn't be worse. It's embarrassing that we spend big money on new hardware and software but it results in worse performance.

azens wrote:We did get some performance gain from switching ZFS to UFS for hashed files location.

Ah... I was wondering if this was you but I hadn't checked the older posts yet.

chulett wrote:

azens wrote:We did get some performance gain from switching ZFS to UFS for hashed files location.

Ah... I was wondering if this was you but I hadn't checked the older posts yet.

Yes, that's me. In the meanwhile, we start to convert some jobs in critical path into parallel and also get IBM to our site to investigate.

There is no guarantee whatsoever that parallel jobs are faster than server jobs, particularly for small to medium data volumes.

Exactly, make sure these 'critical path' Server jobs are actually appropriate for assimilation into the PX collective.

So you are seeing a CPU bottleneck?

Is that normal for all server jobs to run on a single CPU or core? If you start a second server job while the first server job is running on one CPU at 100%, then would you expect the second server job to land on a different CPU or core?

If the operating system behaves conventionally (and since DataStage jobs are processes) that's what I'd expect.

Have you analyzed what part of the overall job process is slower in the new environment?

How do you run your jobs? Are there any scripts, before job/after job routines, etc., which may call ds server commands such as dsadmin?

Regards,