Parallel Vs Server Performance Test - Unexpected Results

Maveric · Post by **Maveric** » Wed Jan 30, 2008 1:03 am

In link sort in the target stage could be one reason. And when both the source and target are stages that run only in sequential mode, the idea of parallelism is lost. And more over the startup times are always more for parallel jobs and increase with the number of stages and number of node as apparent from your experience. But your run times show an abnormal (to me) increase in times. Probably look at the job log to see the start up times and run times and compare them. Try with a DataSet as source and target in the parallel job.

Maveric · Post by **Maveric** » Wed Jan 30, 2008 1:04 am

In link sort in the target stage could be one reason. And when both the source and target are stages that run only in sequential mode, the idea of parallelism is lost. And more over the startup times are always more for parallel jobs and increase with the number of stages and number of node as apparent from your experience. But your run times show an abnormal (to me) increase in times. Probably look at the job log to see the start up times and run times and compare them. Try with a DataSet as source and target in the parallel job.

balajisr · Post by **balajisr** » Wed Jan 30, 2008 1:22 am

What is the mode of operation of the aggregator stage? Is it Hash or Sort? Try using sort mode of operation.

Remove 'Perform sort' option in target sequential file stage. Use sorted merge collection method and verify the results.

What is your server load while executing these tests?

basav_ds · Post by **basav_ds** » Wed Jan 30, 2008 2:51 am

Aggt stage is working in Parallel execution mode, Partition type is 'SAME'.

I tried with removing 'Perform sort' option in target sequential file stage and Used sorted merge collection method but not getting sorted o/p.

Server load is zero during tests.

vmcburney · Post by **vmcburney** » Wed Jan 30, 2008 1:46 pm

What you are seeing is the overheads of partitioning and re-partitioning your data, which is why four nodes takes longer than two. I am surprised a single node is slower than a server job - my own benchmark shows parallel jobs sort and aggregate many times faster than server jobs. During the parallel sort it will write some data out to temporary files, I think it's in the temp directory, and it looks like you have inefficient file i/o. If you are on version 8 I would switch on some of the job monitoring features to see what is slowing your jobs down.