QualityStage plugin for DataStage uvsh.exe performance issue

[caharris]

Hi All,

This is my first post so please be gentle.

Synopsis:
Job Design
ODBC -> Transformer -> Seq (fixed length) -> QS Plugin (STAN) -> Seq (fixed length)
The issue I have relates to poor performance once job reaches the QualityStage plugin stage in DataStage. The stage is simpy a standardise stage using AUNAME ruleset. Job execution time taking 2.5 - 3.0 times longer than raw QualityStage job
Tracelevels are set to 0 in all sections of mngr.cfg and in the properties QualityStage plugin stage.
Sequence of events:
1. uvsh.exe starts using about 40% CPU and ODBC and 1.47 million rows are transferred to seq file - no probs here (have experimented with both and remote files - makes no difference to performance issue in next couple of steps)
2. once file transferred uvsh.exe then ramps up to about 70%, and memory usage up around 1.5 times the size of seq file just created. This is the part A of the perfomance issue
3. 30 mins later... uvsh.exe settles down to < 1% on DataStage server and qsrtserver.exe for small row count (or stn.exe for larger row count - not sure why this happens either?) starts up on remote QualityStage server. This process takes the usual amount of time to run. Have experimiented with multiple instances and they do reduce this time to some degree.
4. qsrtserver.exe or stn.exe finishes up on QualityStage server and then that bl**dy uvsh.exe starts up again (for another 30 mins) on DataStage server using same resources as in step 2. Finally, at the end of this the standardised seq fixed length file is written (remote).
Have tried it with QualityStage and DataStage server both on same box with the intent to set Server Mode property to 'Local'. Though this results the following error message: "src_CMS_CRM_PERSON..Standardise_CRM_PERSON: Invalid value for stage property 'Server Mode': 'Local' and 'Local' are different" Mmmmm... 'Local' and 'Local' are different... right! now that makes sense.
So my friends, can someone please convince me this plugin is worth continuing with or do I go back to a using both DataStage and QualityStage independently... Urrrrr *shivers*

Cheers,

Cameron :D

[vmcburney]

Wish I had a dollar for every time a QualityStage job runs better when executed from the client then from a plugin or command line. First check your data directory, is the job running in stream mode? Is it trying to write out a data file for each step or carrying the data through in memory? Second check that your plugin trace level is set to 0. Check the log file in the log directory for timestamps on the steps. Run in a higher trace mode to try and find out which step is slow.