QS Deduplication Job taking 2jrs for 34000 records
QS Deduplication Job taking 2jrs for 34000 records
Hi all,
I have a job which uses QS PLugin for DS to run a QS job within DS.
Its a Undup Job with 1 input file and 2 output files. It has 6 passes within the job. Strangely it never took that much time in our Quality and Dev environment but in Production it is taking 2hrs for Deduplicating 34000 records.
How can I analyse where the problem lies since QS doesnt provide me any monitoring system.
Please give your valuable suggestions on this topic.
Thanks,
SM
I have a job which uses QS PLugin for DS to run a QS job within DS.
Its a Undup Job with 1 input file and 2 output files. It has 6 passes within the job. Strangely it never took that much time in our Quality and Dev environment but in Production it is taking 2hrs for Deduplicating 34000 records.
How can I analyse where the problem lies since QS doesnt provide me any monitoring system.
Please give your valuable suggestions on this topic.
Thanks,
SM
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
Your QualityStage text log file will tell you a lot, it has an entry for each step with a timestamp so you can roughly tell how long each step is taking (I hope!). Strange that it is slower in prod, are any traces turned on?
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If you really want to nail it you could modify the scripts that QualityStage runs, to add timing points to them. This would help you to identify the hot spots. How may rows were processed in the development environment? What other activities were occurring at the time on the production machine?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You must be the detective - I'm not there. Take the QS pieces out of the DS job and run some timings. Run the QS jobs separately and get more timings. I doubt that qsrtmngr is likely to be the bottleneck, but you could trace that also. Monitor the system while all these things are happening, particularly %Idle for CPU, PF/S for memory. Monitor both systems, in case this shows up any discrepancies.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi Ray,
I run the QS job independently thru QS designer and it executes within few seconds for 34000 recs. So I assume that the QS is not giving problems.
Ascential Support replies: "Then it has to be the server differences with the id's being used. When certain id's are used on the particular servers the user id invokes default settings. Swap space, etc.."
I am confused but the problem still persist.
Pls advice.
Thanks,
SM
I run the QS job independently thru QS designer and it executes within few seconds for 34000 recs. So I assume that the QS is not giving problems.
Ascential Support replies: "Then it has to be the server differences with the id's being used. When certain id's are used on the particular servers the user id invokes default settings. Swap space, etc.."
I am confused but the problem still persist.
Pls advice.
Thanks,
SM
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
This is where you have to begin being a real detective. Maybe a good first step is to use a before/after subroutine to report who you are, your environment, and so on. If you have MKS Toolkit you can use UNIX commands like id and env to report these factors. Look for differences.
Similarly look for free disk space differences. QualityStage is particularly hungry for disk resources. Get your administrator to monitor disk I/O (you can do this with Task Manager, on the Processes tab choose View > Select Columns... - or use Performance Monitor software. Again you're looking for differences between the two systems while the two jobs are running. And take snapshots of disk free space while the jobs are running on each system - use the $DSHOME\bin\avail command for this or, if you have MKS Toolkit, use the df -kPt command.
Similarly look for free disk space differences. QualityStage is particularly hungry for disk resources. Get your administrator to monitor disk I/O (you can do this with Task Manager, on the Processes tab choose View > Select Columns... - or use Performance Monitor software. Again you're looking for differences between the two systems while the two jobs are running. And take snapshots of disk free space while the jobs are running on each system - use the $DSHOME\bin\avail command for this or, if you have MKS Toolkit, use the df -kPt command.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi Ray,
Just now I tried running the QS job from a DS job as Before Job subroutine instead of using a DS job with QS plugin. It runs in 1 min tme.
Also, my server is on Windows server and I cannot using Unix commands.
I also checked the server performance...its is optimum as desired since no other application runs on the system. If the server is a problem then why only for this job which takes 4rows per sec as compared to pther jobs which take some 200rows/sec.
Thx,
SM
Just now I tried running the QS job from a DS job as Before Job subroutine instead of using a DS job with QS plugin. It runs in 1 min tme.
Also, my server is on Windows server and I cannot using Unix commands.
I also checked the server performance...its is optimum as desired since no other application runs on the system. If the server is a problem then why only for this job which takes 4rows per sec as compared to pther jobs which take some 200rows/sec.
Thx,
SM
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Hi Ray and all,
Finally running around the bush will not help...I have realised..Just lack of Documentation and experience will make you do this...
The issue is resolved.
Solution:
When I transported the DS job which uses QS plugin, the Host system within QS plugin was still pointing to the same old D machine instead of P. We need to change the host system in QS plugin manually after transport to P or to any new machine. I assumed that QS plugin would take the information from the run profile which i configured properly. The job was running at 4 rows/sec but did not abort as I had shared the D machine on my P environment i suppose. I am not sure. But now after changing the Host IP in QS plugin, the job runs at 1000rows/sec.
Hope this info helps everyone who is new to usage of QS plugin.
Thanks,
SM
Finally running around the bush will not help...I have realised..Just lack of Documentation and experience will make you do this...
The issue is resolved.
Solution:
When I transported the DS job which uses QS plugin, the Host system within QS plugin was still pointing to the same old D machine instead of P. We need to change the host system in QS plugin manually after transport to P or to any new machine. I assumed that QS plugin would take the information from the run profile which i configured properly. The job was running at 4 rows/sec but did not abort as I had shared the D machine on my P environment i suppose. I am not sure. But now after changing the Host IP in QS plugin, the job runs at 1000rows/sec.
Hope this info helps everyone who is new to usage of QS plugin.
Thanks,
SM
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: