Job Run-Time is inconsistent, minutes can turn to hours
Moderators: chulett, rschirm, roy
So the mystery has been solved! It was not Datastage, not DB, not connectivity, not network speed. It was the priority setting for the scheduler! By default Windows 2008 R2 sets it to have a priority 7.
The net result for any tasks running with priority 7 is that Windows will arbitrarily decide to downgrade the CPU and disk i/o affinity for the task while it is running. This will happen even if the CPU is 99% idle. The affect can be seen in the Disk tab of System Resource Monitor. Normally this would just mean that a task ran slowly, but if that task is a .bat file running _progress or prowin32 connected in shared memory to a local database then the net result is catastrophic performance issues for all users and batches. Disk i/o for the database extent files will rocket to near 100% activity while bytes per second (and records per second in promon) will drop to almost nothing. Windows will upgrade and downgrade the affinity of the task while it is running making it look like the database has serious corruption or fragmentation issues.
We have spent countless hours investigating this from a database and application performance perspective before stumbling across a random post on another non-Progress forum similar to this one. Setting the Task priority to 4 (Normal) resolves this issue completely.
Hope this helps at least one person.
The net result for any tasks running with priority 7 is that Windows will arbitrarily decide to downgrade the CPU and disk i/o affinity for the task while it is running. This will happen even if the CPU is 99% idle. The affect can be seen in the Disk tab of System Resource Monitor. Normally this would just mean that a task ran slowly, but if that task is a .bat file running _progress or prowin32 connected in shared memory to a local database then the net result is catastrophic performance issues for all users and batches. Disk i/o for the database extent files will rocket to near 100% activity while bytes per second (and records per second in promon) will drop to almost nothing. Windows will upgrade and downgrade the affinity of the task while it is running making it look like the database has serious corruption or fragmentation issues.
We have spent countless hours investigating this from a database and application performance perspective before stumbling across a random post on another non-Progress forum similar to this one. Setting the Task priority to 4 (Normal) resolves this issue completely.
Hope this helps at least one person.
Last edited by epsidude on Thu Sep 12, 2013 9:00 am, edited 1 time in total.
