Slow to compile Routines and to delete anything

[rleishman]

It takes about 30sec to 2mins to compile ANY (single) routine on our server. Job and Sequence compilation is fine (about 10sec for a job with b/w 2-10 stages).

In fact, just about anything you do working with routines is slow: renaming, deleting, compiling. Deleting a job or sequence incurs about the same performance.

I suspect that DS is doing some sort of full scan of the repository so that it can invalidate jobs that reference the job/routine you are deleting/compiling.

DS is on a dedicated stand-alone Unix server, and I am the only one logged into the server. This is not a resource contention issue. Look, I'll show you:
ps -ef | grep ds

Code: Select all

root     25678     1  0 Sep13 ?        00:00:01 /opt/Ascential/DataStage/DSEngine/bin/dsrpcd
c985675  30616 25678  0 00:40 ?        00:00:11 dscs 4 0 0
c985675  30617 30616  2 00:40 ?        00:02:23 dsapi_slave 7 6 0
c985675  31910 25678  0 01:57 ?        00:00:04 dscs 4 0 0
c985675  31914 31910  1 01:57 ?        00:00:29 dsapi_slave 7 6 0
c985675  32354 28222  0 02:26 pts/0    00:00:00 grep ds

uptime

Code: Select all

  2:24am  up 10 days 21:27,  1 user,  load average: 0.21, 0.13, 0.10

The performance of these actions never varies (ie. it never gets better). As I mentioned above, Job compile is fine, and job run-time performance is also great (50-400 rows/sec).

It has been like this ever since we started with DS, although it is getting worse as our repository grows. We now have about 50 routines, 150+ server jobs, and 150+ job sequences.

[ray.wurlod]

My guess is that you have a crowded project (lots of jobs, sequences and routines). That is, I agree with your conclusion.

Part of compiling or deleting a routine is a complete usage analysis so that you can be warned that other components use the routine. The check has to be done whether or not this ends up being true.

That's one of the reason the "best practice" is to have as few jobs as possible in the project. Maybe have more projects.

[rleishman]

Thanks Ray.

Problem is the whole thing is inter-related - believe it or not this is all related to a single client project. I have lots of job sequences because there are small ad-hoc streams that just run on request, but the big overnight run does EVERYTHING with dependencies all the way down the line.

My understanding is that projects are insular - a job sequence cannot call a job outside the project (without a very nasty OS call). Is this correct?

[ray.wurlod]

Yep

Maybe you need a new contract where you're paid by the hour to wait for these usage analyses!

[chulett]

Well... those 'Referencing' options can be turned off in the Manager. And it does tend to speed up compiling just a wee bit when 'Warn if compiling referenced routine' is unchecked.

[rleishman]

A "wee bit"!!!!!

I s'pose Katrina did a "tad" of damage in New Orleans, and Bush is under a "smidgeon" of pressure over it!!!

Very nice Craig, thanks. Routine compile times down to sub 1 second.

Does anyone (who knows) actually leave these options turned on? I thought it was Ascential's Machiavellian humour to spend 2 minutes determining that other jobs use the routine without actually telling you which ones

[chulett]

You are welcome.