Long compile times under USS

RobJ · Post by **RobJ** » Tue Oct 25, 2005 2:23 am

We're running DataStage EE under USS on the mainframe and my developers are complaining about the length of time it takes to compile parallel jobs.

The mainframe is quite heavily loaded, but the time required does seem excessive (up to half an hour for jobs with 20 or so stages in them).

We have increased scheduling priority via WLM (Workload Manager), which helped a bit, but it's still pretty bad.

Can anyone suggest further optimisation which we could try?

Thanks for any help.

Rob.

ArndW · Post by **ArndW** » Tue Oct 25, 2005 3:06 am

RobJ,

have you noticed a difference between jobs with and without transform stages? Oftentimes a job (even a very complex one) with no transform will compile quickly whilst a simple job with a transform might take a long time. If this is the case, it might have to do with the c++ compiler.

RobJ · Post by **RobJ** » Tue Oct 25, 2005 3:31 am

ArndW wrote:RobJ,

have you noticed a difference between jobs with and without transform stages? Oftentimes a job (even a very complex one) with no transform will compile quickly whilst a simple job with a transform might take a long time. If this is the case, it might have to do with the c++ compiler.

Yes, it does appear to be the Transformers which take the time.

What, if anything, can be done to speed up the C++ compiler?

We're invoking the compiler like this:

c++ -+ -DRW_COMPILE_INSTANTIATE -D_OS390 -D_ALL_SOURCE -DYYDYNAMIC -D_M
SE_PROTOS -Wc,DLL,EXPORTALL -W "0,langlvl(LONGLONG)" -D_SHR_ENVIRON -D_OPEN_SOUR
CE=3 -Wc,"float(IEEE)" -DIEEE_754=1 -D_AE_BIMODAL -D__IOC_ANSI_STREAM -D_XOPEN_S
OURCE_EXTENDED -c

and the linker like this

c++ '-+ -D_OS390 -D_ALL_SOURCE -DYYDYNAMIC -D_MSE_PROTOS -Wc,DLL,EXPORTALL
-Wl,dll -W "0,langlvl(LONGLONG)" -D_SHR_ENVIRON -D_OPEN_SOURCE=3 -Wc,"float(IEEE
)" -DIEEE_754=1 -D_AE_BIMODAL -D__IOC_ANSI_STREAM -D_XOPEN_SOURCE_EXTENDED

(Obviously, the command lines are wrapping here.)

Any suggestions?

Rob.

ArndW · Post by **ArndW** » Tue Oct 25, 2005 3:36 am

I wish I did have suggestions. I think the compiler is licensed by number of concurrent users, and processes will wait for a free slot when that number is reached; so perhaps part of the long compile times might result from this bottleneck. But this is conjecture on my part and might be completely wrong.

Eric · Post by **Eric** » Tue Oct 25, 2005 4:03 am

You can compile the transformer directly on the Z/os from the command line. (DataStage can generate you a script) You can then see how long that part of the compile takes and what difference tweeks on the mainframe make.

(check the options on the remote tab for generating deployment scripts)

RobJ · Post by **RobJ** » Tue Oct 25, 2005 4:30 am

ArndW wrote:I wish I did have suggestions. I think the compiler is licensed by number of concurrent users, and processes will wait for a free slot when that number is reached; so perhaps part of the long compile times might result from this bottleneck. But this is conjecture on my part and might be completely wrong.

I've been digging a bit further and although the time is related to compiling transformers, it's not actually the C++ compiler, which provides the sub-second response times expected.

The time is all "startup time" for the DataStage process which actually submits the compile jobs. (If you look in the job directory under USS you'll see a bunch of files related to the transformers with extensions such as .trx, .trx.osh and .trx.sh

The .trx.sh scripts are run as part of the job compilation, invoking osh to process the transformers. There seems to be a huge overhead (over a minute) here.

So, the question now becomes: how to optimise the osh startup time?

Rob.

ray.wurlod · Post by **ray.wurlod** » Tue Oct 25, 2005 4:51 am

That, dare I suggest, is an issue for the vendor. How about lodging an enhancement request? They, too, are new to USS, and doubtless are looking for ideas about improving things.

ArndW · Post by **ArndW** » Tue Oct 25, 2005 6:05 am

The problem is (almost) the same on busy PX sites - the last one I was at had similar wait times for compilation of jobs with transform stages. I think the approach DS takes is the same in both implementations, so hopefully by making the compile quicker on native unix it will also speed up the virtual unix one.