What happens in Pre-compilation?

bobyon · Post by **bobyon** » Fri Feb 03, 2012 8:40 am

I am sure this is a loaded question, but, generally speaking, what occurs during the pre-compilation of a parallel job?

The reason I ask is that most of our jobs go through precompilation in just a few seconds. However, I have seen several jobs that consistently take several minutes for precomp and then compile in just a handful of seconds. I've seen a couple that take nearly 10 minutes in precompilation.

I don't see anything significantly different about these jobs either. They are simple 3 or 4 stage jobs:

DB2PX --> Tx --> DB2 Connector

Just curious
Thanks

ray.wurlod · Post by **ray.wurlod** » Fri Feb 03, 2012 8:42 pm

I don't think it's ever been documented. I think of precompilation as a sanity check, for example making sure that no mandatory properties have been missed, that all the stages are connected properly, etc.

But with the relatively new Connector stage types, who knows? Maybe a lot more checking is done with those, because they can be used in multiple tools in the suite.

qt_ky · Post by **qt_ky** » Sat Feb 04, 2012 9:43 am

I've never seen precompilation take minutes. By any chance is your client remote or using a VPN? If not, I would open a support case because that is unusual behavior. You may find clues by running a few ISA Lite checks on the server, unzipping the file it creates, and looking through the HTML for warnings and errors.

ray.wurlod · Post by **ray.wurlod** » Sat Feb 04, 2012 4:19 pm

Pre-compilation of sequences can take a bit longer, because the existence of each of the referenced jobs and routines is checked, and connectivity to the client is also checked - part of sequence compilation actually occurs on the client.

qt_ky · Post by **qt_ky** » Sun Feb 05, 2012 7:49 am

There as a dscc Designer command line for compiling that you could test. I would expect it to have the same timings.

There is also the force compile option, which forces the compiler on the server to recompile any Transformer stage code. If you are patient, you could try using the force option to test if it will consistently take 10 minutes with those job types.

qt_ky · Post by **qt_ky** » Sun Feb 05, 2012 7:57 am

Here is a tech note describing an 8.1 patch for the symptoms you described:

http://www-01.ibm.com/support/docview.w ... wg1JR34531

If that describes you but you're on a different version or OS, then you can contact support and reference JR34531.

kandyshandy · Post by **kandyshandy** » Sun Feb 05, 2012 11:11 pm

Bob, Do you have XML stages (with plenty of columns defined) in those jobs? I recently worked for a project where we used plenty of XML stages and some of them took few mins for compilation. Anyways, 10 mins is something questionable!!

bobyon · Post by **bobyon** » Wed Feb 08, 2012 5:15 pm

kandyshandy wrote:Bob, Do you have XML stages (with plenty of columns defined) in those jobs?

Nope, no XML

I also tried a force compile earlier today and though I did not time it the precomp still took a long time.

And, I looked at the tech note referenced earlier. It seems that one was related to the actual compile, not precompile.

So I'm still impatiently waiting for my precompilations to complete.

qt_ky · Post by **qt_ky** » Sat Feb 11, 2012 7:43 pm

Test the local fix suggested in the tech note to see if it makes any difference in the wait time or consistency.

If not, then you need to open a support case while you're waiting...

bobyon · Post by **bobyon** » Mon Feb 13, 2012 10:48 am

I did not try the local fix as suggested yet, however, I am at the client site today and connected directly to their network as opposed to via VPN and the jobs are now precompiling and compiling within seconds as one would typically expect.

Not sure that helps to answer my question or solve anything, but it is certainly interesting. Must be a LOT of data transferring back and forth.

ray.wurlod · Post by **ray.wurlod** » Mon Feb 13, 2012 3:36 pm

You could always enable server-side tracing to see just how much traffic there is!

kandyshandy · Post by **kandyshandy** » Mon Feb 13, 2012 7:56 pm

Bob, To overcome this issue, try compiling the job from command line. We have faced this issue in the past and command line compilation was the remedy.