Job compilation takes about 4 hrs

ethelvina · Post by **ethelvina** » Tue Jun 15, 2010 5:36 am

Job compilation takes about 4 hrs, is there any way to reduce the compilation time?
There are 1145 columns in the input file and all the columns are used for transformation.

ArndW · Post by **ArndW** » Tue Jun 15, 2010 6:24 am

Historically the transform stages have been a cause for delayed compile times, but nothing on the order of 4 hours. Some process must be busy during this extended period, so please check the following:

1. How long do other jobs take to compile?
2. While compiling, check your PC's I/O and CPU usage
3. While compiling, check the DataStage Server's CPU usage.

chulett · Post by **chulett** » Tue Jun 15, 2010 6:25 am

It always takes four hours or has on occassion taken that long?

datisaq · Post by **datisaq** » Tue Jun 15, 2010 7:21 am

Hi Arndw,
Can you please tell how to check the CPU usage and I/O usage at compile time?

Regards
datisaq

ArndW · Post by **ArndW** » Tue Jun 15, 2010 8:05 am

datisaq - I use the windows taskmgr to see the client PC usage and have sometimes seen the local CPU and particularly the I/O go up significantly during compiles, this is because parts of the compilation are actually done on the PC and not on the server and much data can be passed between the two, hence looking at I/O.
CPU usage on a UNIX server is determined by seeing if a compile process (identified via "ps -ef") is using a lot of CPU for a long period of time. I also use vmstat (most UNIX implementations have their own proprietary tool for detailed measurement which I'll use if I can, but vmstat and iostat are common across all UNIX flavors and give a good overview).
If any compile uses a lot of resources for 4 hours then something is seriously wrong; if the compile takes 4 hours and consumes little or no resources, then something is also seriously wrong. Either way, one needs to collect some statistics first.

chulett · Post by **chulett** » Tue Jun 15, 2010 9:18 am

Any chance of answering my question? Wondering if this is a single-use-license-queueing issue.

agpt · Post by **agpt** » Tue Jun 15, 2010 11:04 am

ArndW wrote:datisaq - I use the windows taskmgr to see the client PC usage and have sometimes seen the local CPU and particularly the I/O go up significantly during compiles, this is because parts of the compilat ...

What does "datisaq" means here

anbu · Post by **anbu** » Tue Jun 15, 2010 11:08 am

datisaq is the username.

ArndW was replying to datisaq's post above

agpt · Post by **agpt** » Tue Jun 15, 2010 11:11 am

anbu wrote:datisaq is the username.

ArndW was replying to datisaq's post above

Oh Sorry... I didn't notice that.

ray.wurlod · Post by **ray.wurlod** » Tue Jun 15, 2010 4:49 pm

I would still like to have an answer to the question about whether it always takes four hours or whether this was a one-off.

chulett · Post by **chulett** » Tue Jun 15, 2010 5:04 pm

Me, too.

ethelvina · Post by **ethelvina** » Tue Jun 15, 2010 8:57 pm

Only this job takes a long time to compile, also this job always takes about 4 hrs to compile.Even import of this job takes about 30 mins.

chulett · Post by **chulett** » Tue Jun 15, 2010 9:41 pm

OK... can you describe the job in more detail? For example, how many transformers does it include? I'm guessing... more than one.

ethelvina · Post by **ethelvina** » Tue Jun 15, 2010 10:36 pm

There are 3 transformers and 3 lookup stages.Input dataset has about 1145 columns and there are 11 output datasets, almost all of the input columns are required.

ArndW · Post by **ArndW** » Wed Jun 16, 2010 1:37 am

Unless your server is an old 386SX processor with the turbo switch turned off, or it is extremely overloaded with other process then 3 transforms and 3 lookups shouldn't take that long!

As in my earlier response to datiseq, please measure your system loads to see if you can identify the bottleneck.