DSXchange

Job compilation takes about 4 hrs, is there any way to reduce the compilation time?
There are 1145 columns in the input file and all the columns are used for transformation.

Historically the transform stages have been a cause for delayed compile times, but nothing on the order of 4 hours. Some process must be busy during this extended period, so please check the following:

1. How long do other jobs take to compile?
2. While compiling, check your PC's I/O and CPU usage
3. While compiling, check the DataStage Server's CPU usage.

It always takes four hours or has on occassion taken that long?

Hi Arndw,
Can you please tell how to check the CPU usage and I/O usage at compile time?


Regards
datisaq

datisaq - I use the windows taskmgr to see the client PC usage and have sometimes seen the local CPU and particularly the I/O go up significantly during compiles, this is because parts of the compilation are actually done on the PC and not on the server and much data can be passed between the two, hence looking at I/O.
CPU usage on a UNIX server is determined by seeing if a compile process (identified via "ps -ef") is using a lot of CPU for a long period of time. I also use vmstat (most UNIX implementations have their own proprietary tool for detailed measurement which I'll use if I can, but vmstat and iostat are common across all UNIX flavors and give a good overview).
If any compile uses a lot of resources for 4 hours then something is seriously wrong; if the compile takes 4 hours and consumes little or no resources, then something is also seriously wrong. Either way, one needs to collect some statistics first.

Any chance of answering my question? Wondering if this is a single-use-license-queueing issue.

ArndW wrote:datisaq - I use the windows taskmgr to see the client PC usage and have sometimes seen the local CPU and particularly the I/O go up significantly during compiles, this is because parts of the compilat ...

What does "datisaq" means here

datisaq is the username.

ArndW was replying to datisaq's post above

anbu wrote:datisaq is the username.

ArndW was replying to datisaq's post above

Oh Sorry... I didn't notice that.

I would still like to have an answer to the question about whether it always takes four hours or whether this was a one-off.

Me, too.

Only this job takes a long time to compile, also this job always takes about 4 hrs to compile.Even import of this job takes about 30 mins.

OK... can you describe the job in more detail? For example, how many transformers does it include? I'm guessing... more than one.

There are 3 transformers and 3 lookup stages.Input dataset has about 1145 columns and there are 11 output datasets, almost all of the input columns are required.

Unless your server is an old 386SX processor with the turbo switch turned off, or it is extremely overloaded with other process then 3 transforms and 3 lookups shouldn't take that long!

As in my earlier response to datiseq, please measure your system loads to see if you can identify the bottleneck.