Job compilation takes about 4 hrs
Moderators: chulett, rschirm, roy
Job compilation takes about 4 hrs
Job compilation takes about 4 hrs, is there any way to reduce the compilation time?
There are 1145 columns in the input file and all the columns are used for transformation.
There are 1145 columns in the input file and all the columns are used for transformation.
Historically the transform stages have been a cause for delayed compile times, but nothing on the order of 4 hours. Some process must be busy during this extended period, so please check the following:
1. How long do other jobs take to compile?
2. While compiling, check your PC's I/O and CPU usage
3. While compiling, check the DataStage Server's CPU usage.
1. How long do other jobs take to compile?
2. While compiling, check your PC's I/O and CPU usage
3. While compiling, check the DataStage Server's CPU usage.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
datisaq - I use the windows taskmgr to see the client PC usage and have sometimes seen the local CPU and particularly the I/O go up significantly during compiles, this is because parts of the compilation are actually done on the PC and not on the server and much data can be passed between the two, hence looking at I/O.
CPU usage on a UNIX server is determined by seeing if a compile process (identified via "ps -ef") is using a lot of CPU for a long period of time. I also use vmstat (most UNIX implementations have their own proprietary tool for detailed measurement which I'll use if I can, but vmstat and iostat are common across all UNIX flavors and give a good overview).
If any compile uses a lot of resources for 4 hours then something is seriously wrong; if the compile takes 4 hours and consumes little or no resources, then something is also seriously wrong. Either way, one needs to collect some statistics first.
CPU usage on a UNIX server is determined by seeing if a compile process (identified via "ps -ef") is using a lot of CPU for a long period of time. I also use vmstat (most UNIX implementations have their own proprietary tool for detailed measurement which I'll use if I can, but vmstat and iostat are common across all UNIX flavors and give a good overview).
If any compile uses a lot of resources for 4 hours then something is seriously wrong; if the compile takes 4 hours and consumes little or no resources, then something is also seriously wrong. Either way, one needs to collect some statistics first.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Unless your server is an old 386SX processor with the turbo switch turned off, or it is extremely overloaded with other process then 3 transforms and 3 lookups shouldn't take that long!
As in my earlier response to datiseq, please measure your system loads to see if you can identify the bottleneck.
As in my earlier response to datiseq, please measure your system loads to see if you can identify the bottleneck.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>