I have read several topics and the Parallel Job Developer's Guide to understand there are several things that are done when a parallel job starts up. What I would like to know is what people have done to reduce startup time short of redesigning the job? The developer's are saying it is taking 15-20 minutes for a parallel job to start.
I do know that transformations and multiple nodes will slow down the start up. What else have you experienced that slows down the startup?
Parallel job startup time
Moderators: chulett, rschirm, roy
Parallel job startup time
Leroy Gharis
Dallas, TX
Dallas, TX
A startup time of 15-20 minutes is quite long. It probably includes the time required for the lookups used (be they lookup stages to databases or to PX files) to be loaded to memory.
The actual startup times for the processes to load (which increases as the number of nodes and stages goes up) will be in the range of several seconds to somewhere around a minute - you should check to see how many processes your job has at runtime by using APT_DUMP_SCORE.
Transformations do not slow down the startup time.
The actual startup times for the processes to load (which increases as the number of nodes and stages goes up) will be in the range of several seconds to somewhere around a minute - you should check to see how many processes your job has at runtime by using APT_DUMP_SCORE.
Transformations do not slow down the startup time.
</a>