IPC Stage

[sb_akarmarkar]

Hi all,

Is there any performance benifit of using IPC stage ?
In which cases we can use IPC stage


Thanks,
Anupam

[balajisr]

Hi all,

Is there any performance benifit of using IPC stage ?
In which cases we can use IPC stage


Thanks,
Anupam

You can use IPC Stage to specify that connected active stage should run as seperate processes.

You can also use IPC stage between two passive stages to speed up transfer of data between them.

[kumar_s]

It can even speed up if it is conected between two passive stage. Its done by pipeline parallelism. i.e., as soon as the link is open for input to the IPC stage, the output link is also opened parallel, so in SMP or MPP the transfer rate will be more faster. It also enables inter process buffering between active stages.

[sb_akarmarkar]

You can use IPC Stage to specify that connected active stage should run as seperate processes.

You can also use IPC stage between two passive stages to speed up transfer of data between them.

Hi balajisr,

If suppose one of my process in ETL is taking 3 min to execute and another 20 min, so is it feasible to use IPC Stage....

Thanks,
Anupam

[kumar_s]

Can you explain bit more detail.

[ray.wurlod]

The answer is a definite maybe. It really does depend on what your job is doing.

If
stage ---> IPC ---> stage ---> IPC ---> stage
runs faster than
stage ---> stage ---> stage
then would
stage ---> IPC ---> IPC ---> stage ---> IPC ---> IPC ---> stage
run even faster?

The answer is no, for the same reason that nine women can't have a baby in one month.

[kumar_s]

You can use IPC Stage to specify that connected active stage should run as seperate processes.

You can also use IPC stage between two passive stages to speed up transfer of data between them.

Hi balajisr,

If suppose one of my process in ETL is taking 3 min to execute and another 20 min, so is it feasible to use IPC Stage....

Thanks,
Anupam

But may I know, how you calculated the time taken by each process?

[sb_akarmarkar]

Thanks all for there suggestion....


kumar ...

If we reading from source stage just put filtter in transformer to false
so we can get reading time ...
Suppose reading time is 3 min
Writing from text file to taget 7 min
whole ETL time is 30 min...
like that we calculate...

Thanks,
Anupam

[ray.wurlod]

Active stage execution times are reported to the nearest millisecond in the job log, in "active stage finishing" messages.

You can also enable statistics collection (to the nearest microsecond) on the Tracing tab of the Job Run Options dialog when requesting a job run from Director.