Inheriting a job: IPC Partition fed direct to IPC Collector

[zbethem]

I've inherited a legacy job and am having a tough time deciding why it was built the way it was. I'm hoping that the community can shed some light on what I'm thinking.

Job design:
SRC --> Transformer --> IPC-part --> IPC-collect --> flat file

* note, that there is _nothing_ between the partitioner and collector.

My thoughts, the use of IPC-partition and IPC-collector is unnecessary. Can anyone think of any benefit for doing this?

[kris007]

On a multiprocessor system, it makes sense, if the design is like this

Code: Select all

SRC---->>LinkPartitioner---->XFM1----->LinkCollector--->>FlatFile
                        |                                   ^
                        |                                   |
                         ----------->>XFM2--------

Use of LinkPartitionar and LinkCollector is not totally ruled out unless you are on a single processor system which I highly doubt. It helps when you are dealing with large amounts of data. But the order of placing them is different IMO.

HTH

[zbethem]

I agree with your statement, but I see that you have put XFM1/2 on each of the partition paths. That makes sense, as you're doing something and breaking the load out over processors.

Having the partitioner going straight to the collector with nothing in between is what doesn't make sense to me.

[kcbland]

With Inter-processing checked on the job properties, the Transformer stage will essentially achieve the same effect. Remove the IPC stages and scratch your head one last time. Folks do strange things.

[pranay]

The LC and LP might have been placed consequently so as to improve the performance of the job. Or might be for renaming some columns.
Why don't u remove these componenets and compare the performance with the original one.
I think for cases like this Trial and Error is teh best solution.

Regards,
Pranay

I've inherited a legacy job and am having a tough time deciding why it was built the way it was. I'm hoping that the community can shed some light on what I'm thinking.

Job design:
SRC --> Transformer --> IPC-part --> IPC-collect --> flat file

* note, that there is _nothing_ between the partitioner and collector.

My thoughts, the use of IPC-partition and IPC-collector is unnecessary. Can anyone think of any benefit for doing this?