Page 1 of 1
Inheriting a job: IPC Partition fed direct to IPC Collector
Posted: Thu Aug 24, 2006 12:13 pm
by zbethem
I've inherited a legacy job and am having a tough time deciding why it was built the way it was. I'm hoping that the community can shed some light on what I'm thinking.
Job design:
SRC --> Transformer --> IPC-part --> IPC-collect --> flat file
* note, that there is _nothing_ between the partitioner and collector.
My thoughts, the use of IPC-partition and IPC-collector is unnecessary. Can anyone think of any benefit for doing this?
Posted: Thu Aug 24, 2006 12:22 pm
by kris007
On a multiprocessor system, it makes sense, if the design is like this
Code: Select all
SRC---->>LinkPartitioner---->XFM1----->LinkCollector--->>FlatFile
| ^
| |
----------->>XFM2--------
Use of LinkPartitionar and LinkCollector is not totally ruled out unless you are on a single processor system which I highly doubt. It helps when you are dealing with large amounts of data. But the order of placing them is different IMO.
HTH
Posted: Thu Aug 24, 2006 12:25 pm
by zbethem
I agree with your statement, but I see that you have put XFM1/2 on each of the partition paths. That makes sense, as you're doing something and breaking the load out over processors.
Having the partitioner going straight to the collector with nothing in between is what doesn't make sense to me.
Posted: Thu Aug 24, 2006 12:51 pm
by kcbland
With Inter-processing checked on the job properties, the Transformer stage will essentially achieve the same effect. Remove the IPC stages and scratch your head one last time. Folks do strange things.
Re: Inheriting a job: IPC Partition fed direct to IPC Collec
Posted: Thu Aug 24, 2006 1:00 pm
by pranay
The LC and LP might have been placed consequently so as to improve the performance of the job. Or might be for renaming some columns.
Why don't u remove these componenets and compare the performance with the original one.
I think for cases like this Trial and Error is teh best solution.
Regards,
Pranay
zbethem wrote:I've inherited a legacy job and am having a tough time deciding why it was built the way it was. I'm hoping that the community can shed some light on what I'm thinking.
Job design:
SRC --> Transformer --> IPC-part --> IPC-collect --> flat file
* note, that there is _nothing_ between the partitioner and collector.
My thoughts, the use of IPC-partition and IPC-collector is unnecessary. Can anyone think of any benefit for doing this?