Page 1 of 1
Link Partitioner
Posted: Fri Sep 17, 2004 9:56 am
by phillip.small
I am trying to use a Link Paritioner to see if it would speed up our 2 million row load. However, I am not sure how to set it up. Where do I link? I am getting an error. Is the link partitioner output from the transformer stage or the input file stage?
Posted: Fri Sep 17, 2004 10:20 am
by ketfos
Hi,
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.
A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.
Please refer to datastage help on this.
What error message are you getting?
Ketfos
Posted: Fri Sep 17, 2004 10:33 am
by tonystark622
When I've used them in the past, it's been something like this:
Code: Select all
->XFormer--->other -->
/ processing \
/ \
--->XFormer-->Link ---->XFormer--->other ---->Link --->XFormer
Partitioner processing Collector
\ /
\ /
->XFormer--->other -->
processing
I have some with sort stages, and transformers and aggregators. Typically, you will be doing the same processing on each "leg" of the split processes.
Hope this helps,
Tony
Posted: Fri Sep 17, 2004 11:57 am
by phillip.small
ketfos wrote:Hi,
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.
A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.
Please refer to datastage help on this.
What error message are you getting?
Ketfos
The error that I am getting is:
The Link Collector does not support in-process active-to-active inputs or outputs. (Jobname.linkcollectorname).
Posted: Fri Sep 17, 2004 1:18 pm
by chucksmith
In Designer, edit your job properties (the yellow icon). On the performance tab, select the Inter process button, the recompile.
Posted: Mon Sep 20, 2004 6:27 pm
by vmcburney
Is this job design any faster or more effective than a non partitioned job? On Unix systems you need to run seperate instances of jobs to get true parallel performance, this job may have several streams but it will still be allocated to just one CPU. The cost of splitting and joining may be greater then the benefits of having three streams.
I'd be interested if you could run a test comparing the performance of the partitioned job to a version of the job that has a single stream.