Link Partitioner

phillip.small · Post by **phillip.small** » Fri Sep 17, 2004 9:56 am

I am trying to use a Link Paritioner to see if it would speed up our 2 million row load. However, I am not sure how to set it up. Where do I link? I am getting an error. Is the link partitioner output from the transformer stage or the input file stage?

ketfos · Post by **ketfos** » Fri Sep 17, 2004 10:20 am

Hi,
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.

A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.

Please refer to datastage help on this.

What error message are you getting?

Ketfos

tonystark622 · Post by **tonystark622** » Fri Sep 17, 2004 10:33 am

When I've used them in the past, it's been something like this:

Code: Select all

                      ->XFormer--->other       -->
                    /               processing     \
                   /                                 \
--->XFormer-->Link ---->XFormer--->other       ---->Link       --->XFormer
              Partitioner           processing      Collector
                   \                                 /
                     \                             /
                       ->XFormer--->other       --> 
                                     processing

I have some with sort stages, and transformers and aggregators. Typically, you will be doing the same processing on each "leg" of the split processes.

Hope this helps,
Tony

phillip.small · Post by **phillip.small** » Fri Sep 17, 2004 11:57 am

ketfos wrote:Hi,
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.

A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.

Please refer to datastage help on this.

What error message are you getting?

Ketfos

The error that I am getting is:

The Link Collector does not support in-process active-to-active inputs or outputs. (Jobname.linkcollectorname).

chucksmith · Post by **chucksmith** » Fri Sep 17, 2004 1:18 pm

In Designer, edit your job properties (the yellow icon). On the performance tab, select the Inter process button, the recompile.

vmcburney · Post by **vmcburney** » Mon Sep 20, 2004 6:27 pm

Is this job design any faster or more effective than a non partitioned job? On Unix systems you need to run seperate instances of jobs to get true parallel performance, this job may have several streams but it will still be allocated to just one CPU. The cost of splitting and joining may be greater then the benefits of having three streams.

I'd be interested if you could run a test comparing the performance of the partitioned job to a version of the job that has a single stream.