Link Partitioner
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 29
- Joined: Mon Aug 16, 2004 8:12 am
Link Partitioner
I am trying to use a Link Paritioner to see if it would speed up our 2 million row load. However, I am not sure how to set it up. Where do I link? I am getting an error. Is the link partitioner output from the transformer stage or the input file stage?
Hi,
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.
A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.
Please refer to datastage help on this.
What error message are you getting?
Ketfos
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.
A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.
Please refer to datastage help on this.
What error message are you getting?
Ketfos
-
- Premium Member
- Posts: 483
- Joined: Thu Jun 12, 2003 4:47 pm
- Location: St. Louis, Missouri USA
When I've used them in the past, it's been something like this:
I have some with sort stages, and transformers and aggregators. Typically, you will be doing the same processing on each "leg" of the split processes.
Hope this helps,
Tony
Code: Select all
->XFormer--->other -->
/ processing \
/ \
--->XFormer-->Link ---->XFormer--->other ---->Link --->XFormer
Partitioner processing Collector
\ /
\ /
->XFormer--->other -->
processing
Hope this helps,
Tony
-
- Participant
- Posts: 29
- Joined: Mon Aug 16, 2004 8:12 am
The error that I am getting is:ketfos wrote:Hi,
The Partitioner stage partitions data, it is processed in parallel, then the Collector stage collects it together again before writing it to a single target.
A sequential file can be input to Link Partition Stage. which can take up to 64 links and colects data from this link (Link Collector) and route them to single output link.
Please refer to datastage help on this.
What error message are you getting?
Ketfos
The Link Collector does not support in-process active-to-active inputs or outputs. (Jobname.linkcollectorname).
-
- Premium Member
- Posts: 385
- Joined: Wed Jun 16, 2004 12:43 pm
- Location: Virginia, USA
- Contact:
In Designer, edit your job properties (the yellow icon). On the performance tab, select the Inter process button, the recompile.
Chuck Smith
www.anotheritco.com
www.anotheritco.com
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
Is this job design any faster or more effective than a non partitioned job? On Unix systems you need to run seperate instances of jobs to get true parallel performance, this job may have several streams but it will still be allocated to just one CPU. The cost of splitting and joining may be greater then the benefits of having three streams.
I'd be interested if you could run a test comparing the performance of the partitioned job to a version of the job that has a single stream.
I'd be interested if you could run a test comparing the performance of the partitioned job to a version of the job that has a single stream.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn