Partitioning

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Ush
Participant
Posts: 55
Joined: Tue Dec 04, 2007 3:15 am

Partitioning

Post by Ush »

Hi

Please let me know the partition method to use in the following:

Requirement:Partition by key:
Description:Distributes data records to its output flow partitions according to key values.

I am using hash.Please let me know if its correct

Partition by Expression: distributes data records to its output flow partitions according to a specified DML expression

if I have phonenumber field ,the following DML expression (phonenumber divided by 10000) divides the total number of data records into 10 sections

Can I use modulus.

Partition by percentage:distributes a specified percentage of the total number of input data records to each output flow.

Please let me know which partition to use in such a case
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You seem to be confusing data partitions (~= processing nodes) with output links from a Filter or Transformer stage. Whatever you were to specify with DML expressions (in a Filter stage) will happen on every processing node - the data are already partitioned, and are being directed onto multiple links within each partition.

The Modulus partitioning algorithm is partitioning by expression, but only if the expression contains only the Mod() function and an integer column reference from the input link.

Partition by percentage is not possible. Distribution of rows to multiple output links by percentage is possible but would occur on each data partition. Note, too, that you would need a fork-join upstream to determine the value that represents 100%.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply