Hi All,
I am new to data stage.I need some information regarding the partition key which need to be appliedon the fallowing stages.
1)Aggregator
2)Stored procedure.
3)filter
4)funnel
5)look up
6)transformer
Please provide me normally which partitions are applied on these stages.
we are applying same partiton on the transformer stage and entire partition on the reference table and hash partition on main table in look up stage .
Please let me know if the partions whcih im am applying is imcorrect.
Thanks,
Ramesh,
Question Regarding partitons
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Key-based partitioning involves the Hash algorithm unless the partitioning key is a single, essentially unbroken, integer sequence, in which case Modulus consumes fewer resources.
(1) Aggregator: partition on grouping keys
(2) Stored procedure: usually executed in sequential mode
(3) Filter: any partitioning algorithm will do
(4) Funnel: any partitioning algorithm will do
(5) Lookup stage: see below
(6) Transformer stage: any partitioning algorithm will usually do
When any partitioning algorithm will do, one ideally chooses one that gives the most even spread of processing over the available processing nodes, at the same time trying to minimize re-partitioning, particularly in an MPP environment.
With a Lookup stage there are two possibilities.
(a) Any kind of partitioning on the stream input with Entire on the reference input(s)
(b) Identical key-based partitioning on the stream and reference inputs
Other stage types, such as Join, Merge and Remove Duplicates, also need their data to be key partitioned.
The Transformer stage usually does not care about how the data are partitioned. However, if you are using stage variables to compare one row with the previous one, then key-based partitioning must be used to ensure that the things being compared are on the same processing node.
(1) Aggregator: partition on grouping keys
(2) Stored procedure: usually executed in sequential mode
(3) Filter: any partitioning algorithm will do
(4) Funnel: any partitioning algorithm will do
(5) Lookup stage: see below
(6) Transformer stage: any partitioning algorithm will usually do
When any partitioning algorithm will do, one ideally chooses one that gives the most even spread of processing over the available processing nodes, at the same time trying to minimize re-partitioning, particularly in an MPP environment.
With a Lookup stage there are two possibilities.
(a) Any kind of partitioning on the stream input with Entire on the reference input(s)
(b) Identical key-based partitioning on the stream and reference inputs
Other stage types, such as Join, Merge and Remove Duplicates, also need their data to be key partitioned.
The Transformer stage usually does not care about how the data are partitioned. However, if you are using stage variables to compare one row with the previous one, then key-based partitioning must be used to ensure that the things being compared are on the same processing node.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
No, for that would undermine the entire reason for having premium membership, which is one means by which the bandwidth costs of DSXchange are met. And without those being met there would be no more DSXchange. It's less than 30 cents per day.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.