When using the auto partition method in a sort stage, if i set the partition key to be column 1 but sort on columns 1 and column 2, the incoming data to the sort stage is partitioned on columns 1 and 2 rather than column 1 only.
My understanding was that auto would partition data based on the partition key specified on the stage input tab.
Auto Partition method in Sort Stage
Moderators: chulett, rschirm, roy
Auto Partition method in Sort Stage
Thanks
Nigel
Nigel
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If partitioning is (Auto) you don't get to set the partitioning key at all.
(Auto) on inputs to a Join stage will use all the sort keys as the partitioning key and Hash as the partitioning algorithm.
(Auto) on inputs to a Join stage will use all the sort keys as the partitioning key and Hash as the partitioning algorithm.
Last edited by ray.wurlod on Sat Apr 25, 2009 2:57 am, edited 1 time in total.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
sureshreddy2009
- Participant
- Posts: 62
- Joined: Sat Mar 07, 2009 4:59 am
- Location: Chicago
- Contact:
Auto is default partiiton method
hiauto is a default partiioning method
when we select auto it dont allow to select key for sort at link level or unique etc, conclusion is what ever we had given in keys place in sort stage then it sorts based on that key or keys, here one doubt when we select column1 as sort key and in partitioning place if we select partition other than auto like hash, modulus etc, and partition keys are column1, column2 then it performs sort on column1 or it aborts
Suresh Reddy
ETL Developer
Research Operations
"its important to know in which direction we are moving rather than where we are"
ETL Developer
Research Operations
"its important to know in which direction we are moving rather than where we are"
Thanks for the quick reply Ray.
In summary i think what this means is that using auto to decide the partitioning keys and method for a sort stage is OK except for the situation were the next stage after the sort is one such as a drop duplicates that needs to drop records based on column1.
In summary i think what this means is that using auto to decide the partitioning keys and method for a sort stage is OK except for the situation were the next stage after the sort is one such as a drop duplicates that needs to drop records based on column1.
Thanks
Nigel
Nigel
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You could always add a second Sort stage sorting just on column1 in mode "don't sort, already sorted" and have that Sort stage generate a Cluster Key Change column.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Thanks for the quick reply Ray.
In summary i think what this means is that using auto to decide the partitioning keys and method for a sort stage is OK except for the situation were the next stage after the sort is one such as a drop duplicates that needs to drop records based on column1.
In summary i think what this means is that using auto to decide the partitioning keys and method for a sort stage is OK except for the situation were the next stage after the sort is one such as a drop duplicates that needs to drop records based on column1.
Thanks
Nigel
Nigel