partitioning data on key1 and aggregating on key2
Posted: Tue Jan 24, 2006 8:00 pm
In a PX job, If data is partitioned on key 1 and then aggregated on key 2, what issues could arise?
Thx
VS
Thx
VS
Are you sorting on key 2?satish_valavala wrote:In a PX job, If data is partitioned on key 1 and then aggregated on key 2, what issues could arise?
Thx
VS
If not, the Aggregator stage will use a lot more memory than otherwise.
Partitioning on other than key2 may mean that some key2 values are on node 1 and some key2 values are on node 2 and so on. That is, you may not get all the key2 values in one group in the final result. This is almost certainly not desirable.