Regarding Partitions

manojbh31 · Post by **manojbh31** » Sat Sep 04, 2010 7:47 am

Hi,

I have job where i am using join stage and doing left outer join. Whenever i run the job the output varies. Means for each run the output count varies. earlier I was not sorting the data before joining. I tried to use hash partition and sort the data in transformer but this is also not working properly. Can anybody help me to understand how to use partitions in PX. Data is about 0.5 Million records.

Thanks

chulett · Post by **chulett** » Sat Sep 04, 2010 8:39 am

The join stage requires sorted input to work properly. Not sure how one would sort 'in a transformer', so make sure that either happens in your source selects (if from a database) or via explicit sort operations before the join. The hash partitioning should be good as long as you are partitioning on the same keys you are sorting / joining on.

Post by **daignault** » Sat Sep 04, 2010 1:11 pm

Actually, in the properties of the transformer, you can define a sort (stable or unstable) as well as partitioning info for the incoming virtual dataset.

When I deliver EE training, I suggest to developers that they use the SORT stage instead of the sort repartitioning on the link. I like the ability of the sort stage to tailor memory usage which is not available on the link properties.

Ray Daignault

chulett · Post by **chulett** » Sat Sep 04, 2010 3:05 pm

I figured what was meant was a sort on a transformer link, which to me isn't "in" the transformer but that could just be semantics... or I could just be wrong.

ray.wurlod · Post by **ray.wurlod** » Sat Sep 04, 2010 5:55 pm

It's always the link.

What confuses people is that they open the link properties via the Input tab in the stage properties.

In most cases, of course, you can right click the stage and open the link properties directly.