Hi,
I have job where i am using join stage and doing left outer join. Whenever i run the job the output varies. Means for each run the output count varies. earlier I was not sorting the data before joining. I tried to use hash partition and sort the data in transformer but this is also not working properly. Can anybody help me to understand how to use partitions in PX. Data is about 0.5 Million records.
Thanks
Regarding Partitions
Moderators: chulett, rschirm, roy
The join stage requires sorted input to work properly. Not sure how one would sort 'in a transformer', so make sure that either happens in your source selects (if from a database) or via explicit sort operations before the join. The hash partitioning should be good as long as you are partitioning on the same keys you are sorting / joining on.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Actually, in the properties of the transformer, you can define a sort (stable or unstable) as well as partitioning info for the incoming virtual dataset.
When I deliver EE training, I suggest to developers that they use the SORT stage instead of the sort repartitioning on the link. I like the ability of the sort stage to tailor memory usage which is not available on the link properties.
Ray Daignault
When I deliver EE training, I suggest to developers that they use the SORT stage instead of the sort repartitioning on the link. I like the ability of the sort stage to tailor memory usage which is not available on the link properties.
Ray Daignault
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It's always the link.
What confuses people is that they open the link properties via the Input tab in the stage properties.
In most cases, of course, you can right click the stage and open the link properties directly.
What confuses people is that they open the link properties via the Input tab in the stage properties.
In most cases, of course, you can right click the stage and open the link properties directly.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.