Partitioning & Sort
Posted: Wed Oct 12, 2011 9:46 am
I have 3 questions:
Question 1:
I have a transformer for which i am doing link sort and hash partitioning to perform some logic
The keys are:
Product (partitioned & Sorted)
minqty (sorted)
maxqty (sorted)
mincost (sorted)
maxcost (sorted)
Now the output of this goes to 2 streams one with remove duplicate stage and the other with join stage
Remove duplicate:
I would like to remove duplicate on product,minqty,maxqty,mincost,maxcost.
Do i have to do link sort and hash partitioning on all the above keys or i can use the same partitioning as all the same product will be on same partition.
Join stage:
Will the same case apply for the join stage as well.
Question 2:
For a transformer stage when we do the link sort.
we have an option in transformer properties to preserve output sort order.
Does the other stages like remove dup, join, aggregator, filter will maintain sort order in their output partitions or no.
Question 3:
I see an option of stable sort in transformer. I read documentation but did not made complete sense to me.
Would appreciate if someone could explain where the stable sort can be used.
Question 1:
I have a transformer for which i am doing link sort and hash partitioning to perform some logic
The keys are:
Product (partitioned & Sorted)
minqty (sorted)
maxqty (sorted)
mincost (sorted)
maxcost (sorted)
Now the output of this goes to 2 streams one with remove duplicate stage and the other with join stage
Remove duplicate:
I would like to remove duplicate on product,minqty,maxqty,mincost,maxcost.
Do i have to do link sort and hash partitioning on all the above keys or i can use the same partitioning as all the same product will be on same partition.
Join stage:
Will the same case apply for the join stage as well.
Question 2:
For a transformer stage when we do the link sort.
we have an option in transformer properties to preserve output sort order.
Does the other stages like remove dup, join, aggregator, filter will maintain sort order in their output partitions or no.
Question 3:
I see an option of stable sort in transformer. I read documentation but did not made complete sense to me.
Would appreciate if someone could explain where the stable sort can be used.