partition
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 236
- Joined: Sun Apr 01, 2007 7:41 am
- Location: Michigan
partition
I have a job like SORT--->filter---->Join (from filter there are 2 outputs to two joins, one for each). Now my problem i am hash partitioning and sorting on same keys in sort stage as well as in join stage (because the filter is not preserving the partition). How can i solve this. I want use the same partiion for 2 joins. Can trnasformer preserve partition
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Partitioning is preserved. Sorting may not be. Place a Sort stage between the Filter stage and the Join stage, with sort mode "Don't sort (previously sorted)".
In any case, DataStage should have inserted a tsort operator if you did not specify sorting on the input link to the Join stage.
Make very sure that both inputs to the Join stage are identically partitioned using a key-based algorithm (hash or modulus) and identically sorted on the join keys.
In any case, DataStage should have inserted a tsort operator if you did not specify sorting on the input link to the Join stage.
Make very sure that both inputs to the Join stage are identically partitioned using a key-based algorithm (hash or modulus) and identically sorted on the join keys.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 236
- Joined: Sun Apr 01, 2007 7:41 am
- Location: Michigan
Re: partition
From Filter there are 2 outputs to 2 join stages. When we filter the records, naturally the next stages which receive the records should loose the partitoned data. Am i right?just4u_sharath wrote:I have a job like SORT--->filter---->Join (from filter there are 2 outputs to two joins, one for each). Now my problem i am hash partitioning and sorting on same keys in sort stage as well as in join stage (because the filter is not preserving the partition). How can i solve this. I want use the same partiion for 2 joins. Can trnasformer preserve partition
Filter wont remove the partition but will remove data from each partition.
Lets say, if you have 4 records with key 1234 wich actually flows in node1.
And due to filter, lets say two of such records has been forced to flow in different link, still that will flow in node1.
Lets say, if you have 4 records with key 1234 wich actually flows in node1.
And due to filter, lets say two of such records has been forced to flow in different link, still that will flow in node1.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'