DSXchange

I have a job like SORT--->filter---->Join (from filter there are 2 outputs to two joins, one for each). Now my problem i am hash partitioning and sorting on same keys in sort stage as well as in join stage (because the filter is not preserving the partition). How can i solve this. I want use the same partiion for 2 joins. Can trnasformer preserve partition

Partitioning is preserved. Sorting may not be. Place a Sort stage between the Filter stage and the Join stage, with sort mode "Don't sort (previously sorted)".

In any case, DataStage should have inserted a tsort operator if you did not specify sorting on the input link to the Join stage.

Make very sure that both inputs to the Join stage are identically partitioned using a key-based algorithm (hash or modulus) and identically sorted on the join keys.

just4u_sharath wrote:I have a job like SORT--->filter---->Join (from filter there are 2 outputs to two joins, one for each). Now my problem i am hash partitioning and sorting on same keys in sort stage as well as in join stage (because the filter is not preserving the partition). How can i solve this. I want use the same partiion for 2 joins. Can trnasformer preserve partition

From Filter there are 2 outputs to 2 join stages. When we filter the records, naturally the next stages which receive the records should loose the partitoned data. Am i right?

Filter wont remove the partition but will remove data from each partition.
Lets say, if you have 4 records with key 1234 wich actually flows in node1.
And due to filter, lets say two of such records has been forced to flow in different link, still that will flow in node1.