DSXchange

I have 2 keys to remove duplicate stage. It is partitioned and sorted on these 2 keys. The next stage is a transformer stage, where i have used a same partitioning. From remove duplicate stage to transformer stage I have not propagated one of the key.

Question: Will the records of previous partition will be there in the same partition after transformation stage? Or records may become haywire?

Please provide the comments.

If you need to drop a column, why not drop it using the transformer stage especially when you have one downstream?

I believe the answer will be:
Yes, the records of previous partition will be there in the same partition after transformation stage.

After transformation stage, I have alots of other stages. But I need to have those records in the same partition.

Can anyone confirm me, wheather these records be placed in same partition.

As long as you explicitly declare "Same" partitioning, the data will be in the same partition.

There is also an option in some stages to utilize the prior sorting you have already done.