Partitioning information Needed

ag_ram · Post by **ag_ram** » Fri Jan 25, 2008 12:43 pm

I have 2 keys to remove duplicate stage. It is partitioned and sorted on these 2 keys. The next stage is a transformer stage, where i have used a same partitioning. From remove duplicate stage to transformer stage I have not propagated one of the key.

Question: Will the records of previous partition will be there in the same partition after transformation stage? Or records may become haywire?

Please provide the comments.

balajisr · Post by **balajisr** » Fri Jan 25, 2008 10:53 pm

If you need to drop a column, why not drop it using the transformer stage especially when you have one downstream?

Raghavendra · Post by **Raghavendra** » Sun Jan 27, 2008 11:18 pm

I believe the answer will be:
Yes, the records of previous partition will be there in the same partition after transformation stage.

ag_ram · Post by **ag_ram** » Mon Jan 28, 2008 12:12 pm

After transformation stage, I have alots of other stages. But I need to have those records in the same partition.

Can anyone confirm me, wheather these records be placed in same partition.

Teej · Post by **Teej** » Mon Jan 28, 2008 6:59 pm

As long as you explicitly declare "Same" partitioning, the data will be in the same partition.

There is also an option in some stages to utilize the prior sorting you have already done.