Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
ag_ram
Premium Member
Posts: 524 Joined: Wed Feb 28, 2007 3:51 am
Post
by ag_ram » Fri Jan 25, 2008 12:43 pm
I have 2 keys to remove duplicate stage. It is partitioned and sorted on these 2 keys. The next stage is a transformer stage, where i have used a same partitioning. From remove duplicate stage to transformer stage I have not propagated one of the key.
Question: Will the records of previous partition will be there in the same partition after transformation stage? Or records may become haywire?
Please provide the comments.
balajisr
Charter Member
Posts: 785 Joined: Thu Jul 28, 2005 8:58 am
Post
by balajisr » Fri Jan 25, 2008 10:53 pm
If you need to drop a column, why not drop it using the transformer stage especially when you have one downstream?
Raghavendra
Participant
Posts: 147 Joined: Sat Apr 30, 2005 1:23 am
Location: Bangalore,India
Post
by Raghavendra » Sun Jan 27, 2008 11:18 pm
I believe the answer will be:
Yes, the records of previous partition will be there in the same partition after transformation stage.
Raghavendra
Dare to dream and care to achieve ...
ag_ram
Premium Member
Posts: 524 Joined: Wed Feb 28, 2007 3:51 am
Post
by ag_ram » Mon Jan 28, 2008 12:12 pm
After transformation stage, I have alots of other stages. But I need to have those records in the same partition.
Can anyone confirm me, wheather these records be placed in same partition.
Teej
Participant
Posts: 677 Joined: Fri Aug 08, 2003 9:26 am
Location: USA
Post
by Teej » Mon Jan 28, 2008 6:59 pm
As long as you explicitly declare "Same" partitioning, the data will be in the same partition.
There is also an option in some stages to utilize the prior sorting you have already done.