I have a small question on SORT stage. I am trying to get only unique records but in my stage property I used Execution mode as Parallel.
Here my question is ?
Does it really removes duplicates after sorting or should i change to sequential ?
Sort Stage Question
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It really removes duplicates. However, it is sorting each node separately so, to get the results you require, your data need to be partitioned based on the first sort key (or more, if that has fewer values than your number of nodes).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 536
- Joined: Thu Oct 11, 2007 1:48 am
- Location: Bangalore
Sort stage is used for sorting the data,if you want to remove duplicate records user "remove dupicate" stage.you have to only make sure all similar records should land up to same partition.
Thanks
Prasoon
ETL Consultant
LinkedIn :- http://www.linkedin.com/profile/view?id ... ab_pro_top
Blog:- http://dsshar.blogspot.com/
Prasoon
ETL Consultant
LinkedIn :- http://www.linkedin.com/profile/view?id ... ab_pro_top
Blog:- http://dsshar.blogspot.com/
Or you can sort and remove duplicates while sorting by setting the "Allow Duplicates" option to False in the Sort stage, i.e. a unique sort akin to a sort -u in UNIX. Of course, you have more control over which duplicates are removed using the RD stage, if you need that.prasson_ibm wrote:Sort stage is used for sorting the data,if you want to remove duplicate records user "remove dupicate" stage.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India