remove duplicates
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 98
- Joined: Thu Feb 05, 2009 1:14 am
- Location: visakhapatnam
- Contact:
remove duplicates
HI,
Instead of using Remove duplicate stage , is they any other way to remove duplicates from a table.
Thanks
sunitha
Instead of using Remove duplicate stage , is they any other way to remove duplicates from a table.
Thanks
sunitha
Yes, there are other ways. All involve sorting the data. The Sort can remove duplicate rows and you can use stage variables in a transform stage to detect duplicate column values and drop those rows.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Re: remove duplicates
If you query from a DB, you can also use the DISTINCT keyword in the query sentence. This is helpful to restrict as early as possible the number of rows that are retrieved.sunitha_cts wrote:HI,
Instead of using Remove duplicate stage , is they any other way to remove duplicates from a table.
Thanks
sunitha
-
- Participant
- Posts: 98
- Joined: Thu Feb 05, 2009 1:14 am
- Location: visakhapatnam
- Contact:
-
- Participant
- Posts: 98
- Joined: Thu Feb 05, 2009 1:14 am
- Location: visakhapatnam
- Contact:
-
- Participant
- Posts: 194
- Joined: Thu May 11, 2006 9:42 am
remove duplicates
Hi Sunita,
In the sort stage keep ALLOW DUPLICATES as False this will help you to remove duplicates.
Thanks,
Chris
In the sort stage keep ALLOW DUPLICATES as False this will help you to remove duplicates.
Thanks,
Chris
Thanks,
Chris Jones
Chris Jones
-
- Participant
- Posts: 98
- Joined: Thu Feb 05, 2009 1:14 am
- Location: visakhapatnam
- Contact:
Yes
Yes, we can remove duplicates using sort stage and also every output link..but make sure that your partition type is other that auto and select perform sort and unique.. that will give you unique records..sunitha_cts wrote:Thanks Cris
hope this makes sense..
krishna
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Re: Yes
Thanks for the info , but do you trap those duplicates records from transformer stage.aladap12 wrote:Yes, we can remove duplicates using sort stage and also every output link..but make sure that your partition type is other that auto and select perform sort and unique.. that will give you unique records..sunitha_cts wrote:Thanks Cris
hope this makes sense..
krishna