Remove Duplicates

arshi · Post by **arshi** » Thu Feb 05, 2009 2:47 am

Hi,

I want to remove the duplicate data on the DATE column data.
The date format is DDMMYYYY.

Using Datastage Server Edition (7.5) , source is Flat file.

Any one have the solution for this?

Regards,
Arshi

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu Feb 05, 2009 4:14 am

What have you tried?

arshi · Post by **arshi** » Thu Feb 05, 2009 5:13 am

Hi,

First sort the data by using the Sort stage and use the stage variables to remove the duplicates.

By using the sort stage , It will work fine for a particular month data but not on entire data.

Any one have the solution for this.

Regards,
Arshi.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu Feb 05, 2009 5:28 am

What about any other columns in the data?

How do you define duplicate in you data?

dr.murthy · Post by **dr.murthy** » Thu Feb 05, 2009 6:06 am

arshi wrote:Hi,

First sort the data by using the Sort stage and use the stage variables to remove the duplicates.

By using the sort stage , It will work fine for a particular month data but not on entire data.

Any one have the solution for this.

Regards,
Arshi.

hi ,

do one thing in the sort stage options tab just specify the options
ALLOW DUPLICATES IS TRUE ,no need to use aditionally remove duplicates stage.make sure that select the execution mode is sequential.it shoud works fine.

arshi · Post by **arshi** » Thu Feb 05, 2009 6:24 am

Hi Murthy,

I didnot found any options tab in the sort stage . I am using the server edition (7.5) . Can you explain where it is exactly?

Hi Sainath,

As per my requirement I have to sort column1 and column2. Here, column2 having the date data (DDMMYYYY).

If i use the sort stage its not giving the correct result.I think you understand my requirement.

Regards,
Arshi

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu Feb 05, 2009 6:28 am

Why don't you write into an hash file and read from it.

dr.murthy · Post by **dr.murthy** » Thu Feb 05, 2009 6:34 am

[quote="arshi"]Hi Murthy,

I didnot found any options tab in the sort stage . I am using the server edition (7.5) . Can you explain where it is exactly?

which version of DS you areusing ?.it is PX or server?

arshi · Post by **arshi** » Thu Feb 05, 2009 6:58 am

Murthy,

Using Server Edition (7.5.1)

chulett · Post by **chulett** » Thu Feb 05, 2009 8:33 am

No such option in the Server sort stage, checking and removing duplicates would need to happen downstream from the Sort.

JRodriguez · Post by **JRodriguez** » Thu Feb 05, 2009 9:35 am

Hi Archi,

Before sorting the data add an extra column where you want to have the date formatted as YYYYMMDD and use it as your sort variable, use the stage variables to remove the duplicates.

It will work fine for on entire data.

ray.wurlod · Post by **ray.wurlod** » Thu Feb 05, 2009 12:14 pm

If you're sorting using the sort command as a filter in the Sequential File stage, why not just add the "unique" (-u) option to that command?