Filter Condition in sequential file

Magesh_bala · Post by **Magesh_bala** » Mon Apr 21, 2008 8:54 am

Hi,

My Source file is an Sequential file, Where i have duplicate values in my Sequential file.
I have select only unique records from my source.
I dont wanted to use any other stage's to filter the duplicates, Because we have Filter option in Sequential file.
By using the filter option can i do it ???

Please revert back with an clear unix commend to be used.

Regards
Magesh

sud · Post by **sud** » Mon Apr 21, 2008 9:16 am

Magesh_bala wrote:
Please revert back with an clear unix commend to be used.

What, is that an order? and by the way, this is not a unix forum.

Try : sort -u

Magesh_bala · Post by **Magesh_bala** » Tue Apr 22, 2008 12:31 am

Its not the right one.
before posting, Please check and post.

ray.wurlod · Post by **ray.wurlod** » Tue Apr 22, 2008 1:26 am

It IS the right one.

The -u option for the sort command means "unique" - it returns unique values of the sort keys.

Therefore, if it is not giving you the answer you require, then you are not specifying the sort command appropriately.

Magesh_bala · Post by **Magesh_bala** » Tue Apr 22, 2008 4:41 am

Sort -u, It only sorts the values.
But it didn't eliminate the duplicate values.

balajisr · Post by **balajisr** » Tue Apr 22, 2008 4:54 am

You need to specify appropriate key values while removing duplicates. Refer man pages for help.

sud · Post by **sud** » Tue Apr 22, 2008 6:34 am

Magesh_bala wrote:Its not the right one.
before posting, Please check and post.

Magesh_bala · Post by **Magesh_bala** » Wed Apr 23, 2008 3:59 am

Some one can help to resolve this problem.

bkumar103 · Post by **bkumar103** » Wed Apr 23, 2008 4:35 am

Magesh_bala
before arguing please check the option.
Sort -u sorts and returns the unique records. Only thing is duplicate removal will be on the record basis not of the field baseis.

For example :

original file
abc xyz
abc xyz
xyz abc
lmn abc
lmn fgh

sort -u file will return
abc xyz
lmn abc
lmn fgh
xyz abc

If your requirement is different state it clearly.

Magesh_bala · Post by **Magesh_bala** » Wed Apr 23, 2008 5:16 am

Thanks Kumar, I thought its removes duplicate in column level.
Thank you .

Magesh_bala · Post by **Magesh_bala** » Wed Apr 23, 2008 5:44 am

Hey Dude, Its not working!!! I have checked it.

ray.wurlod · Post by **ray.wurlod** » Wed Apr 23, 2008 7:05 am

"It's not working" is singularly uninformative.

If you want us to help you need to specify the actual command(s) that you used, what you expected to get, what you actually got, and why you believe that they are different.

keshav0307 · Post by **keshav0307** » Wed Apr 23, 2008 7:10 am

why don;t you google for "unix unique record sorting"
and try those options

Magesh_bala · Post by **Magesh_bala** » Wed Apr 23, 2008 8:56 am

Hi My Source File
11|Anil |99|MIG |POLIC|Motor
12|Guna |66|MI |OBJEC|Home
11|baba |23|MIG |POLIC|Motor
13|Ji |77|MM |cover|Tata
11|Anil |99|MIG |POLIC|Motor
14|qqq |77|QQ |Objec|MIG
22|aaa |99|aa |Gate |zz

I have used sort -u file_name.txt to eliminate duplicate records
But I am not getting the expected result
My output
11|Anil |99|MIG |POLIC|Motor
11|Anil |99|MIG |POLIC|Motor
11|baba |23|MIG |POLIC|Motor
12|Guna |66|MI |OBJEC|Home
13|Ji |77|MM |cover|Tata
14|qqq |77|QQ |Objec|MIG
22|aaa |99|aa |Gate |zz

Expected result

11|Anil |99|MIG |POLIC|Motor
11|baba |23|MIG |POLIC|Motor
12|Guna |66|MI |OBJEC|Home
13|Ji |77|MM |cover|Tata
14|qqq |77|QQ |Objec|MIG
22|aaa |99|aa |Gate |zz

help me!!! to resolve these problem.

yiminghu · Post by **yiminghu** » Wed Apr 23, 2008 9:05 am

Use following command
sort -u filename|uniq

Sort does not get rid of duplicate records, it only sort the file, uniq does the trick.

DSXchange

Filter Condition in sequential file

Filter Condition in sequential file

Re: Filter Condition in sequential file