Page 1 of 2
Filter Condition in sequential file
Posted: Mon Apr 21, 2008 8:54 am
by Magesh_bala
Hi,
My Source file is an Sequential file, Where i have duplicate values in my Sequential file.
I have select only unique records from my source.
I dont wanted to use any other stage's to filter the duplicates, Because we have Filter option in Sequential file.
By using the filter option can i do it ???
Please revert back with an clear unix commend to be used.
Regards
Magesh
Re: Filter Condition in sequential file
Posted: Mon Apr 21, 2008 9:16 am
by sud
Magesh_bala wrote:
Please revert back with an clear unix commend to be used.
What, is that an order? and by the way, this is not a unix forum.
Try : sort -u
Posted: Tue Apr 22, 2008 12:31 am
by Magesh_bala
Posted: Tue Apr 22, 2008 1:26 am
by ray.wurlod
It IS the right one.
The -u option for the sort command means "unique" - it returns unique values of the sort keys.
Therefore, if it is not giving you the answer you require, then you are not specifying the sort command appropriately.
Posted: Tue Apr 22, 2008 4:41 am
by Magesh_bala
Sort -u, It only sorts the values.
But it didn't eliminate the duplicate values.
Posted: Tue Apr 22, 2008 4:54 am
by balajisr
You need to specify appropriate key values while removing duplicates. Refer man pages for help.
Posted: Tue Apr 22, 2008 6:34 am
by sud
Posted: Wed Apr 23, 2008 3:59 am
by Magesh_bala
Some one can help to resolve this problem.
Posted: Wed Apr 23, 2008 4:35 am
by bkumar103
Magesh_bala
before arguing please check the option.
Sort -u sorts and returns the unique records. Only thing is duplicate removal will be on the record basis not of the field baseis.
For example :
original file
abc xyz
abc xyz
xyz abc
lmn abc
lmn fgh
sort -u file will return
abc xyz
lmn abc
lmn fgh
xyz abc
If your requirement is different state it clearly.
Posted: Wed Apr 23, 2008 5:16 am
by Magesh_bala
Thanks Kumar, I thought its removes duplicate in column level.
Thank you .
Posted: Wed Apr 23, 2008 5:44 am
by Magesh_bala
Hey Dude, Its not working!!! I have checked it.
Posted: Wed Apr 23, 2008 7:05 am
by ray.wurlod
"It's not working" is singularly uninformative.
If you want us to help you need to specify the actual command(s) that you used, what you expected to get, what you actually got, and why you believe that they are different.
Posted: Wed Apr 23, 2008 7:10 am
by keshav0307
why don;t you google for "unix unique record sorting"
and try those options
Posted: Wed Apr 23, 2008 8:56 am
by Magesh_bala
Hi My Source File
11|Anil |99|MIG |POLIC|Motor
12|Guna |66|MI |OBJEC|Home
11|baba |23|MIG |POLIC|Motor
13|Ji |77|MM |cover|Tata
11|Anil |99|MIG |POLIC|Motor
14|qqq |77|QQ |Objec|MIG
22|aaa |99|aa |Gate |zz
I have used sort -u file_name.txt to eliminate duplicate records
But I am not getting the expected result
My output
11|Anil |99|MIG |POLIC|Motor
11|Anil |99|MIG |POLIC|Motor
11|baba |23|MIG |POLIC|Motor
12|Guna |66|MI |OBJEC|Home
13|Ji |77|MM |cover|Tata
14|qqq |77|QQ |Objec|MIG
22|aaa |99|aa |Gate |zz
Expected result
11|Anil |99|MIG |POLIC|Motor
11|baba |23|MIG |POLIC|Motor
12|Guna |66|MI |OBJEC|Home
13|Ji |77|MM |cover|Tata
14|qqq |77|QQ |Objec|MIG
22|aaa |99|aa |Gate |zz
help me!!! to resolve these problem.
Posted: Wed Apr 23, 2008 9:05 am
by yiminghu
Use following command
sort -u filename|uniq
Sort does not get rid of duplicate records, it only sort the file, uniq does the trick.
