remove duplicate from a file on local

saini · Post by **saini** » Wed Jan 07, 2009 10:43 am

I have a huge '|' separated data file. I want to remove duplicate from Sequential File. The file is on my local machine I donot want to put it on the server. Is there any way we can remove duplicate from a file in local.

narasimha · Post by **narasimha** » Wed Jan 07, 2009 10:53 am

Depends on how huge it is ...
Try downloading a free version of TextPad.
There is a feature to sort the delete duplicate lines.

chulett · Post by **chulett** » Wed Jan 07, 2009 10:53 am

There's no way to do that in DOS ("on local") that I am aware of without loading third-party tools or writing C programs and the like.

With PX you should have the MKS Toolkit installed on your 'server' and thus have 'UNIX' capabilities there. Try "sort -u" to get unique records. You'll probably need to tell it what fields make the record unique, otherwise it will consider the entire record when removing duplicates.

MarkB · Post by **MarkB** » Wed Jan 07, 2009 11:09 am

...or, you could get Ultra-Edit (and there is a free trial) which can easily handle massive files and can do sorts and remove duplicates.

chulett · Post by **chulett** » Wed Jan 07, 2009 11:12 am

D'oh. I own a registered copy of UltraEdit and didn't know you could use it for that. Learn something new every day.