Page 1 of 1
how to handle duplicate records in sequential file
Posted: Mon Jun 14, 2010 5:51 am
by ds_dwh
hi,
how to handle duplicate records in sequential file
either deleting or sending to rej link
Posted: Mon Jun 14, 2010 5:55 am
by chulett
There are multiple ways to identify duplicate records regardless of source. What have you tried?
Posted: Mon Jun 14, 2010 8:10 am
by nagarjuna
duplicates stage , sort stage , transformer stage ......
with using sequential file only
Posted: Tue Jun 15, 2010 3:57 am
by ds_dwh
nagarjuna wrote:duplicates stage , sort stage , transformer stage ......
with using sequential file only
not using sort,remove duplicate,T/F
Posted: Tue Jun 15, 2010 4:01 am
by Sainath.Srinivasan
Is this an interview question ?
Maybe you can edit the file and manually remove them !!
Posted: Tue Jun 15, 2010 4:19 am
by kondeti
There is an option available in Sequential File Stage called Filter. This option directly deals with Operating System where your DataStage server is Installed either Unix Flavour or Windows. Write respective script(either dos/Unix based on your Operating system) in the Filter option and ristrict duplicate records. Thank you.
Re: with using sequential file only
Posted: Tue Jun 15, 2010 6:27 am
by chulett
ds_dwh wrote:nagarjuna wrote:duplicates stage , sort stage , transformer stage ......
with using sequential file only
Why?
Posted: Tue Jun 15, 2010 4:28 pm
by ray.wurlod
Use Filter command in Sequential File stage. The command sort -u #filepath# might be a good choice.