Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
ds_dwh
Participant
Posts: 39 Joined: Fri May 14, 2010 6:06 am
Post
by ds_dwh » Mon Jun 14, 2010 5:51 am
hi,
how to handle duplicate records in sequential file
either deleting or sending to rej link
ANJI
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Mon Jun 14, 2010 5:55 am
There are multiple ways to identify duplicate records regardless of source. What have you tried?
-craig
"You can never have too many knives" -- Logan Nine Fingers
nagarjuna
Premium Member
Posts: 533 Joined: Fri Jun 27, 2008 9:11 pm
Location: Chicago
Post
by nagarjuna » Mon Jun 14, 2010 8:10 am
duplicates stage , sort stage , transformer stage ......
Nag
ds_dwh
Participant
Posts: 39 Joined: Fri May 14, 2010 6:06 am
Post
by ds_dwh » Tue Jun 15, 2010 3:57 am
nagarjuna wrote: duplicates stage , sort stage , transformer stage ......
with using sequential file only
not using sort,remove duplicate,T/F
ANJI
Sainath.Srinivasan
Participant
Posts: 3337 Joined: Mon Jan 17, 2005 4:49 am
Location: United Kingdom
Post
by Sainath.Srinivasan » Tue Jun 15, 2010 4:01 am
Is this an interview question ?
Maybe you can edit the file and manually remove them !!
kondeti
Premium Member
Posts: 67 Joined: Sat Mar 04, 2006 11:38 am
Post
by kondeti » Tue Jun 15, 2010 4:19 am
There is an option available in Sequential File Stage called Filter. This option directly deals with Operating System where your DataStage server is Installed either Unix Flavour or Windows. Write respective script(either dos/Unix based on your Operating system) in the Filter option and ristrict duplicate records. Thank you.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Tue Jun 15, 2010 6:27 am
ds_dwh wrote: nagarjuna wrote: duplicates stage , sort stage , transformer stage ......
with using sequential file only
Why?
-craig
"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Tue Jun 15, 2010 4:28 pm
Use Filter command in Sequential File stage. The command sort -u #filepath# might be a good choice.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.