Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
kumar66
Participant
Posts: 265 Joined: Thu Jul 26, 2007 12:14 am
Post
by kumar66 » Tue Jun 30, 2009 12:45 pm
Hi All,
Is there any way to remove the header from the file . Where I am using file type as file pattern.
Bascially I am reading multiple files.
Please hel me on this.
Thanks & Regards,
Kumar66
nagarjuna
Premium Member
Posts: 533 Joined: Fri Jun 27, 2008 9:11 pm
Location: Chicago
Post
by nagarjuna » Tue Jun 30, 2009 1:13 pm
Since you are using file pattern option , i think filter option wont be available in seq file stage .So , we can remove the headers with the help of transformer constraints
Nag
nirdesh2
Participant
Posts: 56 Joined: Thu Nov 20, 2008 12:18 pm
Location: Noida
Post
by nirdesh2 » Wed Jul 01, 2009 12:34 am
You can use filter stage after reading the data from multiple files to drop the rows which is the header of the file.
Nirdesh Kumar
ArndW
Participant
Posts: 16318 Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:
Post
by ArndW » Wed Jul 01, 2009 12:37 am
Under "options" in the sequential file stage you can specify "First line is Column Names" which effectively skips the first line.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Wed Jul 01, 2009 6:16 am
Not with a file pattern and multiple files, unfortunately.
-craig
"You can never have too many knives" -- Logan Nine Fingers
ArndW
Participant
Posts: 16318 Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:
Post
by ArndW » Wed Jul 01, 2009 6:19 am
Craig - I just had to run to a designer client and try that out, but now I know that you are correct.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Wed Jul 01, 2009 6:34 am
Yah, would be cool if it worked and would certainly simplify things.
-craig
"You can never have too many knives" -- Logan Nine Fingers
Sainath.Srinivasan
Participant
Posts: 3337 Joined: Mon Jan 17, 2005 4:49 am
Location: United Kingdom
Post
by Sainath.Srinivasan » Wed Jul 01, 2009 2:57 pm
Maybe you can filter them out in a stage or do 'grep -v' for something special in header.
Kryt0n
Participant
Posts: 584 Joined: Wed Jun 22, 2005 7:28 pm
Post
by Kryt0n » Wed Jul 01, 2009 3:34 pm
Just had a look for myself, I still have the "First Line is Column Names" option when selecting File Pattern, haven't run it but why can't it be used?
Or stream in as single column and do a DCount - assuming header record has different number of columns as other rows - then send through a column import.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Wed Jul 01, 2009 3:56 pm
Oh, you can certainly use it, it's just that it won't work on anything other than the first file from what I recall.
-craig
"You can never have too many knives" -- Logan Nine Fingers
Kryt0n
Participant
Posts: 584 Joined: Wed Jun 22, 2005 7:28 pm
Post
by Kryt0n » Wed Jul 01, 2009 4:25 pm
Ahhh, I see... so pretty pointless then...