Page 1 of 2
to remove first line from mutiple files
Posted: Thu Aug 04, 2011 8:10 am
by developeretl
Hi ,
I have three files which contain first line as UTF8. I have used sed command to remove the first line. It worked but I'm testing the job so each time when I run the job it deletes the first line. Is there any other way so that I can exclude the first line and run the job using the logic in the stages ?
Posted: Thu Aug 04, 2011 8:22 am
by chulett
Parallel job as marked or Server job as per the forum posted in?
If you are only processing one file at a time then simply tell the Sequential File stage that you have 1 header record, it will automatically be skipped.
Posted: Thu Aug 04, 2011 8:26 am
by developeretl
I'm processing 3 files at a time. I have to skip the first line of the files and process the files. This is related to parallel jobs.
Posted: Thu Aug 04, 2011 9:38 am
by chulett
So here we are in the right place now. Are you using the "File Pattern" option to read the multiple files? From what I recall, it will only skip the header record from the first file rather than all of them which I find rather... stupid.
If your sed command is working, have you tried leveraging that in the Filter option of the stage?
Posted: Thu Aug 04, 2011 9:41 am
by developeretl
Im reading the 3 files using three sequential stages. Can you please tell me how to set it in the filter ?
Posted: Thu Aug 04, 2011 9:48 am
by chulett
Three files in three stages = one at a time. Use the option to skip header records in each of the stages.
Posted: Thu Aug 04, 2011 9:53 am
by developeretl
Can you please tell me how to do that in the sequential stage . Im new to datastage.
Posted: Thu Aug 04, 2011 10:02 am
by developeretl
Can you please tell me how to do that in the sequential stage . Im new to datastage.
Posted: Thu Aug 04, 2011 10:09 am
by chulett
It's a property in the stage and documented as well. I'm not anywhere with access so I can't quote chapter and verse where to find it but I can't imagine it would be all that hard to look up.
Worst case maybe someone else can provide the gory details.
Posted: Thu Aug 04, 2011 10:45 am
by rbpty
Using the filter option in Sequential file is simple. Here you go...
Under the options tab, choose the filter option and use the below command..
sed -e '1d' Your_full_file_path_including_name
Posted: Thu Aug 04, 2011 10:50 am
by developeretl
rbpty wrote:Using the filter option in Sequential file is simple. Here you go...
Under the options tab, choose the filter option and use the below command..
sed -e '1d' Your_full_file_path_including_name
Thank you rbpty for the command
Posted: Thu Aug 04, 2011 11:08 am
by chulett
That's not the right approach for your particular job.
Posted: Thu Aug 04, 2011 12:25 pm
by developeretl
Please guide me which one is the right approach.
Posted: Thu Aug 04, 2011 12:54 pm
by chulett
I have.
Posted: Thu Aug 04, 2011 1:12 pm
by FranklinE
I'm using 7x, and not yet familiar with differences with 8x, but here's how I'd do it:
Code: Select all
Properties
Options
First line is column names = True