to remove first line from mutiple files

developeretl · Post by **developeretl** » Thu Aug 04, 2011 8:10 am

Hi ,
I have three files which contain first line as UTF8. I have used sed command to remove the first line. It worked but I'm testing the job so each time when I run the job it deletes the first line. Is there any other way so that I can exclude the first line and run the job using the logic in the stages ?

chulett · Post by **chulett** » Thu Aug 04, 2011 8:22 am

Parallel job as marked or Server job as per the forum posted in?

If you are only processing one file at a time then simply tell the Sequential File stage that you have 1 header record, it will automatically be skipped.

developeretl · Post by **developeretl** » Thu Aug 04, 2011 8:26 am

I'm processing 3 files at a time. I have to skip the first line of the files and process the files. This is related to parallel jobs.

chulett · Post by **chulett** » Thu Aug 04, 2011 9:38 am

So here we are in the right place now. Are you using the "File Pattern" option to read the multiple files? From what I recall, it will only skip the header record from the first file rather than all of them which I find rather... stupid.

If your sed command is working, have you tried leveraging that in the Filter option of the stage?

developeretl · Post by **developeretl** » Thu Aug 04, 2011 9:41 am

Im reading the 3 files using three sequential stages. Can you please tell me how to set it in the filter ?

chulett · Post by **chulett** » Thu Aug 04, 2011 9:48 am

Three files in three stages = one at a time. Use the option to skip header records in each of the stages.

developeretl · Post by **developeretl** » Thu Aug 04, 2011 9:53 am

Can you please tell me how to do that in the sequential stage . Im new to datastage.

developeretl · Post by **developeretl** » Thu Aug 04, 2011 10:02 am

Can you please tell me how to do that in the sequential stage . Im new to datastage.

chulett · Post by **chulett** » Thu Aug 04, 2011 10:09 am

It's a property in the stage and documented as well. I'm not anywhere with access so I can't quote chapter and verse where to find it but I can't imagine it would be all that hard to look up.

Worst case maybe someone else can provide the gory details.

rbpty · Post by **rbpty** » Thu Aug 04, 2011 10:45 am

Using the filter option in Sequential file is simple. Here you go...

Under the options tab, choose the filter option and use the below command..

sed -e '1d' Your_full_file_path_including_name

developeretl · Post by **developeretl** » Thu Aug 04, 2011 10:50 am

rbpty wrote:Using the filter option in Sequential file is simple. Here you go...

Under the options tab, choose the filter option and use the below command..

sed -e '1d' Your_full_file_path_including_name

Thank you rbpty for the command

chulett · Post by **chulett** » Thu Aug 04, 2011 11:08 am

That's not the right approach for your particular job.

developeretl · Post by **developeretl** » Thu Aug 04, 2011 12:25 pm

Please guide me which one is the right approach.

chulett · Post by **chulett** » Thu Aug 04, 2011 12:54 pm

I have.

FranklinE · Post by **FranklinE** » Thu Aug 04, 2011 1:12 pm

I'm using 7x, and not yet familiar with differences with 8x, but here's how I'd do it:

Code: Select all

Properties
    Options
        First line is column names = True