Reading multiple files with header

[sweta rai]

Hi everyone ,

I am using a sequential file stage . and reading multiple files with Read Method set as "File Pattern" . The problem which i'm facing is .....all files are having header in it , although i'm specifying "First Line is Column Name" to TRUE ..... the header is also getting inserted into the target except the first file's header .
I do not want to edit the multiple source files (like removing the header from each of them) .

I do not find the solution in previous posts for this problem . If anybody knows... kindly provide the solution .

[nani0907]

In sequential file we have a filter option we can filter the header using below command.

sed'1d'.


let me know how it works.

Thanks
Sana.

[nani0907]

In sequential file we have a filter option we can filter the header using below command.

sed'1d'.


let me know how it works.

Thanks
Sana.

[sweta rai]

@ nani0907

Yes it worked .But the "Filter" option is not available if the Read Method is "File Pattern" and not "Specific File(s)" .

In my case , the number of files are varying ...so i have to use "File Pattern" only to read multiple files.

So, what needs to be done in this scenario ?

[ray.wurlod]

Allow a reject link to capture the headers.

[sweta rai]

Ya , that could be done .. but all the columns in source are of varchar type so ..it smoothly getting transferred to the output .

Anyways .. i can handle that partcular header record in the transformer but i wanted to know is not there any option available in sequential file stage itself like the Filter thing ??

[kpsita]

Hi,

I am facing same issue. I did you handle this issue in transformer.

Thanks

[varunndschange]

You are reading multiple files from file pattern. Consider, if fields f1, f2, f3 and so on are there in files. Now go to filter stage and there set the condition as:
f1<>'f1'. This way only file records will be passed, not the column names.

E.g.
file1: f1, f2, f3
1, 1, 1
2, 2, 2

file2: f1, f2, f3
3,3,3
4,4,4

file3: f1, f2, f3
5,5,5
6,6,6

When you read the file pattern, records will come like as:

f1, f2, f3
1, 1, 1
2, 2, 2
f1, f2, f3
3,3,3
4,4,4
f1, f2, f3
5,5,5
6,6,6

Now, in filter stage if you define the constraint like this:
f1<>'f1'
Then from output link of filter you will get the result as:
1, 1, 1
2, 2, 2
3,3,3
4,4,4
5,5,5
6,6,6
.

Hope this solves your purpose

[ray.wurlod]

If it suits your requirements, the easiest solution is to use multiple File properties rather than read method "file pattern".