Multiple file reading using file pattern

[Roopanwita]

Hi,

I am trying to read multiple file of same layout and similar file name using File Pattern of Sequential file stage.

All the files has file header , I defined First Line has header in Seq file Stage, but it is considering header of only 1 file , for all other files , it is still reading header as first record.

Is there a way , where it skips header of all the files

Thank you in Advance.

[chulett]

Sorry, I have nothing useful to say other than - dang, I would have thought they would have fixed / handled that by now.

Well... you could probably handle this via sed / awk / grep in the Filter option of the stage.

[eph]

Hi,

I guess you can do it that way :
-activate both ouput filename in one column & row number in one column output
-hash on filename&sort on filename&row number in a transformer, then use variable logic to get rid of the first line, like below

Code: Select all

svIN=filename
svTestFirstLine=if svIN<>svOLDIN then 1 else 0
svOLDIN=filename

Let us know if it works

Eric

[Roopanwita]

Hi,

Thank you for reply.

I have filtered the header in Transformer stage , job is working fine , but just wanted to check if there is any option /parameter where it will treat header of each file as header while reading using file pattern.

Thank you.

[eph]

Hi,

As I understand it, the sequential file stage is applying a "cat" (abbreviation of concatenate) command line to the file pattern, which means it concatenate all files matching the pattern into one stream. This is why it can't handle the header in each file.

Eric

[qt_ky]

Version 11.3 processes a file pattern in parallel by default.