Reading multiple files using sequential file or other stage

[highpoint]

Hi,

I have a file which will have list of files that i need to read. All this files will be in same directory and but non similar names.

How could i make this read these files using sequential file stage or any other stage.

Say My main file is: Filelist.txt. It contains following data

input1.txt
hours2.txt
customer3.txt

The number of files will vary each time.

I like to read content of all these files and perform transformations and load it into target table.

I could do this using external source stage.


I am looking all other options i have and the easiest and optimum on performance.


Note: These files are huge around 10+ million each file.

Help is appreciated.

[ray.wurlod]

Read the file of file names using an Execute Command activity using the cat command.
In a User Variables activity convert the line terminator characters into - for example - commas so that you have a comma-delimited list.
Use this comma-delimited list in a Start Loop activity.
A job within the loop is passed the next file name off the list as StartLoop.$Counter activity variable.

[chulett]

Why not just set the Sequential File stage to File Pattern and then supply the name of your filelist as the file to process. It will read the list of files and load each one, much the same as an "indirect" load in Informatica.

[qt_ky]

So the file pattern in that case would need to be something like *.txt or *.* ?

[chulett]

No, from what I understand it would be the name of the file with the list of files in it.

[qt_ky]

Cool. Will have to try that one... !

[kandyshandy]

So the file pattern in that case would need to be something like *.txt or *.* ?

This works for sure as i have used in the past.

IBM documentation says like Craig said but no luck when i tried few years ago.

[eph]

Hi,

You can use file pattern like said before, or use a bash command in the file pattern expression to read your configuration file and pass through files names.

Eric