Reading multiple input files using external source stage

[Abhijeet]

Hi,

I have to read N input files all with the same schema.Their complete path along with the filenames are stored in file A. Now i have to concatenate all these files to create one input file.

Right now I have a job which reads the file A from a external source stage using the unix command

"cat A | xargs zcat -f"

And then links the output of the stage to a sequential file.

My question is , is there a better approach to handle the above scenario?

Thanks
~Abhi

[sanjay]

Hi

u can use fileset .

Thanks
Sanjay

Hi,

I have to read N input files all with the same schema.Their complete path along with the filenames are stored in file A. Now i have to concatenate all these files to create one input file.

Right now I have a job which reads the file A from a external source stage using the unix command

"cat A | xargs zcat -f"

And then links the output of the stage to a sequential file.

My question is , is there a better approach to handle the above scenario?

Thanks
~Abhi

[Abhijeet]

File set cannot read file names from a sequential file

Do you mean creating a .fs file{which will need some scripting} and then read the file using fileset stage ?

[sanjay]

use file pattern option of sequential file stage and load all the file in fileset.

Thanks
Sanjay

File set cannot read file names from a sequential file

Do you mean creating a .fs file{which will need some scripting} and then read the file using fileset stage ?

[ray.wurlod]

Could you generate the stream of data using something like cat `cat FileA`?