DSXchange

Hi I have a file list which contains list of files to be read using sequential file with same metadata. Using file pattern, Can I just use the file name which contains the list of files to be read. and extract the data?

file_list.txt --> datafile1.dat
datafile2.dat

In other words, I have to use only the file_list.txt to read the data files. Please advise.

No, file pattern means you'd need to supply a wildcard pattern that would match all of the files you'd want to process, not a specific list of filenames. Typical solution for a 'list of files' would be a looping Sequence job that runs a single job once per filename.

I prefer the transformer looping to the wildcard approach as you can then log to a control table the names of the files you have processed and the success of that processing. If you use the wildcard approach and the job aborts half way through you have no idea which of the files have been processed.

I usually prefer to do this looping in a controlling sequence.

Sorry - when I said Transformer looping I meant Sequence looping! Wrong looping.

The file pattern option allows you to also specify a file containing a list of files, or a shell command which will return a list of files.

Regards,

In Sequential file stage?

Yup. Crap, how did I miss that? It's right there in black and white pixels in the documentation of all places:

File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.

To make matters worse, there's no "they musta just added that in the 8.x release" - I just found the same text in the 7.x docs as well. D'oh.

Worth logging in today. Learned something.

Careful, you'll make me feel like I accomplished something...

I knew I had read about it somewhere but when I looked at the sequential file stage I was expecting a "Filename file" kind of option but could find nothing so gave up. Certainly didn't know about it available in 7.x... thought it was something new in 8.5!

Hi,

Can you please ellobarate on this. I tried to read from a file which contains the list of file names but it's not considering the columns in the listed files.

Please explain for us what "considering the columns" means. If by that you mean that it doesn't respect the "First record is column names" setting for any but the first file, that's a known limitation when reading multiple files using any of the available mechanisms. Unless they've managed to fix that rather silly issue and you mean something else entirely...

Does the metadata in the link properties mention these columns that it's "not considering"?

[quote="chulett"]Yup. Crap, how did I miss that? It's right there in black and white pixels in the documentation of all places:

File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.

Hi chulett \ray

Good morning:

As per chulett, I have created the file which contains the list of file names and those files have the same metadata. And I created job which having :

Sequential file (File pattern and the file name which have file names) --> Tfr --> Sequential file.

Now I'm tring to load the data from the listed files. The job is not performaing as expected (Means it's not loading the data properly). It just taking the file names as values and outputing the same file names.
Now, i think you guys can understand the issue. Please clarrify me in this . Thank you....